I'm having a huge list of dependencies in my package.json and at some build step, which is completely independent from others, I only need to resolve a single dependency, but in the version specified in the package.json.
Is this even possible? A solution based on npm would also be fine.
I thought yarn add package-name --force could do it because it will determine the changes, but as there's no node_modules yet, it will also install everything first.
Related
This question applies to either npm or yarn and I would like to know if what I need to do can be done with one or the other or both. For the sake of clarity I will only refer to yarn commands as I would prefer the yarn solution.
I have a bash script where I conditionally yarn upgrade a node package that is guaranteed to be in the package.json file but has not yet been installed. I would like to potentially reduce the number of times I call yarn install. As it stands I need to call yarn install and then yarn upgrade but I don't need to because I will be calling yarn install later in the script.
I think I can save a call to yarn install (via a yarn upgrade) in my script by simply updating the version number for the node package I want upgraded in package.json but then do not actually install any of those files since I will be calling yarn install at a later time in the script.
I would simply like to change the version number of a specific node package without installing any files using either a yarn or npm command. If this is not possible is this a good use case to parse package.json by hand and insert the version number with brute force?
Dependencies can be added to your package.json without installing them by using npm-add-dependencies
You can use it on-they-fly via npx, for example:
npx add-dependencies browser-sync-webpack-plugin#^2 browser-sync#^2 --dev
This will add browser-sync-webpack-plugin version ^2 and browser-sync version 2^ to devDependencies in your package.json
I'm working with continuous integration and discovered the npm ci command.
I can't figure what the advantages are of using this command for my workflow.
Is it faster? Does it make the test harder, okay, and after?
From the official documentation for npm ci:
In short, the main differences between using npm install and npm ci are:
The project must have an existing package-lock.json or npm-shrinkwrap.json.
If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.
npm ci can only install entire projects at a time: individual dependencies cannot be added with this command.
If a node_modules is already present, it will be automatically removed before npm ci begins its install.
It will never write to package.json or any of the package-locks: installs are essentially frozen.
Essentially,
npm install reads package.json to create a list of dependencies and uses package-lock.json to inform which versions of these dependencies to install. If a dependency is not in package-lock.json it will be added by npm install.
npm ci (also known as Clean Install) is meant to be used in automated environments — such as test platforms, continuous integration, and deployment — or, any situation where you want to make sure you're doing a clean install of your dependencies.
It installs dependencies directly from package-lock.json and uses package.json only to validate that there are no mismatched versions. If any dependencies are missing or have incompatible versions, it will throw an error.
Use npm install to add new dependencies, and to update dependencies on a project. Usually, you would use it during development after pulling changes that update the list of dependencies but it may be a good idea to use npm ci in this case.
Use npm ci if you need a deterministic, repeatable build. For example during continuous integration, automated jobs, etc. and when installing dependencies for the first time, instead of npm install.
npm install
Installs a package and all its dependencies.
Dependencies are driven by npm-shrinkwrap.json and package-lock.json (in that order).
without arguments: installs dependencies of a local module.
Can install global packages.
Will install any missing dependencies in node_modules.
It may write to package.json or package-lock.json.
When used with an argument (npm i packagename) it may write to package.json to add or update the dependency.
when used without arguments, (npm i) it may write to package-lock.json to lock down the version of some dependencies if they are not already in this file.
npm ci
Requires at least npm v5.7.1.
Requires package-lock.json or npm-shrinkwrap.json to be present.
Throws an error if dependencies from these two files don't match package.json.
Removes node_modules and install all dependencies at once.
It never writes to package.json or package-lock.json.
Algorithm
While npm ci generates the entire dependency tree from package-lock.json or npm-shrinkwrap.json, npm install updates the contents of node_modules using the following algorithm (source):
load the existing node_modules tree from disk
clone the tree
fetch the package.json and assorted metadata and add it to the clone
walk the clone and add any missing dependencies
dependencies will be added as close to the top as is possible
without breaking any other modules
compare the original tree with the cloned tree and make a list of
actions to take to convert one to the other
execute all of the actions, deepest first
kinds of actions are install, update, remove and move
npm ci will delete any existing node_modules folder and relies on the package-lock.json file to install the specific version of each package. It is significantly faster than npm install because it skips some features. Its clean state install is great for ci/cd pipelines and docker builds! You also use it to install everything all at once and not specific packages.
While everyone else has answered the technical differences none explain in what situations to use both.
You should use them in different situations.
npm install is great for development and in the CI when you want to cache the node_modules directory.
When to use this? You can do this if you are making a package for other people to use (you do NOT include node_modules in such a release). Regarding the caching, be careful, if you plan to support different versions of Node.js remember that node_modules might have to be reinstalled due to differences between the Node.js runtime requirements. If you wish to stick to one version, stick to the latest LTS.
npm ci should be used when you are to test and release a production application (a final product, not to be used by other packages) since it is important that you have the installation be as deterministic as possible, this install will take longer but will ultimately make your application more reliable (you do include node_modules in such a release). Stick with LTS version of Node.js.
npm i and npm ci both utilize the npm cache if it exists, this cache lives normally at ~/.npm.
Also, npm ci respects the package-lock.json file. Unlike npm install, which rewrites the file and always installs new versions.
Bonus: You could mix them depending on how complex you want to make it. On feature branches in git you could cache the node_modules to increase your teams productivity and on the merge request and master branches rely on npm ci for a deterministic outcome.
The documentation you linked had the summary:
In short, the main differences between using npm install and npm ci are:
The project must have an existing package-lock.json or npm-shrinkwrap.json.
If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.
npm ci can only install entire projects at a time: individual dependencies cannot be added with this command.
If a node_modules is already present, it will be automatically removed before npm ci begins its install.
It will never write to package.json or any of the package-locks: installs are essentially frozen.
The commands are very similar in functionality however the difference is in the approach taken to install the dependencies specified in your package.json and package-lock.json files.
npm ci performs a clean install of all the dependencies of your app whereas npm install may skip some installations if they already exist on the system. A problem may arise if the version already installed on the system isn't the one your package.json intended to install i.e. the installed version is different from the 'required' version.
Other differences would be that npm ci never touches your package*.json files. It will stop installation and show an error if the dependency versions do not match in the package.json and package-lock.json files.
You can read a much better explanation from the official docs here.
Additionally, you may want to read about package locks here.
It is worth having in mind that light node docker images like alpine do not have Python installed which is a dependency of node-gyp which is used by npm ci.
I think it's a bit opinionated that in order to have npm ci working you need to install Python as dependency in your build.
More info here Docker and npm - gyp ERR! not ok
npm ci - install exactly what is listed in package-lock.json
npm install - without changing any versions in package.json, use package.json to create/update package-lock.json, then install exactly what is listed in package-lock.json
npm update - update package.json packages to latest versions, then use package.json to create/update package-lock.json, then install exactly what is listed in package-lock.json
Or said a different way, npm ci changes 0 package files, npm install changes 1 package file, and npm update changes 2 package files.
It does a clean install, use it in situations where you would delete node_modules and re-run npm i.
I have no idea why some people think it's short for "continuous integration". There is an npm install command that can be run as npm i and an npm clean-install command that can be run as npm ci.
npm install is the command used to install the dependencies listed in a project's package.json file, while npm ci is a command that installs dependencies from a package-lock.json or npm-shrinkwrap.json file. The npm ci command is typically used in continuous integration (CI) environments, where the package-lock.json or npm-shrinkwrap.json file is checked into version control and should not be modified. Because npm ci installs dependencies from a locked file, it is a faster and more reliable way to install dependencies than npm install, which could install different versions of dependencies based on the state of the package.json file.
I've created my own npm package, let's call it XYZ, it has #material-ui dependency in it's package.json file.
When I install it in project A I have nested node_modules inside of XYZ folder(so it's A\node_modules\XYZ\node_modules\#material-ui), but when I install it in project B I don't have nested node_modules folder. Both project A and B has #material-ui in their package.json files with same versions.
How to force my XYZ package to use #material-ui from A\node_modules?
There are upside of having less nested folders and downside having more folders in node_modules folder directly and version control problems.
Use correct npm version
Correct yarn and npm (ie: npm v3) should not have such structure issue. It should always flatten the whole structure where possible and only have nested node_modules if the versions are incompatible with the one at top.
Check versions
So if you have it working properly on one project and not on another, its probably due to version. Check out if the #material-ui is same version on both. Maybe two different packages are conflicting with each other at some point.
Check how you are installing them
From your question, it says it's same version. However, you did not mention how you installed your package on both project. If you install with yarn link or npm link it should install dependencies properly as expected.
Check if you are using different packages
If you check the package, recently material-ui has been deprecated, and the notice says to upgrade to #material-ui/core instead. It might be some packages inside that folder is not same. Either way, it's like this whenever there is some dependency conflict. Check inside the #material-ui folder.
Flatten them manually (dangerous)
There are several packages to forcefully resolve this issue. They will go thru the nested node_modules folders and flatten them into single folder.
flatten-packages
Install with, npm install -g flatten-packages.
Run executable flatten-packages to rearrange all packages in node_modules folder in the project directory.
Flatten will delete older version of a package. You should take care of version breaking changes related errors.
You can use npm dedupe command to accomplish this.
You can put the command in postinstall script in package.json, and every time NPM installs package, the npm dedupe command will flatten all the duplicated packages in same version for you.
For more information, see https://docs.npmjs.com/cli/dedupe
npm postinstall script
I had the same issue in a React Native app with my NPM package.
The problem was that in project A the version of React Native used was (0.59.5) below the version used in my package (0.59.8).
Installing the package in a brand new project (B), of course was using the latest version of React Native in that moment, that was the same of my package (0.59.8).
I have another addition to the accepted answer:
Clear Local node_modules folder Cache
rm -rf node_modules
Handle with care: Sometimes migrating projects to new npm modules can cause weird cache issues inside a node_modules folder, especially those that have been around for a while, or happened to have newer versions of packages installed in sub-dependencies that differed from the installed version in root.
Once you remove direct dependencies via the package.json dependencies, the packages will be removed from the <root>/node_modules. This can cause a bug where the new modules are still nested under your dependency instead of being moved to root as expected.
So by wiping out your local node_modules, you can do a clean reinstall and let the flattening to its work.
Is there any short command to move a module from devDependencies to dependencies in package.json?
I find myself always doing this:
npm uninstall <module_name> --save-dev
npm install <module_name> --save
Is there a shorter approach to this?
Shorthand to move from devDependencies to dependencies (prod):
npm i <module_name> -P
If you want to do the opposite (i.e. move a module from dependencies to devDependencies) just do:
npm install <module_name> --save-dev
or shorthand:
npm i <module_name> -D
Yes! to move a module from devDependencies to dependencies:
npm install <module_name> --save-prod
In yarn:
Move a module from devDependencies to dependencies:
yarn remove <module_name> --dev && yarn add <module_name>
Move a module from dependencies to devDependencies :
yarn remove <module_name> && yarn add <module_name> --dev
As said in the comments, the command actually deletes the module and reinstall it in the new place.
The problem with using npm or yarn commands is that there is a chance that the version that is re-added is a different version than the one that is currently used. If this is what you want - both a move and an upgrade - then go ahead and use the accepted answer.
If not, simply manually edit your package.json to move the line from the devDependencies object to the dependencies object (creating it if necessary). You can go the other direction too.
The lock file doesn't hold any information about if things are prod or dev dependencies, so that doesn't need to be updated. You can do a npm/yarn install afterwards to fix up any flags in the lock files.
The issue of using npm install is that you end up with updated versions. What worked for me is:
Moving them to the intended part (dev, or prod)
Removing them from node_modules folder
Execute npm install
That kept all versions intact.
If your project doesn't have a lockfile or a shrinkwrap file yet, you can simply move the corresponding line in your package.json.
(I'm not recommending not using lockfiles)
I was trying to find an answer for this question for people that uses Yarn, but it hasn't a command for this matter yet.
Although, I believe it is not essential anyway.
Physically (in the Node modules folder) there are no difference between a dependency listed for production and the ones listed for development in your package.json, they'll go to the same place (node_modules).
So, if you need to switch a dependency from devDependencies to dependecies you can go to your package.json and move manually with no need to run a new install or remove the dependency and then install it again with the dev flag.
For me, it's not so great at all to manage the package.json manually, but Yarn is not as advanced as NPM in all functionalities, thus that's a thing to consider.
For Yarn, I moved them manually inside package.json and then ran yarn install. Yarn lock file does not get updated, so I think this is fine.
In my Node.js project, I have a dependency on another local project. Oftentimes, I need to make a small change to the dependency and see how it affects my main project. In order to do this, I have to reinstall my dependency using npm.
I can use npm update to try to update my dependency, but this seems like it will only work if the version number has changed on the dependency. I don't want to have to change the version number on my dependency every time I change a line of code or two to make an experimental change in development.
I can rm -rf node_modules/; npm install to ensure that I get the latest versions of all of my dependencies. Downloading all of my non-local dependencies takes several minutes, breaking up my train of thought.
Is there a way to force npm to reinstall a single dependency, even if that dependency's version number hasn't changed?
When you run npm install, it will install any missing dependencies, so you can combine it with an uninstall like this:
npm uninstall some_module; npm install
With npm 5, uninstalled modules are removed from the package.json, so you should use:
npm uninstall some_module; npm install some_module
On npm v 6.14:
npm install module_name --force --no-save
You get a message stating:
npm WARN using --force I sure hope you know what you are doing.
And then it proceeds to uninstall and reinstall the package.
Note: if you don't specify the --no-save option, npm updates the package version on package.json to the highest version that is compatible with the existing SemVer rule.
If you do not want npm to update the package's version on package.json, keep the --no-save option.
Not the best answer, but just for information, you can run
npm ci
It is the same as npm install, but it will remove the existing node_modules folder, if any, and do a fresh install for all packages. This is useful if the files in node_modules have been changed for some reason and you want to revert them to their original state.