Context
I have a .NET project and we use some npm packages for the UI. I use a pre-built check if the folder node_modules does not exist, run the command npm install.
When committing an update to package.json, npm install will not trigger for other people who are updating the repository. Because node_modules exists on their machine which leads to errors like Can't resolve....
Question(s)
Is the folder check obsolete? Is npm install smart enough to download only the necessary things and not all the dependencies? Or do i need a hash check on package.json?
There's no harm in running npm install multiple times as it will simply not do any operation in case there's nothing to do.
You don't need to check for the node_modules folder. npm will download and update any dependencies that are missing.
It's also important to run npm install since other machines may be running different systems and the dependencies may need to be compiled differently.
You can hash check package.json and/or package-lock.json for caching purposes, but it's not really necessary.
I am on different servers and run npm install
One server is ok. Another failed showing below error:
no matching version found for es-abstract#1.14.0
I tried npm ci. Same deal.
Then I did npm install es-abstract-1.14.0.tgz
But the size node_modules/ are different. I am using du -shc --apparent-size node_modules/ to ignore the sparse files within the folder. Why?
Could I accept that this is the network issue of the failed server？ And just keep doing this? I mean, is the installation still going to be ok in this way?
At the time that this answer was written, there are 33 release versions for es-abstract on github, but only 32 release versions are listed on its npm registry. The missing version in the registry is 1.14.0.
Perhaps on one of your servers, you had this package cached (maybe it was previously listed on the NPM registry?, or maybe from downloading it from github?), and on the other server you did not have this package cached.
I had this same error message when trying to npm install a project from github.
In my situation, the es-abstract package was not explicitly listed in the package.json file, but it was a dependency of another package. Therefore I explicitly added it with the next highest version listed on the registry, and it worked.
e.g.
"dependencies": {
"es-abstract": "1.14.1",
...
Just a guess, but maybe 1.14.0 used to be listed on the registry, but now it's not?
I'm working with continuous integration and discovered the npm ci command.
I can't figure what the advantages are of using this command for my workflow.
Is it faster? Does it make the test harder, okay, and after?
From the official documentation for npm ci:
In short, the main differences between using npm install and npm ci are:
The project must have an existing package-lock.json or npm-shrinkwrap.json.
If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.
npm ci can only install entire projects at a time: individual dependencies cannot be added with this command.
If a node_modules is already present, it will be automatically removed before npm ci begins its install.
It will never write to package.json or any of the package-locks: installs are essentially frozen.
Essentially,
npm install reads package.json to create a list of dependencies and uses package-lock.json to inform which versions of these dependencies to install. If a dependency is not in package-lock.json it will be added by npm install.
npm ci (also known as Clean Install) is meant to be used in automated environments — such as test platforms, continuous integration, and deployment — or, any situation where you want to make sure you're doing a clean install of your dependencies.
It installs dependencies directly from package-lock.json and uses package.json only to validate that there are no mismatched versions. If any dependencies are missing or have incompatible versions, it will throw an error.
Use npm install to add new dependencies, and to update dependencies on a project. Usually, you would use it during development after pulling changes that update the list of dependencies but it may be a good idea to use npm ci in this case.
Use npm ci if you need a deterministic, repeatable build. For example during continuous integration, automated jobs, etc. and when installing dependencies for the first time, instead of npm install.
npm install
Installs a package and all its dependencies.
Dependencies are driven by npm-shrinkwrap.json and package-lock.json (in that order).
without arguments: installs dependencies of a local module.
Can install global packages.
Will install any missing dependencies in node_modules.
It may write to package.json or package-lock.json.
When used with an argument (npm i packagename) it may write to package.json to add or update the dependency.
when used without arguments, (npm i) it may write to package-lock.json to lock down the version of some dependencies if they are not already in this file.
npm ci
Requires at least npm v5.7.1.
Requires package-lock.json or npm-shrinkwrap.json to be present.
Throws an error if dependencies from these two files don't match package.json.
Removes node_modules and install all dependencies at once.
It never writes to package.json or package-lock.json.
Algorithm
While npm ci generates the entire dependency tree from package-lock.json or npm-shrinkwrap.json, npm install updates the contents of node_modules using the following algorithm (source):
load the existing node_modules tree from disk
clone the tree
fetch the package.json and assorted metadata and add it to the clone
walk the clone and add any missing dependencies
dependencies will be added as close to the top as is possible
without breaking any other modules
compare the original tree with the cloned tree and make a list of
actions to take to convert one to the other
execute all of the actions, deepest first
kinds of actions are install, update, remove and move
npm ci will delete any existing node_modules folder and relies on the package-lock.json file to install the specific version of each package. It is significantly faster than npm install because it skips some features. Its clean state install is great for ci/cd pipelines and docker builds! You also use it to install everything all at once and not specific packages.
While everyone else has answered the technical differences none explain in what situations to use both.
You should use them in different situations.
npm install is great for development and in the CI when you want to cache the node_modules directory.
When to use this? You can do this if you are making a package for other people to use (you do NOT include node_modules in such a release). Regarding the caching, be careful, if you plan to support different versions of Node.js remember that node_modules might have to be reinstalled due to differences between the Node.js runtime requirements. If you wish to stick to one version, stick to the latest LTS.
npm ci should be used when you are to test and release a production application (a final product, not to be used by other packages) since it is important that you have the installation be as deterministic as possible, this install will take longer but will ultimately make your application more reliable (you do include node_modules in such a release). Stick with LTS version of Node.js.
npm i and npm ci both utilize the npm cache if it exists, this cache lives normally at ~/.npm.
Also, npm ci respects the package-lock.json file. Unlike npm install, which rewrites the file and always installs new versions.
Bonus: You could mix them depending on how complex you want to make it. On feature branches in git you could cache the node_modules to increase your teams productivity and on the merge request and master branches rely on npm ci for a deterministic outcome.
The documentation you linked had the summary:
In short, the main differences between using npm install and npm ci are:
The project must have an existing package-lock.json or npm-shrinkwrap.json.
If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.
npm ci can only install entire projects at a time: individual dependencies cannot be added with this command.
If a node_modules is already present, it will be automatically removed before npm ci begins its install.
It will never write to package.json or any of the package-locks: installs are essentially frozen.
The commands are very similar in functionality however the difference is in the approach taken to install the dependencies specified in your package.json and package-lock.json files.
npm ci performs a clean install of all the dependencies of your app whereas npm install may skip some installations if they already exist on the system. A problem may arise if the version already installed on the system isn't the one your package.json intended to install i.e. the installed version is different from the 'required' version.
Other differences would be that npm ci never touches your package*.json files. It will stop installation and show an error if the dependency versions do not match in the package.json and package-lock.json files.
You can read a much better explanation from the official docs here.
Additionally, you may want to read about package locks here.
It is worth having in mind that light node docker images like alpine do not have Python installed which is a dependency of node-gyp which is used by npm ci.
I think it's a bit opinionated that in order to have npm ci working you need to install Python as dependency in your build.
More info here Docker and npm - gyp ERR! not ok
npm ci - install exactly what is listed in package-lock.json
npm install - without changing any versions in package.json, use package.json to create/update package-lock.json, then install exactly what is listed in package-lock.json
npm update - update package.json packages to latest versions, then use package.json to create/update package-lock.json, then install exactly what is listed in package-lock.json
Or said a different way, npm ci changes 0 package files, npm install changes 1 package file, and npm update changes 2 package files.
It does a clean install, use it in situations where you would delete node_modules and re-run npm i.
I have no idea why some people think it's short for "continuous integration". There is an npm install command that can be run as npm i and an npm clean-install command that can be run as npm ci.
npm install is the command used to install the dependencies listed in a project's package.json file, while npm ci is a command that installs dependencies from a package-lock.json or npm-shrinkwrap.json file. The npm ci command is typically used in continuous integration (CI) environments, where the package-lock.json or npm-shrinkwrap.json file is checked into version control and should not be modified. Because npm ci installs dependencies from a locked file, it is a faster and more reliable way to install dependencies than npm install, which could install different versions of dependencies based on the state of the package.json file.
Can someone explain to me what it means to --save-dev and how this impacts distribution and how NPM is aware of what you're trying to do?
First, see the answer to this question, What's the difference between dependencies, devDependencies and peerDependencies.
That will explain a TON.
Second, npm will install devDependencies by default unless one of two things is true, in which case devDependencies will be skipped. These things are:
You explicitly tell npm it's production with npm install --production
You set an environment variable that npm checks, NODE_ENVIRONMENT=production
In general, if you are distributing to something like Heroku, they will have the production flag set and your devDependencies will not be installed. So only install things with the --save-dev or -D flag (both do the same thing) if it is a module used for development, such as tests/mocks/scaffolding/etc.
--save-dev is useful for dependencies such as unit testing libraries. These dependencies are not required by the application to run in production; therefore if you published your package, consumers of your package would not get these dev dependencies in their node_modules folder.
NPM doesn't necessarily know that your in dev mode, it's just a package manager that allows you to install packages into your working directory and publish your own package from said directory.
I tried to maintain package.json with the list of node modules dependencies. when I call npm install it installs the node modules.and generates a folder for it in my app. I call the npm shrinkwrap. But this generates the dependency on the local node module
"dependencies": {
"async": {
"version": "0.2.5",
"from": "async#0.2.5",
"resolved": "https://registry.npmjs.org/async/-/async-0.2.5.tgz"
},
when I upload the app to the appfog server it can install from the npm-shrinkwrap.json. So Ideally I want to remove the node modules folder and just pass the shrinkwrap.json file. But it has this "from". I had in the past generated the shrinkwrap & it didn't have the "from" field in there.
How to generate without "from"/ can I just get a shrinkwrap file from package.json. so my app will be leaner. I can maintain all the node module globally.
Thanks
I'm a bit confused by your question.
Shrinkwrap does not install, package, upload or do anything to your dependencies.
All it does is scan your installed node_modules and record the versions (recursively) into a file. Invoking npm install after that file is defined becomes repeatable, which is a principle of software engineering.
"from" was introduced a few months back. The npm shrinkwrap command seems to set it to the URL from which a module was installed. This is probably for portability. npm install takes a module name, consults a registry (whose URL is configurable as an npm config setting) and installs it. I could take the same package.json and npm-shrinkwrap.json, put them on another machine and theoretically get a different result if that machine's npm config settings point it to a different registry. Therefore, embedding the resolved URL in the shrinkwrap file adds an additional level of repeatability to npm install
See the npm config man page for details of setting the registry parameters.
According to npm issue 3145 on github, the "from" setting is known to cause backwards-compatibility issues with pre-1.2.x npm systems. Upgrading is the only resolution.
https://github.com/isaacs/npm/issues/3145
I think that you are looking for shrinkpack: https://www.npmjs.com/package/shrinkpack
from the doc:
Shrinkpack complements the npm shrinkwrap command by maintaining a node_shrinkwrap directory in your project, containing the exact same tarballs that npm install downloads from https://registry.npmjs.org.
The rest of the npm install process is exactly the same. The only difference is that no network activity is necessary when installing and building your project. The node_shrinkwrap directory can be ignored in your editor (much like is done with the node_modules directory) but is instead checked into source control.