I'm using transitive dependencies that point to git URLs (not on GitHub) and I've found that NPM skips over these dependencies when installing the direct parent dependency.
For example, let's say I have package A, which has a dependency B:
"B": "git+https://username:password#giturl.com/username/B"
Now in a new project, C, I install A.
npm i git+https://username:password#giturl.com/username/A
Now if I look at C's package.json and package-lock.json, I can't find B anywhere! Interestingly enough, if I then install B manually:
npm i git+https://username:password#giturl.com/username/B
then B shows up in C's package.json and package-lock.json, both as a transitive dependency for A and a direct dependency for C.
It's also worth mentioning that this behavior does not occur when using github.com, only when using custom git hosts.
Is there any way around this so I don't have to manually install all my transitive dependencies each time? I'd like NPM to behave like it does for github.com.
I found a workaround, although I'm bewildered as to why NPM decided to skip over these dependencies. I'm assuming it's a bug and have filed a report.
I added the --global-style flag to my npm install command, and NPM will now install my transitive dependencies, except now I don't get the benefits of deduping. --legacy-bundling (disables deduping) also seems to do the trick - which leads me to believe the problem lies in deduping.
The --global-style argument will cause npm to install the package into
your local node_modules folder with the same layout it uses with the
global node_modules folder. Only your direct dependencies will show in
node_modules and everything they depend on will be flattened in their
node_modules folders. This obviously will eliminate some deduping.
Related
My project depends on analytics-node which has a dependency on axios: "^0.16.2".
We have been flagged with npm audit that axios has a vulnerability in it but its fixed in 0.18.1.
However, analytics-node has no release candidate (only beta) which depends on 0.18.1 or higher.
We have tried:
npm audit fix,
npm update axios --depth 2,
npm install axios#0.18.1
The part I am most confused about is why doesnt npm allow us to override analytics-node version of axios given that the version should be compatible.
npm-force-resolutions specifically fixes transitive dependency version resolutions for this exact reason.
It would be nicer if there was a supported way to do this with package-lock.json though.
NPM 8 introduced "overrides" which allows you to override specific transitive dependencies of your direct dependency. For your usecase, you would declare something like below in your package.json.
{
"overrides": {
"analytics-node": {
"axios": "0.18.1"
}
}
}
More details # https://docs.npmjs.com/cli/v8/configuring-npm/package-json#overrides
To bump (i.e. update) a transitive dependency in a package-lock.json file, you can use the npm command line interface (CLI). To update a transitive dependency, you will first need to identify the package that you want to update. Then, you can use the following steps:
Navigate to your project directory in the terminal and run the npm ls command to list all of the dependencies in your project, including transitive dependencies. This will show you the full dependency tree, with each package and its dependencies listed in a hierarchical structure.
Find the package that you want to update in the dependency tree, and note the version number of the transitive dependency that you want to update.
Run the npm install command, followed by the name and version of the package that you want to update, in the following format: npm install <package-name>#<version>. This will update the specified package to the specified version.
Run the npm ls command again to verify that the transitive dependency has been updated. You should see the new version number for the package listed in the dependency tree.
If you want to save the updated dependency in your package-lock.json file, run the npm shrinkwrap or npm update command, depending on which version of npm you are using. This will update the package-lock.json file to reflect the updated transitive dependency.
Note: Bumping a transitive dependency in package-lock.json can potentially cause conflicts or other issues if the updated dependency is not compatible with your project's dependencies. It is recommended to carefully review the dependencies and their versions before updating any packages.
I'm working with continuous integration and discovered the npm ci command.
I can't figure what the advantages are of using this command for my workflow.
Is it faster? Does it make the test harder, okay, and after?
From the official documentation for npm ci:
In short, the main differences between using npm install and npm ci are:
The project must have an existing package-lock.json or npm-shrinkwrap.json.
If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.
npm ci can only install entire projects at a time: individual dependencies cannot be added with this command.
If a node_modules is already present, it will be automatically removed before npm ci begins its install.
It will never write to package.json or any of the package-locks: installs are essentially frozen.
Essentially,
npm install reads package.json to create a list of dependencies and uses package-lock.json to inform which versions of these dependencies to install. If a dependency is not in package-lock.json it will be added by npm install.
npm ci (also known as Clean Install) is meant to be used in automated environments — such as test platforms, continuous integration, and deployment — or, any situation where you want to make sure you're doing a clean install of your dependencies.
It installs dependencies directly from package-lock.json and uses package.json only to validate that there are no mismatched versions. If any dependencies are missing or have incompatible versions, it will throw an error.
Use npm install to add new dependencies, and to update dependencies on a project. Usually, you would use it during development after pulling changes that update the list of dependencies but it may be a good idea to use npm ci in this case.
Use npm ci if you need a deterministic, repeatable build. For example during continuous integration, automated jobs, etc. and when installing dependencies for the first time, instead of npm install.
npm install
Installs a package and all its dependencies.
Dependencies are driven by npm-shrinkwrap.json and package-lock.json (in that order).
without arguments: installs dependencies of a local module.
Can install global packages.
Will install any missing dependencies in node_modules.
It may write to package.json or package-lock.json.
When used with an argument (npm i packagename) it may write to package.json to add or update the dependency.
when used without arguments, (npm i) it may write to package-lock.json to lock down the version of some dependencies if they are not already in this file.
npm ci
Requires at least npm v5.7.1.
Requires package-lock.json or npm-shrinkwrap.json to be present.
Throws an error if dependencies from these two files don't match package.json.
Removes node_modules and install all dependencies at once.
It never writes to package.json or package-lock.json.
Algorithm
While npm ci generates the entire dependency tree from package-lock.json or npm-shrinkwrap.json, npm install updates the contents of node_modules using the following algorithm (source):
load the existing node_modules tree from disk
clone the tree
fetch the package.json and assorted metadata and add it to the clone
walk the clone and add any missing dependencies
dependencies will be added as close to the top as is possible
without breaking any other modules
compare the original tree with the cloned tree and make a list of
actions to take to convert one to the other
execute all of the actions, deepest first
kinds of actions are install, update, remove and move
npm ci will delete any existing node_modules folder and relies on the package-lock.json file to install the specific version of each package. It is significantly faster than npm install because it skips some features. Its clean state install is great for ci/cd pipelines and docker builds! You also use it to install everything all at once and not specific packages.
While everyone else has answered the technical differences none explain in what situations to use both.
You should use them in different situations.
npm install is great for development and in the CI when you want to cache the node_modules directory.
When to use this? You can do this if you are making a package for other people to use (you do NOT include node_modules in such a release). Regarding the caching, be careful, if you plan to support different versions of Node.js remember that node_modules might have to be reinstalled due to differences between the Node.js runtime requirements. If you wish to stick to one version, stick to the latest LTS.
npm ci should be used when you are to test and release a production application (a final product, not to be used by other packages) since it is important that you have the installation be as deterministic as possible, this install will take longer but will ultimately make your application more reliable (you do include node_modules in such a release). Stick with LTS version of Node.js.
npm i and npm ci both utilize the npm cache if it exists, this cache lives normally at ~/.npm.
Also, npm ci respects the package-lock.json file. Unlike npm install, which rewrites the file and always installs new versions.
Bonus: You could mix them depending on how complex you want to make it. On feature branches in git you could cache the node_modules to increase your teams productivity and on the merge request and master branches rely on npm ci for a deterministic outcome.
The documentation you linked had the summary:
In short, the main differences between using npm install and npm ci are:
The project must have an existing package-lock.json or npm-shrinkwrap.json.
If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.
npm ci can only install entire projects at a time: individual dependencies cannot be added with this command.
If a node_modules is already present, it will be automatically removed before npm ci begins its install.
It will never write to package.json or any of the package-locks: installs are essentially frozen.
The commands are very similar in functionality however the difference is in the approach taken to install the dependencies specified in your package.json and package-lock.json files.
npm ci performs a clean install of all the dependencies of your app whereas npm install may skip some installations if they already exist on the system. A problem may arise if the version already installed on the system isn't the one your package.json intended to install i.e. the installed version is different from the 'required' version.
Other differences would be that npm ci never touches your package*.json files. It will stop installation and show an error if the dependency versions do not match in the package.json and package-lock.json files.
You can read a much better explanation from the official docs here.
Additionally, you may want to read about package locks here.
It is worth having in mind that light node docker images like alpine do not have Python installed which is a dependency of node-gyp which is used by npm ci.
I think it's a bit opinionated that in order to have npm ci working you need to install Python as dependency in your build.
More info here Docker and npm - gyp ERR! not ok
npm ci - install exactly what is listed in package-lock.json
npm install - without changing any versions in package.json, use package.json to create/update package-lock.json, then install exactly what is listed in package-lock.json
npm update - update package.json packages to latest versions, then use package.json to create/update package-lock.json, then install exactly what is listed in package-lock.json
Or said a different way, npm ci changes 0 package files, npm install changes 1 package file, and npm update changes 2 package files.
It does a clean install, use it in situations where you would delete node_modules and re-run npm i.
I have no idea why some people think it's short for "continuous integration". There is an npm install command that can be run as npm i and an npm clean-install command that can be run as npm ci.
npm install is the command used to install the dependencies listed in a project's package.json file, while npm ci is a command that installs dependencies from a package-lock.json or npm-shrinkwrap.json file. The npm ci command is typically used in continuous integration (CI) environments, where the package-lock.json or npm-shrinkwrap.json file is checked into version control and should not be modified. Because npm ci installs dependencies from a locked file, it is a faster and more reliable way to install dependencies than npm install, which could install different versions of dependencies based on the state of the package.json file.
Does it make sense to put any modules into package.json dependencies when I use webpack?
When I want to develope a package, I use git clone <url> then npm install, then npm installs all dependencies and devDependencies from package.json file and it makes sense.
When I'm end-user and I just want to install some package into my node_modules to use it in my project, I run npm install package-name, then npm installs package-name with only its dependencies, and it makes sense too.
But does it make sense to put any modules into dependencies when I use webpack? The webpack will bundle all dependencies into eg. bundle.js, so, for me, there is no need to install dependencies then (while they're included into bundle.js file).
Let's assume that I put all neccessary modules into devDependencies (keep the dependencies object empty) for my project: my-project, bundle it with webpack and publish:
the developer-user will use git clone <url to my_project>, then run npm install, then npm will install devDependencies from package.json (and ommit empty dependencies object), then it is ready to develope.
the end-user will use npm install my-project, then npm will install my-project, do not install devDependencies (because this is for production) and do not install dependencies (because dependencies object in package.json remain empty). Putting anything into dependencies would double the dependencies: both the dependencies would be installed, and the same dependencies would be accessible in the bundle.js file.
Am I right?
You are correct that there may be no dependencies once it's been transpiled with webpack. However, some packages are multi-purpose and may be used in multiple ways, so in some circumstances dependencies may still be required.
If you look at the package.json specification, there are two possible entry points, 'main' and 'browser'. There is also the proposed 'module' entry point. It is currently under discussion about how to handle these in webpack and users seem to want webpack to prioritize them as module > browser > main, however browser is currently used by webpack first.
The idea behind prioritizing them in the order module > browser > main is presumably that browsers could use the pre-transpiled stuff directly in "browser", whereas another project calling require() or include() on your package would use non-transpiled code from the "module" entry. The "module" entry code could contain modern JavaScript with new features and the project/package requiring it could then transpile it to their own specifications, using "browserslist" for example.
I found this question because I was wondering the same thing...
My package.json only has express, mongodb, and pug listed as dependencies. When I delete my node_modules folder and run npm install in the root of my app, it installs 51 dependencies (shown below). not sure what is going on...
npm v3 dependency resolution works that way and might install "some secondary dependencies (dependencies of dependencies) in a flat way".
You can also refer to this stackoverflow question.
I just upgraded to npm version 3 and noticed one of the biggest changes it made is that it enforces a flat dependency tree.
Your dependencies will now be installed maximally flat. Insofar as is possible, all of your dependencies, and their dependencies, and THEIR dependencies will be installed in your project's node_modules folder with no nesting. You'll only see modules nested underneath one another when two (or more) modules have conflicting dependencies.
So for example if package A is dependent on package B, when you npm install A you will get this file structure:
--- root/
|--- node_modules/
|--- A/
|--- B/
instead of the old file structure from version 2 or lower:
--- root/
|--- node_modules/
|--- A/
|--- node_modules/
|--- B/
The first (and I’m sure not the last) problem I ran into was this:
Package A isn’t aware of npm v3’s behavior and is dependent on package B. But A assumes the old (v2) file structure because it has node_modules/B in its code, instead of the proper ../node_modules/B. Now the code from A won’t compile because it’s looking for B/ in the wrong directory.
If I don’t feel like nagging the developer to fix the code and waiting for an update of A, I wonder if there’s a way I can set an option that will force npm to install A’s dependencies inside its own node_modules folder, the same way npm v2 would have done it.
Have you tried --legacy-bundling for npm install?
https://docs.npmjs.com/cli/install
The --legacy-bundling argument will cause npm to install the package such that versions of npm prior to 1.4, such as the one included with node 0.8, can install the package. This eliminates all automatic deduping.
This is what happens when I don’t get enough sleep. The obvious solution somehow escaped me.
$ cd node_modules/A/
$ npm install
$ cd ../../
I've just ran into this issue and found an alternative to fix this issue. Since Node.js comes with NPM, the only way that I could think of was to downgrade to Node.js 0.10 version, which comes with an older version on NPM, which will install dependencies in the old fashion way. To downgrade I used NVM and ran nvm use 0.10.0. You can switch back and forth between npm versions by following this trick. Hope it helps!