Yarn: use yarn.lock of dependencies - npm

I use yarn to install packages directly from the companies' GitLab:
yarn add git+ssh://<user>#<host>:<repo>
For first-level dependencies, I use yarn --pure-lockfile to reconstruct my node_modules according to the yarn.lock.
However, for second level dependencies yarn seems to always install the latest version.
So let's say I depend on A which is tested with a specific version of B. In A's package.json I don't specify the version, but it is contained in the yarn.lock.
When I now install package A yarn will get the latest version of B despite the entry in A/yarn.lock
I know that I might resolve this by passing a specific version in A/package.json (at least I think).
But is there an option to tell yarn to look at the yarn.lock of dependencies?

TLDR:
When you install dependencies in your
application, only your own yarn.lock file is respected. Lockfiles
within your dependencies will be ignored. Reference
Let's get some things cleared first:
--pure-lockfile is same as normal yarn install except that it won't generate a yarn.lock file or update one if present.
Yarn always reads from the yarn.lock by default for resolving dependencies while installing unless supplied with --no-lockfile. So, there is no need to tell it to read from yarn.lock.
What is yarn.lock used for?
yarn.lock is used for resolving what version should be fetched
given the semver version of a module in package.json. It is not used to determine what semver version should a module be resolved to. That is simply not its use-case.
As mentioned in yarn DOCS: In order to get consistent
installs across machines, Yarn needs more information than the
dependencies you configure in your package json.. Yarn needs to store
exactly which versions of each dependency were installed.
To do this Yarn uses a yarn.lock file in the root of your project.
So, for resolving semver version of a dependency, yarn always depends on package.json. For a given semver version, yarn checks the yarn.lock file to see what version should it fetch. This is what makes yarn Deterministic (Same tecknique is used by npm which uses npm-shrinkwrap.json).
Example: Semver Versions like ^1.2.4 can resolve to any version number which is >= 1.2.3 and < 2.0.0. Without yarn, npm would install 1.2.4 in one machine but 1.9.9 in some other machine, depending on the latest version present at the time of install. This is the problem that yarn solves using yarn.lock.
The semver version is determined by the
package.json file. The yarn.lock file is only a lookup for which
version and commit hash to install for the given semver version number.
How does yarn resolve version of a module given its semver version?
Suppose currently our yarn.lock file looks like this:
bluebird#2.9.6:
version "2.9.6"
resolved "https://<...>/bluebird-2.9.6.tgz#1fc3a6b1685267dc121b5ec89b32ce069d81ab7d"
bluebird#^2.9.30:
version "2.11.0"
resolved "https://<...>/bluebird-2.11.0.tgz#534b9033c022c9579c56ba3b3e5a5caafbb650e1"
...
myModule#5.1.0:
version "5.1.0"
resolved "https://<...>/moduleA-5.1.0.tgz#ce97130858add59d616ee80675383b0c127290a0"
dependencies:
bluebird "^1.0.0"
If package.json has bluebird: "^2.9.30", yarn looks for an entry bluebird#^2.9.30 in the lockfile. It is present and hence is resolved to version=2.11.0.
If package.json has bluebird: "^2.9.0", yarn looks for an entry bluebird#^2.9.0 in the lockfile. It is not present. Suppose the latest stable version which satisfies semver criteria is 2.13.0, then yarn adds an entry for bluebird#^2.9.0, resolved to 2.13.0. While resolving version for a given semver version of bluebird, it does not matter what entry is present for bluebird in moduleA's dependencies in the lockfile.
Semver Version is not affected by what entries are
present in the dependencies map for a module in yarn.lock file.
So, if package.json has bluebird: "", yarn looks for an entry bluebird# in the lockfile but is unable to find it. Hence, it resolves bluebird: "" to the latest version, suppose 3.5.0. Now, yarn will add an entry for bluebird# resolved to 3.5.0.
bluebird#:
version "3.5.0"
resolved "https://<...>/bluebird-3.5.0.tgz#d9551f9de98f1fcda1e683d17ee91a0602ee2eb9"
Form now on whenever yarn encounters {bluebird: ""}, it will find an entry for bluebird# in the lockfile and hence will always resolve it to 3.5.0.
Solution to your problem
To resolve B: "" to version say 1.0.0, you need to have an entry for B# in yarn.lock resolved to 1.0.0. Once, yarn.lock has an entry for B#, all the successive installs will always fetch version 1.0.0 for B="".
Following are the steps you need to take to accomplish the same:
Approach 1 (Recommended)
If you want B to resolve to latest version:
Add B:"" in A's package.json
Run yarn install. This will add an entry for B# resolved to latest version.
Push yarn.lock file.
Form now on, whoever runs yarn install will get the same version.
Approach 2
If you want B to have an older version: (Highly Not Recommended)
Add B: 1.0.0 in A's package.json.
Run yarn install. This will add an entry B#1.0.0 in the lockfile.
Add B# alongside B#1.0.0 in yarn.lock. B#, B#1.0.0: ...
Change B's version to "" in A's package.json.
Push yarn.lock file.
Form now on, whoever runs yarn install will get the B's version as 1.0.0.
This approach is highly dangerous as you can break something easily. Your yarn.lock file should always be managed by yarn.
Approach 3 (Recommended)
If you want B to stay at 1.0.0
Fix B's version to 1.0.0 in A's package.json.
Run yarn install. This will add an entry B#1.0.0 in the lockfile.
Push yarn.lock file
Form now on, whoever runs yarn install will get the B's version as 1.0.0.
Edit: Using the yarn.lock file present in the dependencies
If you check this doc:, they have clearly mentioned that yarn will use only the top level yarn.lock file and ignore the lock files present in the dependencies.
There is currently no way of locking down second level dependencies using yarn.lock present in them. I don’t see any need for it. In fact the creators of yarn explain here why that is the case. The reasons being:
The versions to be installed for second level dependencies can be
captured well by the top-level yarn.lock file, as I have explained
above.
You would never be able to update the versions of sub-dependencies in your own application when using them directly because they would be locked by other yarn.lock files. You can verify this point by my explanation of how yarn resolves dependencies.
Yarn would never be able to fold (de-duplicate) dependencies so that compatible version ranges only install a single version.
Also, as in your use-case, if A has a dependency B which works only with version 1.0.0, A’s package.json should have version mentioned for B as 1.0.0 and not “”. You can always fix your top-level yarn.lock to add an entry for B# resolved to 1.0.0 but it is not recommended to manually fix a yarn.lock file as I have mentioned above.
Hope this helped! Please ping me in the comments for any doubts.

Related

How to have `pnpm install` install everything exactly to the specs of the pnpm-lock file?

If you connect a github project to a product like cloudflare pages or Vercel, commits to the remote repo trigger new builds. These builds will run the appropriate install and build commands. I haven't updated a site in months, but major changes have come to dependencies being used and it's causing me so many headaches to try and go through one-by-one and address each and every issue that has surfaced. I'm using pnpm, is there anyway I can have pnpm install look at the existing pnpm-lock.yaml so I can eventually build a project that is entirely the same as a previous build I had 6 months ago? I just want to edit some text on my site and not have to make all these updates. I tried "freezing" the versions of all my dependencies and dev dependencies in package.json by removing instances of ^ to match what I see in my lock file, but that didn't work.
Package Management with PNPM
See also why-does-npm-install-rewrite-package-lock-json
Semver
The semver specification explains how to use semantic versioning though you can probably skip to the npm docs.
As you probably know the numbers are in the form major.minor.patch. If you don't mind which patch release you have as long as it is the specified major and minor version you can use the ~ prefix. Similarly, to allow any minor version use ^.
Walkthrough
Inital Setup
pnpm init
pnpm add express
The package.json will contain (at time of writing):
"express": `"^4.18.2"`
A pnpm-lock.yaml is also created:
specifiers:
express: ^4.18.2
dependencies:
express: 4.18.2
express -> '.pnpm/express#4.18.2/node_modules/express'/
Using pnpm install
Giving it a first run without changing anything produces:
$ pnpm install
Lockfile is up to date, resolution step is skipped
Already up to date
Done in 653ms
Now if I change package.json to be exactly v4.16.0 we shall see an update to pnpm-lock.yaml
specifiers:
express: 4.16.0
dependencies:
express: 4.16.0
Adding the patch wildcard ~4.16.0 and running pnpm install again gives:
specifiers:
express: ~4.16.0
dependencies:
express: 4.16.0
Note that the install version did not change. If I delete the node_modules/ directory and reinstall, still no change.
Ok, now try updating the minor version in package.json to ~4.17.0.
specifiers:
express: ~4.17.0
dependencies:
express: 4.17.3
This time it did update the dependency and installed the latest patch version but did install the exact major and minor version. If you think about what the ~ means then this is expected.
The specifiers section in the lock file is just what we specify as the dependency in the package.json file. The dependencies section in the lock file should reflect the version that is installed, or will be installed.
If I delete the node_modules/ folder and pnpm install again then we still have 4.17.3.
Explanation
What confuses a lot of people about pnpm install/npm install is how the lock-file works with the semver specifier:
The installed version listed as a dependency in the lockfile must be compatible with the version specified in the package file.
If it is compatible, no changes will be made.
If it is incompatible, then the latest compatible version will be installed.
Perhaps because sometimes it seems to install the latest version, and not othertimes, the behaviour is not clear. To state this again, changes will only be made when there is an incompatibility between the packge version and lockfile version. The lockfile dependency never has the ~ or ^ wildcards because only one version is actually installed and that's what the lockfile is supposed to track.
Using --frozen-lockfile in a CI environment
The docs for pnpm install describe how the install will fail if the lockfile is out of sync or needs updating.
Changing the package.json back to ~4.16.0 and then doing the install:
$ pnpm install --frozen-lockfile
Lockfile is up to date, resolution step is skipped
 ERR_PNPM_OUTDATED_LOCKFILE  Cannot install with "frozen-lockfile" because pnpm-lock.yaml is not up to date with package.json
Note that in CI environments this setting is true by default. If you still need to run install in such cases, use "pnpm install --no-frozen-lockfile"
In fact, even if I specify the installed version exactly 4.17.3, because it differs to the specifier ~4.17.0, then it will err. The package.json and pnpm-lock.yaml are out of sync even though the version are compatible.
Finally I will make our package compatible with the latest version that was installed with the first pnpm add express command. To do this I use the minor version wildcard ^4.0.0 and unfreeze the lockfile with pnpm install --no-frozen-lockfile.
specifiers:
express: ^4.0.0
dependencies:
express: 4.17.3
While the specifier is updated to match the package file, the version is not chaged; it is compatible.
Running pnpm install --frozen-lockfile will work again, but not update the installed version.
Conclusion
In a normal environment the lockfile will determine the exact version installed unless it is not compatible with the package file, in which case it will install the lastest version specified by the package file.
In a CI environment the lockfile will not by default be updated and will need to be compatible with the package file for installs to occur.
If you want the latest version specified pnpm update will do the update to the lastest compatible version given in the package file.
Disclaimer
I've tested out everything here but it is complex and I have limited experience using pnpm in a real CI environment.
before deploying your project delete pnpm-lock.yaml

version in package-lock file of a dependency is being ignored

I am importing a dependency in my package.json. That dependency has it's own package-lock.json file in which a the versions of its own dependencies are written. However, when I do my npm install ... --save, the version of the dependency's dependency is not the right one, npm is picking up the latest version. It seems to be ignoring the lock file that belongs to the dependency.
So my dependency contains this in it's package.json file: "ccxt": "^1.52.41". And in it's package-lock.json there is ... "ccxt": { "version": "1.52.41"....
When I add this dependency to my project, the version of ccxt that gets installed (the version in my lock file) is 1.52.94.
What am I doing wrong?
Using npm v6.14.13
Ok, so the package-lock files that belong to your dependencies are ignored by npm. This is by "design" apparently. So at npm install time, you will essentially get a totally random bunch of dependencies installed and added to your package-lock file. From that point on those dependencies are fixed by your lock file until you manually update them using the npm command line.
This means that if you are unlucky and install at a 'bad' time, like I did, then you will pick up some completely untested code that could be broken and/or incompatible with its parent project or your project.

What is the correct use of package.lock.json file, when exactly it works

What is the exact use of package.lock.json file? I have read about it but it's confusing.
Let's say I do npm install so that it will create a package.lock.json file, and the next time when I do npm install, will it retain installation of the same packages as it was mentioned in the package.lock.json file?
Let's say I have a package version defined as "^1.0.0" in my package.json file and a new version becomes available in the npm registry. Will it update the package when I do npm install, or will it keep the same as what was there in the package.json?
Is there any easy way to update the package.json file with the exact no of package version without manually updating it one by one? I want to keep maintain the version of the package the same so that next time anybody does npm install, they should get exactly the same package. It's need for a release branch
Actually I need what was the original definition of package.lock.json file but I found inconsistency in its behavior.
I tried using NPM CI but it fails with error node_gyp needs python https://github.com/nodejs/node-gyp/issues/1977
What is the exact use of package.lock.json file?
Ans: The file is used to describe the exact tree that was generated initially, that is the version of the dependency that was used initially to install (NOTE: It will always be modified if package.json or node_modules are modified by npm)
source: https://docs.npmjs.com/configuring-npm/package-lock-json.html
Now coming to your specific questions :
Let's say I do npm install so that it will create a package.lock.json file, and the next time when I do npm install, will it retain installation of the same packages as it was mentioned in the package.lock.json file?
The simple answer here is: Yes
When you do npm install initially, you specify the package name. npm will create node_modules folder and package-lock.json file and the entry of the package with the version that was used to install will be added. next time when you do npm install, without specifying the package it will not update any of your package even with the caret (^) symbol because this time npm will look at the package-lock.json file (as it is present there) and it will install the same version that is specified in the (package-lock.json) file.
Let's say I have a package version defined as "^1.0.0" in my package.json file and a new version becomes available in the npm registry. Will it update the package when I do npm install, or will it keep the same as what was there in the package.json?
As explained in first question, it will not, if the package-lock.json is present in the directory. Updating of the package will only happen if package-lock.json file and node_modules folder are not present in the directory.
Is there any easy way to update the package.json file with the exact no of package version without manually updating it one by one? I want to keep maintain the version of the package the same so that next time anybody does npm install, they should get exactly the same package. It's need for a release branch
It is a good practice to use npm ci, but you can also use npm update, this will update the dependencies to its latest minor version, and your package.json as well as package-lock.json file will also be updated
For the production, its preferred way to use the same version of dependencies that was used initially. In this case it is better to have package-lock.json file, so in case of dockerizing application when you do npm install it will install the dependencies with versions that are used in package-lock.json

How to override package.json "latest" dist-tag with version from package-lock.json?

I want to use latest distribution tag in my package.json for internal packages. This allows me to always get their latest versions when I npm install in local environment, without updating all external 3rd parties.
The issue comes when I'm hotfixing deployed verion:
For hotfix purpose I generate and save package-lock.json for each deployed version of the application.
But when I npm install during hotfix preparation, there is a conflict between versions of internal package in package.json and package-lock.json: package-lock.json points to version that was used in deployed application, but package.json point to latest distribution tag, which itself points to later version.
Since version specified in package-lock.json doesn't suit to version range specified in package.json (which is very specific - only the latest version will suit), npm install ignores package-lock.json and installs the latest version.
I searched through documentation and internet and didn't find any existing solution for the issue:
I didn't find any npm install flag that would treat package-lock.json versions with higher priority than distribution tag in package.json
I dind't find any tool that would reconstruct package.json from package-lock.json, or at least replace aliases (distribution tags) in package.json with specific versions from package-lock.json.
Is there any solution for my issue (besides writing a tool that will implement last approach)?
Sandbox:
https://github.com/maxlk/npm-lock-version-should-override-latest (clone and run npm install or its alternative)
I found a solution - to use npm ci instead of npm install.
It doesn't exit with error, despite the claim in the documentation: https://docs.npmjs.com/cli/ci
If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.

Update npm package with fixed dependency from command line

I have an npm package with a fixed version that has an update.
Example package.json extract:
devDependencies: {
"someFixedVersionPackage": "1.0.0", //1.1.0 is latest
"anotherFixedVersionPackage": "2.3.2", //2.3.4 is latest
}
Does an npm command exist which installs the latest version of that package and updates the package.json, preferably all packages at once?
To be clear, I want the package.json snippet above to be updated to this, in addition to the packages themselves being updated:
devDependencies: {
"someFixedVersionPackage": "1.1.0", //latest
"anotherFixedVersionPackage": "2.3.4", //latest
}
Thank you.
Why doesn't npm update work here?
As per the documentation on npm update:
This command will update all the packages listed to the latest version (specified by the tag config), respecting semver.
It will also install missing packages. As with all commands that install packages, the --dev flag will cause devDependencies to be processed as well.
Since your packages are defined with a fixed version, the update sub-command will not update those to respect semantic versioning. Therefore, it will only automatically update your packages if you specify a greater version range for each package. Note that it is actually typical in an npm project to specify a loose range version; one that is meant to avoid breaking changes but still leaves room for improvements and fixes.
Still, why shouldn't I fix dependency versions in my package.json?
But they are fixed because I wanted them so. After testing newer versions, I want to update them via command line as were created.
Having a list of dependencies with a fixed version does not mean that the dependencies installed will always be the same, because the dependencies of your dependencies will most likely also be defined with a version range. In order to keep track of a list of tested version-tagged dependencies, npm provides another mechanism: package locks.
Before version 5 of npm, you can create a "npm-shrinkwrap.json" file with the shrinkwrap command:
npm shrinkwrap
This command locks down the versions of a package's dependencies so that you can control exactly which versions of each dependency will be used when your package is installed.
Since npm 5, a "package-lock.json" is automatically generated when an npm operation modifies the "node_modules" tree or "package.json".
Rather than modifying package.json, either one of these package locks will override the default behaviour of npm install, installing dependencies with the versions specified by the lock, right when they were created or manually updated. With that out of the way, your dependencies can now be expanded without the risk of dependents installing untested package versions.
Shrinkwraps are used for publishing packages. To shrinkwrap a package:
Run npm install in the package root to install the current versions of all dependencies.
Validate that the package works as expected with these versions.
Run npm shrinkwrap, add npm-shrinkwrap.json to git, and publish your package.
At this point, dependency versions can be loosened in your package.json (this will hopefully be done only once every major dependency update), so that later on they can be updated at will with npm update:
"devDependencies": {
"someFixedVersionPackage": "^1.0.0",
"anotherFixedVersionPackage": "^2.3.2",
}
The package-lock.json file can be used instead of a shrinkwrap, and is more suitable for reproducing a development environment. It should also be committed to the repository.
So how do I update my dependencies?
Calling npm update will do what's mentioned above: update dependencies while respecting semantic versioning. To add or upgrade a dependency in a package:
Run npm install in the package root to install the current versions of all dependencies.
Add or update dependencies. npm install --save each new or updated package individually to update the package.json, as well as the existing package locks ("package-lock.json" and "npm-shrinkwrap.json"). Note that they must be explicitly named in order to be installed: running npm install with no arguments will merely reproduce the locked dependencies.
Validate that the package works as expected with the new dependencies.
Commit the new package locks.
Moreover, here are a few tips for a smooth transition from a project with fixed dependencies:
If you haven't done so, expand the version range by adding a tilde (~) before the version specifier, or a caret (^). npm update will then attempt to install all patch revisions and minor revisions, respectively (major version 0 is a corner-case, see the documentation). For instance, "^1.0.0" can now be updated to "^1.1.0", and "~2.3.2" can be updated to "~2.3.4". Adding the --save or --save-dev flags will also update the "package.json" with the installed version (while keeping the previous range specifiers).
Run npm outdated to check which packages are outdated. Entries in red will be updated automatically with npm update. Other entries will require a manual intervention.
For packages with major version bumps, install that package with a version specification (e.g. npm install browserify#11.2.0 --save-dev). Further issues that may arise with the update will have to be handled manually. It usually helps to read the news feed or the release history on that package to further understand what has changed from previous versions.
This is not simple enough, is there another way to do this?
Before continuing, it is always worth mentioning that packages have a SemVer-compliant version definition for a reason. One should avoid blindly installing the latest version of every single package. Although such a full update can be done and tools are available for that, some caution is advised. For instance, you would not want to install React 15 if the remaining React components and libraries are not compatible with react#15.x.x. See also npm's blog post: Why use SemVer?
I'll take my chances. What other tools are there?
To name a few:
npm-check-updates will do what was initially asked in the question: install and update the versions of all dependencies, regardless of the given range constraint. This would be the least recommended tool for the job, however.
updtr will update dependencies one by one and roll back to the previous version if the project's tests fail, which may save time in projects with good test coverage.
npm-check provides an interactive command-line interface, which allows you to easily select which packages to update.
Is this any different with npm 5?
Since major version 5, npm will automatically create a "package-lock.json", which will fill the role of specifying the dependency tree when a shrinkwrap does not exist. A more detailed description can be found in the package-locks documentation. In general, npm-shrinkwrap.json is meant to be used when publishing, whereas package-lock.json is to be used in development. This is why you should also commit "package-lock.json" to the repository.
What about with Yarn?
Yarn, an npm-compatible dependency manager, creates a lock file automatically on use, which behaves similarly to the npm shrinkwrap. Calling yarn upgrade «package» will update one dependency to the version in the latest tag, regardless of the version range recorded in the package.json or the lock file. Using yarn upgrade-interactive also allows you to selectively upgrade packages to the latest version, not unlike npm-check.
$ yarn outdated
yarn outdated v0.16.1
Package Current Wanted Latest
babel-eslint 7.0.0 7.0.0 7.1.0
chai 3.0.0 3.0.0 3.5.0
Done in 0.84s.
$ yarn upgrade babel-eslint chai
yarn upgrade v0.16.1
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 2 new dependencies.
├─ babel-eslint#7.1.0
└─ chai#3.5.0
Running the following command will do what you want:
npm install someFixedVersionPackage#latest anotherFixedVersionPackage#latest --save-dev --save-exact
Breakdown:
npm install someFixedVersionPackage#latest will install the latest version of the package
The --save-dev flag will cause it to update the version in your package.json's devDependencies
The --save-exact flag will cause it to save a fixed version instead of a semver range operator
Link to the npm install docs
I've been looking for an easy way to update npm dependencies for a long time. Then I found this tool: https://github.com/dylang/npm-check
It shows you which dependencies are out of date in a nice ui and allows you to update them. It even tells you which ones are likely to break due to major changes and warns you of unused dependencies.