Our developers pull all of their npm dependencies via an artifactory proxy. The artifactory setup uses a virtual repository that consists of a local npm repository and a remote npm repository.
When developers perform an npm install, the process is slow and often hangs at
fetchMetadata -> network
Any ideas on tracing the source of this issue?
Does artifactory provide a 'trace' api for npm resources, similar to what they have for maven dependencies here maven trace?
Would re-indexing the virtual repository on artifactory help?
Struggled with the same thing. By adding the debug flag when running install helped me. I don't know why, might be some race condition or to many half open connections or something else. When the debug flag is added each step takes a fraction of a second longer to complete, and that seems to be enough. Here's an example for a local package:
npm install -d
or a longer example:
sudo npm install -g -ddd eslint-cli
The number of d's tells npm how verbose debug you want. More d's means more info and longer time to execute. One d did it for me, but my laptop is slow to begin with.
Related
I am studying about npm and I have some questions.
Where the npm get the package from? i.e. when run npm install <package-name> or yarn add <package-name>.
When get the package, do npm get the package as raw or get then build it(like run the build script written in package.json)?
When publish the package, the repository field of package.json is required?
Can be different between the repository for publishing and the repository in pacakge.json?
To answer your questions:
npm gets them from the NPM package registry, and so does yarn, but Yarn probably has a proxy registry in front of it. In general, you can say, both tools fetch their packages from https://npmjs.com by default.
It gets the package as it was published (so, in short, the answer is "raw"). Building is up to the publisher and depends on the type of package. Often, some prepublish task builds something into dist/ (or any other location in the package), and these files are also shipped with the package others then download. Building rarely happens after installing a package (exception here are library-wrapping packages built with node-gyp).
The repository field is not required, to my knowledge, but it is good practise to include it (it will be displayed on the NPM website, for example).
Technically, yes. You can just specify any repository in repository, but it wouldn't make much sense to specify one that isn't the source of the package.
If you in general want to read up more on how npm works, check out it's documentation over at https://docs.npmjs.com/
I am on different servers and run npm install
One server is ok. Another failed showing below error:
no matching version found for es-abstract#1.14.0
I tried npm ci. Same deal.
Then I did npm install es-abstract-1.14.0.tgz
But the size node_modules/ are different. I am using du -shc --apparent-size node_modules/ to ignore the sparse files within the folder. Why?
Could I accept that this is the network issue of the failed server？ And just keep doing this? I mean, is the installation still going to be ok in this way?
At the time that this answer was written, there are 33 release versions for es-abstract on github, but only 32 release versions are listed on its npm registry. The missing version in the registry is 1.14.0.
Perhaps on one of your servers, you had this package cached (maybe it was previously listed on the NPM registry?, or maybe from downloading it from github?), and on the other server you did not have this package cached.
I had this same error message when trying to npm install a project from github.
In my situation, the es-abstract package was not explicitly listed in the package.json file, but it was a dependency of another package. Therefore I explicitly added it with the next highest version listed on the registry, and it worked.
e.g.
"dependencies": {
"es-abstract": "1.14.1",
...
Just a guess, but maybe 1.14.0 used to be listed on the registry, but now it's not?
We have a private JFrog artifactory (name anonymised below) that npm is configured in a project root .npmrc -file:
registry=https://artifactory.jfrog.private.com:443/api/npm/npm-registry-virtual/
The resolved-field in the package-lock.json file shared via Git between developers is constantly changing between runs of "npm install" without any changes to package.json.
Some times a dl query parameter (pointing to the exactly same URL) gets added to the resolved URL:
- "resolved": "https://artifactory.jfrog.private.com:443/api/npm/npm-registry-virtual/#sailshq/lodash/-/lodash-3.10.3.tgz",
+ "resolved": "https://artifactory.jfrog.private.com:443/api/npm/npm-registry-virtual/#sailshq/lodash/-/lodash-3.10.3.tgz?dl=https://artifactory.jfrog.private.com/#sailshq/lodash/-/lodash-3.10.3.tgz",
Some times the query parameter points to npmjs.org registry:
- "resolved": "https://artifactory.jfrog.private.com:443/api/npm/npm-registry-virtual/aproba/-/aproba-1.2.0.tgz",
- "resolved": "https://artifactory.jfrog.private.com:443/api/npm/npm-registry-virtual/aproba/-/aproba-1.2.0.tgz?dl=https://registry.npmjs.org/aproba/-/aproba-1.2.0.tgz",
And some times the field points directly to npmjs.org repository:
- "resolved": "https://artifactory.jfrog.private.com:443/api/npm/npm-registry-virtual/acorn/-/acorn-3.3.0.tgz",
+ "resolved": "https://registry.npmjs.org/acorn/-/acorn-3.3.0.tgz",
Any of these changes may also go to the inverse direction.
This is really irritating, since it means we constantly have meaningless changes in package-lock.json, which causes merge conflicts and often prevents npm ci from executing correctly. npm cache clean --force does not seem to help. I know that npm install can resolve package-lock.json merge conflicts automatically, but that does not help with npm ci (since the whole point is to not run npm install in the CI environment). And, anyway, what is the benefit of seeing how the virtual npm registry resolves the packages internally (as I suspect is happening here)?
Is there some kind of configuration option to prevent JFrog Artifactory from making these kinds of changes to the resolved package URLs in a virtual npm registry? Or is it maybe a bug in npm?
Environment:
npm 6.11.3
JFrog Artifactory 6.10.6
I don't know why those alternate URLs are appearing or how to make them stop. But you can reduce (or maybe even eliminate!) the merge conflict pain for your developers by using npm-merge-driver. It was written by one of the devs who was employed on the npm cli team for years, and its sole purpose is to automate away package-lock.json merge conflicts.
Our team has had success running npm ci first to ensure our locally pulled down and cached dependencies match the package-lock.json file.
Then, further npm installs should resolve as expected.
This sort of thing is normally caused by developers having slightly different versions of npm installed. Version 7 of npm just got released, so it is the perfect time to make sure the team all have exactly the same version installed.
If that doesn’t work try switching the team to yarn or pnpm.
How can I have npm tell me if all dependencies are already installed without checking the network?
My goal is to first test locally if anything needs to be installed, and then only if something is missing, I'll run a normal npm install to install it. I'm trying to avoid the initial check across the network though if everything is already there.
This is also given a package.json file with fixed versions, since obviously allowing auto upgraded packages would always require a remote repository check.
Update:
I've tested npm list which doesn't seem to access the network, and it prints out "UNMET DEPENDENCY" if something is in package.json but not installed. Is that the best way to accomplish this?
I'll probably end up with something like:
npm list | grep -c 'UNMET DEPENDENCY'
I'm not aware of anything that will explicitly tell you whether dependencies were installed from a remote repo or not. However I think that the shrinkpack package will help you achieve your aim.
Shrinkpack will cache your npm modules locally and only contact a remote repository when existing modules change or new modules are added.
I've used this in the past to reduce the number of network requests required for an npm install.
I've setup an Nexus (2.10.0) NPM repository and administrate an user with full access to this repo.
By setting the "_auth" and "always-auth" param in my local .npmrc the publishing and reading of npm artifacts are working well. But if i try to unpublish or deprecate an npm artifact i get still an error "...This request requires auth credentials. Run npm login and repeat the request...".
What is the preferred way to remove artifacts from an nexus npm repository?
thx,
David
Unpublish is currently not supported. Details see https://issues.sonatype.org/browse/NEXUS-6892
Also keep in mind that is not considered good practice to use unpublish as you can see from the npm documentation itself linked in the issue.
Unfortunately, Nexus 2.11 doesn't support npm unpublish or npm deprecate.
If the goal is to make your previously published versions of npm-packages inaccessible:
You can delete the packages from Nexus by manually removing the tarballs from .../nexus/storage/<your_npm_repo>/<your_package>/-/ on your Nexus server and creating a scheduled task to 'rebuild hosted npm metadata':
The npm metadata for a hosted repository
can be rebuilt based on the components found in the storage of a
hosted repository. The task can serve as a recovery tool in cases
where the npm metadata database got corrupted or the component storage
was created manually or via some external process like e.g. an rsync
copying.
This also works great for injecting old versions of packages into your npm-repo. (To get these tarballs, use npm pack.)