npm install throws ECONNRESET error but only for certain packages - npm

I'm working behind a corporate proxy. I have npm configured like this:
$ npm config get
...
https_proxy = "http://proxy.my-domain.com:8080"
https-proxy = "http://proxy.my-domain.com:8080"
proxy = "http://proxy.my-domain.com:8080"
...
With these settings, I can install some packages fine, but not others. For example $ npm i react works perfectly fine, whereas installing #babel/core throws an ECONNRESET error.
$ npm i #babel/core
npm ERR! code ECONNRESET
npm ERR! syscall read
npm ERR! errno -54
npm ERR! network read ECONNRESET
...
Strangely, I'm able to install the package with yarn (which is configured for the proxy the same way npm is), but it too tells me that I have network issues (even though it successfully installs the package 🤔)
$ yarn add #babel/core
...
✨ Done in 2.99s.
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
info There appears to be trouble with your network connection. Retrying...
I can't tell why it works with yarn, but not with npm. Here's what the npm debug log looks like:
73 silly tarball no local data for #jridgewell/sourcemap-codec#https://registry.npmjs.org/#jridgewell/sourcemap-codec/-/sourcemap-codec-1.4.14.tgz. Extracting by manifest.
74 silly tarball no local data for #jridgewell/trace-mapping#https://registry.npmjs.org/#jridgewell/trace-mapping/-/trace-mapping-0.3.17.tgz. Extracting by manifest.
75 silly tarball no local data for #jridgewell/set-array#https://registry.npmjs.org/#jridgewell/set-array/-/set-array-1.1.2.tgz. Extracting by manifest.
76 verbose stack Error: read ECONNRESET
76 verbose stack at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
77 verbose cwd /Users/justinsmith/Dev/test-npm
78 verbose Darwin 21.4.0
79 verbose node v18.9.0
80 verbose npm v8.19.1
81 error code ECONNRESET
82 error syscall read
83 error errno -54
Anyone have any ideas why this is happening?

tl;dr adding this to my ~./npmrc provided a temporary fix:
# This is what I already had set
proxy=http://proxy.my-domain.com:8080/
https-proxy=http://proxy.my-domain.com:8080/
https_proxy=http://proxy.my-domain.com:8080/
# This is what I added
strict-ssl=false
registry=http://registry.npmjs.org/
maxsockets=3
This forces npm to make requests with http instead of https and limits the number of parallel requests to three.
One reason this works is because evidently the corporate proxy was scanning files for malware before forwarding them to the client. Evidently npm making too many requests in parallel over TLS overwhelms the malware scanner which eventually refuses some connections.
Yarn on the other hand will retry a couple of times whereas npm just throws an error and gives up.
A longer-term solution would be to get your IT department to add npm to a malware scanning exclusion list.
Full credit goes to this blog post, which provided these helpful solutions.

Related

npm install fails with 405 Method Not Allowed

I am currently working on upgrade some projects to use node 18 with npm 8, from node 12 with npm 6.
We're using github as npm repository for our private packages.
Now I am getting this error on npm install:
npm ERR! code E405
npm ERR! 405 Method Not Allowed - GET https://npm.pkg.github.com/<our-github-organisation>/#types/mime/-/mime-3.0.1.tgz
In the error log:
456 verbose stack HttpErrorGeneral: 405 Method Not Allowed - GET https://npm.pkg.github.com/<our-github-organisation>/#types/mime/-/mime-3.0.1.tgz
456 verbose stack at /home/cclausen/.nvm/versions/node/v18.7.0/lib/node_modules/npm/node_modules/npm-registry-fetch/lib/check-response.js:93:15
456 verbose stack at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
457 verbose statusCode 405
458 verbose pkgid #types/mime#https://npm.pkg.github.com/<our-github-organisation>/#types/mime/-/mime-3.0.1.tgz
This does not happen on every package, only on a few select ones, I can't see pattern.
None of the affected patterns our ours, just random packages.
Why would this happen?
It seems the content of the .npmrc file used in the repository to specifc from where to get packages has changed, updating that file to comply with the newest information from githubs documentation has fixed this issue.
So the new content for me is:
#OWNER:registry=https://npm.pkg.github.com
Where OWNER is my organisation.

What is the correct way of installing firebase-tools (a npm package) in NixOS/Nix?

I am watching this Udemy course on Firebase. The course is a bit dated, initially made maybe 6 years ago.
At some point the author indicates how to use firebase CLI, hence he indicates the command:
~/projects/chatlive]$ npm install -g firebase-tools
I added sudo to avoid permission problems:
~/projects/chatlive]$ sudo npm install -g firebase-tools
Unfortunately, this error message is retrieved:
npm WARN checkPermissions Missing write access to /nix/store/nm7vm9d7xbvibazz7kl7xkqgjddqgiby-nodejs-14.18.3/lib/node_modules
npm ERR! code EROFS
npm ERR! syscall access
npm ERR! path /nix/store/nm7vm9d7xbvibazz7kl7xkqgjddqgiby-nodejs-14.18.3/lib/node_modules
npm ERR! errno -30
npm ERR! rofs EROFS: read-only file system, access '/nix/store/nm7vm9d7xbvibazz7kl7xkqgjddqgiby-nodejs-14.18.3/lib/node_modules'
npm ERR! rofs Often virtualized file systems, or other file systems
npm ERR! rofs that don't support symlinks, give this error.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2022-07-28T15_33_55_544Z-debug.log
I am using NixOS and this is my config file. Since the error message mentions the nix/store I thought it could be related.
Usually, I install things in NixOS changing my configuration.nix and doing sudo nixos-rebuild switch. But I thought it would not be necessary to do it considering npm was managing the installation.
Searching on Nix packages, I can see firebase-tools: link. Now, it seems that tweaking the config file is the way...
In situations like this, what is the best practice?
Should I force trying to install via npm (how?)? Should I go for a new generation of NixOS and a full system rebuild just because of this package?
Is there another possible path? Am I missing something?
Thanks.
If you just want to play around with firebase-tools in a development environment, create a shell.nix file in your working directory and add the following lines:
{ pkgs ? import <nixpkgs> {}}:
pkgs.mkShell {
nativeBuildInputs = [ pkgs.nodePackages.firebase-tools ];
}
Then run nix-shell to start a shell with firebase-tools included. This is considered best practice for local development environments, see https://nixos.wiki/wiki/Development_environment_with_nix-shell .
If you want to try out packages not in nixpkgs repo you should use npm and install a package locally (without -g ). See https://nixos.wiki/wiki/Node.js#Using_npm_install_-g_fails.

Unable to install NPM packages due to path length constraint using NPM 5.X

I'm running Windows 10, NPM v5.5.1, and node v8.9.1.
Edit: I have since updated to node v8.9.4 and npm v5.6.0. The problem remains unchanged.
I set up my development environment on this machine about 2 months ago. I'm using Angular CLI to build my client apps.
Everything was going fine. I added a few packages via NPM without a problem, such as Gulp and ngx-bootstrap. Then I started to run into errors on some of my package installs.
The errors are:
npm ERR! code PATH_LENGTH_EXCEEDED
npm ERR! errno PATH_LENGTH_EXCEEDED
npm ERR! request to https://registry/npmjs.org/{{package}} failed, reason: path length constraint exceeded
npm ERR! A complete log of this run can be found in:
npm ERR! {{npm_cache path}}\_logs\2018-02-08T14_43_40_856Z-debug.log
The contents of that log file are:
0 info it worked if it ends with ok
1 verbose cli [ 'C:\\Program Files\\nodejs\\node.exe',
1 verbose cli 'C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli 'install',
1 verbose cli 'ngx-cookie-service',
1 verbose cli '--save' ]
2 info using npm#5.5.1
3 info using node#v8.9.1
4 verbose npm-session d55be12849534a0a
5 silly install loadCurrentTree
6 silly install readLocalPackageData
7 silly fetchPackageMetaData error for ngx-cookie-service#latest request to https://registry.npmjs.org/ngx-cookie-service failed, reason: path length constraint exceeded
8 verbose type system
9 verbose stack FetchError: request to https://registry.npmjs.org/ngx-cookie-service failed, reason: path length constraint exceeded
9 verbose stack at ClientRequest.req.on.err (C:\Program Files\nodejs\node_modules\npm\node_modules\pacote\node_modules\make-fetch-happen\node_modules\node-fetch-npm\src\index.js:68:14)
9 verbose stack at emitOne (events.js:116:13)
9 verbose stack at ClientRequest.emit (events.js:211:7)
9 verbose stack at TLSSocket.socketErrorListener (_http_client.js:387:9)
9 verbose stack at emitOne (events.js:116:13)
9 verbose stack at TLSSocket.emit (events.js:211:7)
9 verbose stack at emitErrorNT (internal/streams/destroy.js:64:8)
9 verbose stack at _combinedTickCallback (internal/process/next_tick.js:138:11)
9 verbose stack at process._tickCallback (internal/process/next_tick.js:180:9)
10 verbose cwd C:\Development\test
11 verbose Windows_NT 10.0.15063
12 verbose argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "ngx-cookie-service" "--save"
13 verbose node v8.9.1
14 verbose npm v5.5.1
15 error code PATH_LENGTH_EXCEEDED
16 error errno PATH_LENGTH_EXCEEDED
17 error request to https://registry.npmjs.org/ngx-cookie-service failed, reason: path length constraint exceeded
18 verbose exit [ 1, true ]
I researched this, and found the Windows 10 file path 260 character limitation. I thought that npm version 3.1.0 should have fixed this issue, but I decided to followed the instructions in this answer, because I'm still running into this issue despite using NPM v5.5.1.
I found that my version of Windows was prior to the roll out of the feature that lets you disable this limitation. I updated to Version 1703 (OS Build 15063.483), and to enable Win32 long paths.
Yet I'm still getting the error, and unable to install a variety of npm packages.
In testing to see if it was an issue with packages that have a lot of dependencies, I tried installing or reinstalling a couple of other packages.
Some fail with the errors above. Others succeed with the following warnings:
npm WARN Unexepcted warming for https://registry.npmjs.org/: Miscellaneous Warning PATH_LENGTH_EXCEEDED: request to https://registry.npmjs.org/{{package name}} failed, reason: path length constraint exceeded
npm WARN Using stale package data from https://registry.npmjs.org/ due to a request error during validation
Packages throwing the error include:
jslint
ngx-cookie-service
chart.js
angular-cli
Packages that throw the warnings include:
font-awesome
#angular/cli
moment
ngx-bootstrap
bootstrap
jasmine
I suspect that something has changed, either in a policy on my workstation, or in our network security, that is causing this. All of the packages that succeed seem to be packages that I had previously installed, either as a direct install, or as a dependency of #angular/cli (one of the first packages I installed). I'm assuming that the stale package data warning indicates that whatever problem I have with the path length is preventing updated data being retrieved, and therefore npm is falling back to cached data from previous installations of the selected package and its dependencies.
So now I'm stumped as to how to diagnose exactly where the issue is happening. If its a security policy that changed, then I need to be able to describe exactly what is being blocked so that I can work with our security department to fix this. But the only information I can find about the PATH_LENGTH_EXCEEDED errors lead me to old information about versions of npm prior to 3.1.0.
How can I identify the source of this problem?
After additional searching, I found references to path length exceeded with regards to SSL/TLS certificate authorities. I tested this issue by changing the npm registry to use http instead of https (note: this is a temporary step only; I have turned https back on once I verified the issue):
npm config set registry http://registry.npmjs.org/
This allowed me to install the problematic packages.
It seems the underlying cause is SSL Inspection performed by our network security team. The decryption, inspection, and re-encryption process, and subsequent changes in the Issuing Authority on the certificate, apparently triggers this error.

npm ERR! 405 Method Not Allowed: express#latest

Of the several 405 related Questions posted here, none apply to my issue.
Here is the command I issued from Windows 7 on my Desktop PC:
npm install express
Here is the Error message:
npm ERR! code E405
npm ERR! 405 Method Not Allowed: express#latest
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\mr9832\AppData\Roaming\npm-cache\_logs\2018-02-21T20_36_09_581Z-debug.log
Here is the verbose Error Log mentioned above
C:\Users\mr9832\AppData\Roaming\npm-cache_logs\2018-02-21T20_36_09_581Z-debug.log
How can I fix it?
This answer is probably a little bit too late, but I had the same problem. I fixed it by putting following settings in the npm config (located in %USERDATA%/.npmrc):
strict-ssl=false
registry=http://registry.npmjs.org
It requests the packages over plain http instead of the secure https.
Use these commands:
npm config set proxy http://username:password#host:port
npm config set https-proxy http://username:password#host:port
Or you may edit the ~/.npmrc file:
proxy=http://username:password#host:port
https-proxy=http://username:password#host:port
https_proxy=http://username:password#host:port
I had the same problem, turns out the proxy and/or https-proxy config pointed to the wrong proxy server.
The following commands set the proxy to the correct address:
npm config set https-proxy http://127.0.0.1:3128
npm config set proxy http://127.0.0.1:3128

How should I set _auth in .npmrc when using a Nexus https npm registry proxy?

Context
My .npmrc file seems to be correctly read (checked with npm config ls -l both from command line and from Maven build).
the machine on which npm is run cannot connect directly to the net, it can only connect to a Nexus npm registry url. Therefore, proxy properties proxy and https-proxy are not set.
As access to Nexus is restricted, I have generated an access token from within Nexus.
Nexus security tokens are made from a username and a password which both contain characters such as / which usually have to be "url encoded"
as expected, with this configuration, when running npm install detects no proxy.
Nexus npm registry proxy seems to be correctly set (I can both access json files and download tgz files using a web browser after having connected using the token generated)
If I set registry to http://registry.npmjs.org/ and comment _auth, email, always-auth, strict-ssl properties, and add proxy and https-proxy configuration, npm install works as expected (but I won't be able to do it on target environment)
Content of .npmrc file
; Nexus proxy registry pointing to http://registry.npmjs.org/
registry = https://<host>/nexus/content/repositories/npmjs-registry/
; base64 encoded authentication token
_auth = <see question below>
; required by Nexus
email = <valid email>
; force auth to be used for GET requests
always-auth = true
; we don't want to put certificates in .npmrc
strict-ssl = false
loglevel = silly
Question
How should I generate the _auth property properly in order to have npm install work as expected?
I tried so far
base64Encode(<username>:<password>)
results in npm info retry will retry, error on last attempt: Error: socket hang up
base64Encode(urlencode(<username>:<password>))
results in npm info retry will retry, error on last attempt: Error: This request requires auth credentials. Run `npm login` and repeat the request.
base64Encode(urlencode(<username>):urlencode(<password>))
results in npm info retry will retry, error on last attempt: Error: socket hang up
When getting the socket hang up error I have the following stack trace:
http request GET https://<host>/nexus/content/repositories/npmjs-registry/fsevents
sill fetchPackageMetaData Error: socket hang up
sill fetchPackageMetaData at TLSSocket.onHangUp (_tls_wrap.js:1035:19)
sill fetchPackageMetaData at TLSSocket.g (events.js:260:16)
sill fetchPackageMetaData at emitNone (events.js:72:20)
sill fetchPackageMetaData at TLSSocket.emit (events.js:166:7)
sill fetchPackageMetaData at endReadableNT (_stream_readable.js:905:12)
sill fetchPackageMetaData at doNTCallback2 (node.js:441:9)
sill fetchPackageMetaData at process._tickCallback (node.js:355:17)
sill fetchPackageMetaData error for fsevents#^1.0.0 { [Error: socket hang up] code: 'ECONNRESET' }
WARN install Couldn't install optional dependency: socket hang up
verb install Error: socket hang up
verb install at TLSSocket.onHangUp (_tls_wrap.js:1035:19)
verb install at TLSSocket.g (events.js:260:16)
verb install at emitNone (events.js:72:20)
verb install at TLSSocket.emit (events.js:166:7)
verb install at endReadableNT (_stream_readable.js:905:12)
verb install at doNTCallback2 (node.js:441:9)
verb install at process._tickCallback (node.js:355:17)
When getting the This request requires auth credentials error I have the following stack trace:
npm sill fetchPackageMetaData Error: This request requires auth credentials. Run `npm login` and repeat the request.
npm sill fetchPackageMetaData at CachingRegistryClient.authify (<root>\ui\target\node\node_modules\npm\node_modules\npm-registry-client\lib\authify.js:17:14)
npm sill fetchPackageMetaData at CachingRegistryClient.makeRequest (<root>\ui\target\node\node_modules\npm\node_modules\npm-registry-client\lib\request.js:103:17)
npm sill fetchPackageMetaData at <root>\ui\target\node\node_modules\npm\node_modules\npm-registry-client\lib\request.js:66:17
npm sill fetchPackageMetaData at RetryOperation._fn (<root>\ui\target\node\node_modules\npm\node_modules\npm-registry-client\lib\attempt.js:18:5)
npm sill fetchPackageMetaData at null._onTimeout (<root>\ui\target\node\node_modules\npm\node_modules\retry\lib\retry_operation.js:49:10)
npm sill fetchPackageMetaData at Timer.listOnTimeout (timers.js:92:15)
npm sill fetchPackageMetaData error for fsevents#^1.0.0 [Error: This request requires auth credentials. Run `npm login` and repeat the request.]
npm WARN install Couldn't install optional dependency: This request requires auth credentials. Run `npm login` and repeat the request.
npm verb install Error: This request requires auth credentials. Run `npm login` and repeat the request.
npm verb install at CachingRegistryClient.authify (<root>\ui\target\node\node_modules\npm\node_modules\npm-registry-client\lib\authify.js:17:14)
npm verb install at CachingRegistryClient.makeRequest (<root>\ui\target\node\node_modules\npm\node_modules\npm-registry-client\lib\request.js:103:17)
npm verb install at <root>\ui\target\node\node_modules\npm\node_modules\npm-registry-client\lib\request.js:66:17
npm verb install at RetryOperation._fn (<root>\ui\target\node\node_modules\npm\node_modules\npm-registry-client\lib\attempt.js:18:5)
npm verb install at null._onTimeout (<root>\ui\target\node\node_modules\npm\node_modules\retry\lib\retry_operation.js:49:10)
npm verb install at Timer.listOnTimeout (timers.js:92:15)
Sources: https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/npm-registry/npm-security
& https://help.sonatype.com/repomanager3/nexus-repository-administration/formats/npm-registry/publishing-npm-packages
Configure registry (its important doing this before configuring the authentication in step 2, because the authentication settings will be based on the registry):
npm config set registry="http://localhost:8081/repository/npm-internal/"
Configure authentication using a line like the following example:
npm config set _auth="$(echo -n 'username:password' | base64)"
Check the current configuration using the following:
npm config ls
Publish your npm package:
npm publish --registry http://localhost:8081/repository/npm-internal/
If you have authorization token you should not use username:password.
I suggest you:
Generate token
Delete your ~/.npmrc or rename it.
Make sure your env settings like $NPM_CONFIG_* are unset.
Verify that email and other settings are unset by using: npm config list
Log into the npm using: npm login --registry=https://nexus.whatever.registry/respository/npm-whatever-group/
Once you are logged - you are logged. The npm should generate a token for it in your ~/.npmrc. It will look like:
//nexus.whatever.registry/respository/npm-whatever-group/:_authToken=NpmToken.YOUR-LOVELY-TOKEN-IN-HEX
You can use that token in your project, CI pipeline, and other ones. Make sure in your project .npmrc there is:
//nexus.whatever.registry/respository/npm-whatever-group/:_authToken=NpmToken.YOUR-LOVELY-TOKEN-IN-HEX
email = <EMAIL_USED_FOR_TOKEN_GENERATION>
always-auth = true
registry = https://nexus.whatever.registry/respository/npm-whatever-group/
If you have problems with authentication/certs:
add env variable (also to your CI/CD pipline)
$NODE_EXTRA_CA_CERTS to point to /home/wherever/is/your/cert.pem
For CI/CD pipelines (like gitlabs or jenikins):
consider replacing actual values from your .npmrc project file with ${RELEVANT_ENV_VARIABLES}. This way you will make them less visible and always self-updating on change of pipline.
Hope this help.
After having looked at registry-client code I found the answer, here it is. I post it as it may help other people:
base64Encode(<username>:<password>)
By the way, there is an URL encoding, but it's authify.js that takes care of it.
The "socket hang up" problem I'm facing is due to the fact that if a proxy is set in Windows configuration, when launching npm from CLI (and not from a Maven build) all ```.npmrc`` proxy settings seem to be ignored while native proxy exclusions (for corporate urls) are ignored by npm. I'll open a ticket to report this weird behavior.
Before you run npm login, please follow the instructions below :
1) Create an ~/.npmrc file with the following contents:
registry=https://example.com/repository/npm-group/
email=username#example.com
always-auth=true
//example.com/repository/npm-group/:_authToken=
2) run `npm login`
# npm login
Username: firstname.lastname
Password:
Email: (this IS public) firstname.lastname#example.com
Logged in as firstname,lastname on https://example.com/repository/npm-group/.
Use the same password you use to login to example.com
I don't know about Nexus, but we use artifactory as an npm repo, and there I can create my auth token by calling base64encode(username:encryptedPassword) with encryptedPassword being the one I get from my personal artifactory profile.
Maybe this helps.
I just wrote a wrapper that stores the credentials in your systems keychain and provides them on the fly. Check out: https://github.com/Xiphe/npm_keychain_auth