How can I authenticate a npm registry with azure devops build pipeline? - authentication

I have an app I'm creating a build pipeline for in Azure DevOps. It uses npm packages which are in a private npm registry (with code created from a different Azure DevOps organisation). When I run npm ci (or npm install) it fails with the following error:
npm ERR! code E401
npm ERR! Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/b2d01466-6e2c-4b55-8b90-e3ed41afca4a, Basic realm="https://pkgsproduks1.pkgs.visualstudio.com/", TFS-Federated
The specific packages which fail are the ones from the other organisation, which return a 401 when trying to get them.
I thought the best practice to authenticate this was to create a Service Connection within Azure DevOps. I've created a Personal Access Token within the organisation which hosts the npm packages, and used it to create a Service Connection in the organisation which contains my build pipeline. I then included it in my build pipeline yaml as follows:
- task: Npm#1
displayName: Install npm packages
inputs:
command: 'ci'
workingDir: 'Path/To/Working/Directory'
customEndpoint: 'Custom npm registry'
I've also tried using the npm authenticate build step before this (both with and without the customEndpoint: 'Custom npm registry' in the install step) and while the npm authenticate runs successfully it doesn't make any difference to the error I'm getting. I've also tried setting up the Service Connection to use my username and password rather than a PAT, but that made no difference either.
The .npmrc within my project is as follows (modified slightly):
registry=https://registry.npmjs.org/
#{scope}:registry=https://pkgs.dev.azure.com/{organisation}/_packaging/{feedName}/npm/registry/
#{scope}:always-auth=true
Can anyone see what's wrong with the authentication, or link to an article giving an example of doing this across multiple Azure DevOps organisations?

I just killed a few hours troubleshooting a similar NPM authentication issue with a hosted build agent for Azure DevOps.
I did have the NPM Authenticate job in the pipline, and I was still experiencing this error:
npm ERR! code E401
npm ERR! Unable to authenticate, your authentication token seems to be invalid.
npm ERR! To correct this please trying logging in again with:
npm ERR! npm login
As it turned, the project had a mixture of https://pkgs.dev.azure.com/<myorg>/_packaging and https://<myorg>.pkgs.visualstudio.com/_packaging (the legacy URL for the same NPM registry) in scattered .npmrc files.
Making them all to consistently use https://pkgs.dev.azure.com/<myorg>/_packaging had solved my problem.

The .npmrc should look like:
registry=https://pkgs.dev.azure.com/{organization}/{project}/_packaging/{feed}/npm/registry/
#{scope}:registry=https://pkgs.dev.azure.com/{otherorganization}/_packaging/{feed}/npm/registry/
#{otherscope}:registry=https://{thirdPartyRepository}/npm/registry/
always-auth=true

Related

Azure Devops: Sudden failure of NPM package download

Since 2 days a download/install (no changes to the env at all) fails in my azure devops pipeline:
npm ERR! code E404 npm ERR! 404 Not Found - GET
https://pkgs.dev.azure.com/.../NPM-Mirror/npm/registry/esprima-fb/-/esprima-fb-15001.1001.0-dev-harmony-fb.tgz
Cannot find the package 'esprima-fb' in feed '' npm ERR! 404 npm ERR! 404
'esprima-fb#https://pkgs.dev.azure.com/.../NPM-Mirror/npm/registry/esprima-fb/-/esprima-fb-15001.1001.0-dev-harmony-fb.tgz'
is not in this registry. npm ERR! 404 You should bug the author to
publish it (or use the name yourself!) npm ERR! 404 npm ERR! 404 Note
that you can also install from a npm ERR! 404 tarball, folder, http
url, or git URL.
which basically points to that package here:
https://www.npmjs.com/package/esprima-fb/v/15001.1001.0-dev-harmony-fb
The package was last changed 7 years ago and worked fine until Wednesday of this week (even the underlying source was not changed for a while).
I tried Node version 16 and 17, both have same effect.
I tried to install the package by
- task: Npm#1
displayName: 'NPM: install'
inputs:
command: 'ci'
workingDir: 'path/'
and
- task: Npm#1
displayName: 'NPM: install'
inputs:
command: 'install'
workingDir: 'path/'
none worked.
I've recognized that this version of the package is not downloaded to the Azure Artifacts Upstream Archive (only an older version is available).
Tried to recreate another Feed, also no effect. Tried to add the dependency directly into the package.json file - all showed no effect at all.
Do you have any idea how to fix that problem? I ran out of ideas.
Azure DevOps feed for npm package sometimes throws incorrect HTTP status code when encountering authentication issue (it should be 401 instead of 404).
Typically npm install issues with Azure DevOps relies on an expired PAT token into your project's .npmrc file.
To be sure npm is always hitting npmjs.org for public packages, I'd suggest scoping your private packages (the ones coming from Azure DevOps feed) and defining the npmjs.org registry for all the others.
#[your_private_package_scope]:registry=[your_feed_url]
registry=https://registry.npmjs.org/

Force NPM to skip package already installed from another source

I'm trying to npm install a package from a heavily proxied Enterprise computer. One of the dependencies of this package tries to install via GitHub, which throws a 407 error.
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/ethereumjs/ethereumjs-abi.git
npm ERR!
npm ERR! fatal: unable to access 'https://github.com/ethereumjs/ethereumjs-abi.git/': Received HTTP code 407 from proxy after CONNECT
I've discussed this with the team that manages these servers- even with proper proxy authentication, access to sites outside of internal registries are very uncommon as a security measure.
As an alternative, I found the package on the NPM registry, and successfully installed it using the internal mirror we have. However, it still tries to install the GitHub version as a dependency of the first package, and then the installation fails. Is there any way to prevent this? I think I've seen something about editing the package's package.json to remove the dependency, but I don't have any experience with this, so I'll only do it if it's the only option to prevent this. Thank you

Issues with publishing npm package on private gitlab registry

I have a (test) npm package named #myScope/test which I want to publish into a private gitlab registry within a gitlab project whose address is
https://gitlab.<something>/myId/test-npm-registry/
with ID 9630
First, I'm a bit confused with npm login and .npmrc. If I understand correctly, answers given to npm login end-up as a couple of lines into .npmrc, is that right?
So essentially, to instruct npm about authenticating to a registry, one can either:
manually edit .npmrc
use npm config set
use npm login
Based on what I read on the gitlab help, I've inserted the following lines into my .npmrc:
#myScope:registry=https://gitlab.<something>/api/v4/projects/9630/packages/npm/
'//gitlab.<something>/api/v4/projects/9630/packages/npm/:_authToken'="${GITLAB_AUTH_TOKEN}"
Note: I had to use the project ID, having issues with the url format.
I have also added the following bit into the package.json :
publishConfig":{
"#myScope:registry": "https://gitlab.<something>/api/v4/projects/9630/packages/npm/"
}
Finally, I've created a token in gitlab, with "api, read_api, read_registry, write_registry" rights and I have assigned the token to the GITLAB_AUTH_TOKEN variable:
GITLAB_AUTH_TOKEN="xk4L7xxvzHuykyKawxQZ"
When I do npm publish I'm getting the following message :
npm ERR! code E401
npm ERR! 401 Unauthorized - PUT https://gitlab.<something>/api/v4/projects/9630/packages/npm/#myScope%2ftest-npm-registry
npm ERR! A complete log of this run can be found in:
The logs doesn't help, nor does the verbose mode. Also, I'm not getting if I have to perform an explicit npm login or if the configuration in the .npmrc (with the token) should do the trick. I actually cannot do npm login because the username has an # character in it and npm refuses it.
Any ideas ?

Error installing a package published to Azure DevOps NPM Artifacts in other organization project

I'm doing a proof-of-concept for my organization using Azure DevOps Pipelines to handle our front-end CI builds.
I've created two Angular projects: a library project and an application project that consumes that library. I've each to its own DevOps Project within my Organization, each of which has its own Repo. (e.g., the library Angular code is in My-Org/My-Library's Project's Repo, and the application that consumes that library is in My-Org/My-Application's Project's Repo.)
I've successfully gotten DevOps to publish that library's package to its Artifacts. I've successfully installed that package from Artifacts for my application from the CLI using npm install.
When I try to build the same application using an Azure Pipeline, things start out looking good but then I get warnings:
...
2020-09-25T01:40:22.9633584Z npm verb npm-session b9c6c5c07bc27d0f
2020-09-25T01:40:22.9634637Z npm info lifecycle #<myorganization>/<my-application-package-name>#0.0.0~preinstall: #<myorganization>/<my-application-package-name>#0.0.0
...
2020-09-25T01:40:22.9652940Z npm http fetch GET 200 https://registry.npmjs.org/ansi-colors/-/ansi-colors-4.1.1.tgz 888ms
2020-09-25T01:40:22.9653589Z npm http fetch GET 200 https://registry.npmjs.org/path-exists/-/path-exists-4.0.0.tgz 885ms
...
2020-09-25T01:40:22.9696448Z npm http fetch GET 200 https://registry.npmjs.org/tar/-/tar-6.0.5.tgz 256ms
2020-09-25T01:40:22.9697172Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/source-map-655ef13e/dist/source-map.js'
2020-09-25T01:40:22.9697948Z npm http fetch GET 200 https://registry.npmjs.org/minizlib/-/minizlib-2.1.2.tgz 254ms
2020-09-25T01:40:22.9698728Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/#angular/cli-095a8231/commands/build-impl.js'
2020-09-25T01:40:22.9699541Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/engine.io-client-a00fe2c5/LICENSE'
...
culminating with the unhappy finale:
...
2020-09-25T01:40:23.9366311Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/rxjs-77a83855/LICENSE.txt'
2020-09-25T01:40:23.9367111Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/rxjs-77a83855/src/LICENSE.txt'
2020-09-25T01:40:23.9367879Z npm verb unlock done using /home/vsts/.npm/_locks/staging-b6ade8de5fa1f467.lock for /home/vsts/work/1/s/node_modules/.staging
2020-09-25T01:40:23.9369511Z npm verb stack Error: 404 Not Found - GET https://pkgs.dev.azure.com/<MyOrganization>/<My-Library-Project>/_packaging/<My-Library-Project-Artifact-Feed>/npm/registry/#<myorganization>/<my-library-package-name>/-/<my-library-package-name>-0.0.6.tgz
...
2020-09-25T01:40:23.9371625Z npm verb statusCode 404
2020-09-25T01:40:23.9372024Z npm verb pkgid #<myorganization>/<my-library-package-name>#0.0.6
2020-09-25T01:40:23.9372260Z npm verb cwd /home/vsts/work/1/s
2020-09-25T01:40:23.9372620Z npm verb Linux 5.4.0-1025-azure
2020-09-25T01:40:23.9373034Z npm verb argv "/opt/hostedtoolcache/node/12.18.4/x64/bin/node" "/opt/hostedtoolcache/node/12.18.4/x64/bin/npm" "install"
2020-09-25T01:40:23.9373392Z npm verb node v12.18.4
2020-09-25T01:40:23.9373573Z npm verb npm v6.14.6
2020-09-25T01:40:23.9373749Z npm ERR! code E404
2020-09-25T01:40:23.9374536Z npm ERR! 404 Not Found - GET https://pkgs.dev.azure.com/<MyOrganization>/<My-Library-Project>/_packaging/<My-Library-Project-Artifact-Feed>/npm/registry/#<myorganization>/<my-library-package-name>/-/<my-library-package-name>-0.0.6.tgz
2020-09-25T01:40:23.9375074Z npm ERR! 404
2020-09-25T01:40:23.9375516Z npm ERR! 404 '#<myorganization>/<my-library-package-name>#0.0.6' is not in the npm registry.
...
But here's the weird part: if I click on that url in the browser for which the Pipeline is reporting a 404---https://pkgs.dev.azure.com/<MyOrganization>/<My-Library-Project>/_packaging/<My-Library-Project-Artifact-Feed>/npm/registry/#<myorganization>/<my-library-package-name>/-/<my-library-package-name>-0.0.6.tgz---the browser downloads my package!
So to summarize:
When doing an npm install from the command-line for my application project, I'm able to install the package from the library's DevOps Artifacts, and
When clicking on the very URL that the application Pipeline log says is failing, the browser downloads the package from the library's Artifacts, but
The application's Pipeline that runs inside my DevOps Organization can't find it.
I suspect some kind of permissions or authorization issue, but I'm not sure where to go from here.
I've tried the npmAuthenticate#0 task, and indeed the logs say encouraging things like:
2020-09-25T01:40:04.2511306Z ##[debug]Got auth token
..
2020-09-25T01:40:04.2540281Z ##[debug]Created webApi client for https://dev.azure.com/<MyOrganization>/; options: {"proxy":null,"allowRetries":true,"maxRetries":5,"ignoreSslError":false}
2020-09-25T01:40:04.2581233Z ##[debug]Getting URI for area ID <some GUID> from https://dev.azure.com/<MyOrganization>/
2020-09-25T01:40:04.3973124Z ##[debug]Found resource area with locationUrl: https://pkgs.dev.azure.com/<MyOrganization>/
2020-09-25T01:40:04.3976465Z ##[debug]Found serviceUri: https://pkgs.dev.azure.com/<MyOrganization>/
2020-09-25T01:40:04.3978178Z ##[debug]Getting credentials for local feeds
2020-09-25T01:40:04.3978962Z SYSTEMVSSCONNECTION exists true
2020-09-25T01:40:04.3979926Z ##[debug]SYSTEMVSSCONNECTION exists true
2020-09-25T01:40:04.4003325Z ##[debug]Got auth token
2020-09-25T01:40:04.4004250Z ##[debug]Agent.ProxyUrl=undefined
2020-09-25T01:40:04.4005572Z ##[debug]Created webApi client for https://pkgs.dev.azure.com/<MyOrganization>/; options: {"proxy":null,"allowRetries":true,"maxRetries":5,"ignoreSslError":false}
2020-09-25T01:40:04.4007252Z ##[debug]Acquiring Packaging endpoints...
2020-09-25T01:40:04.6490830Z ##[debug]Successfully acquired the connection data
2020-09-25T01:40:04.6502681Z ##[debug]Acquired location
2020-09-25T01:40:04.6503915Z ##[debug]{"PackagingUris":["https://dev.azure.com/<MyOrganization>/","https://pkgs.dev.azure.com/<MyOrganization>/","https://pkgsprodcus1.pkgs.visualstudio.com/","https://pkgs.dev.azure.com/<MyOrganization>/","https://<myorganization>.pkgs.visualstudio.com/","https://pkgs.dev.azure.com/<MyOrganization>/"],"DefaultPackagingUri":"https://pkgs.dev.azure.com/<MyOrganization>/"}
Yet it still fails.
Any suggestions on what I can try?
Thanks!!
P.S. I should add that I have a lot more information I can share including my package.json, .npmrc, and the entire log but I wanted to keep this brief. If you need me to share additional details, please let me know and I'll add them. Thank you!
When connecting to a private project scoped feed from an Azure DevOps pipeline that is in the same organization but in a different project, the project that the feed is scoped to must allow access to the other project's build service. The build service must also be separately added to the feed permissions, regardless of the scope of the feed.
This is a problem with project scoped feed permissions. In short, to access a project scoped feed that is scoped to a project that is different than the project that the pipeline is running in, the project that the pipeline is running in must have access to BOTH the project that the feed is scoped to and the feed itself.
Here's how to set the proper permissions.
Check the project that the pipeline is running in. The build service permission that needs to be added to the feed permission and the feed's project permissions is going to look like something like [Project name] Build Service ([Organization name]).
In the project that the feed is scoped to, go to the permission settings to add the pipeline's project build service ([Project name] Build Service ([Organization name])) to a the Contributors group, or some other group your project may have that allows contributor access to its users.
In the feed permission page, add the [Project name] Build Service ([Organization name]) at least Collaborator access, so packages can be ingested from upstream sources. If you only give read permissions, packages cannot be ingested from upstream sources.

npm publish azure artifacts

I'm trying to publish a scoped package to a private azure devops artifact feed. I followed the instructions here. I have a project .npmrc with the following entries:
#my-scope:registry=https://pkgs.dev.azure.com/my-org/_packaging/my-feed/npm/registry/
#my-scope:always-auth=true
In my global user .npmrc I have the following entries:
prefix=/usr/local
strict-ssl=false
unsafe-perm=true
//registry.npmjs.org/:_authToken="my-real-token"
//pkgs.dev.azure.com/my-org/_packaging/my-feed/npm/registry/:username=${NPM_USERNAME}
//pkgs.dev.azure.com/my-org/_packaging/my-feed/npm registry/:_password="my-real-base64-token"
//pkgs.dev.azure.com/my-org/_packaging/my-feed/npm/registry/:email=${NPM_EMAIL}
//pkgs.dev.azure.com/my-org/_packaging/my-feed/npm/registry/:always-auth=true
#my-scope:registry=https://pkgs.dev.azure.com/my-org/_packaging/my-feed/npm/registry/
//pkgs.dev.azure.com/my-org/_packaging/my-feed/npm registry/:_authToken="my-real-base64-token"
cafile=${NPM_CERT_LOCATION}
When I try: npm publish I get the following error:
Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/some-guid, Basic realm="https://pkgsprodcus1.pkgs.visualstudio.com/", TFS-Federated
Based on previous posts I see that I might need to do npm login. Executing npm login gives me this error:
npm verb node v6.9.2
npm verb npm v6.8.0
npm ERR! code E400
npm ERR! 400 Bad Request - PUT https://pkgs.dev.azure.com/my-org/_packaging/my-feed/npm/registry/-/user/org.couchdb.user:my-username
I looked at this and this which seemed to be related. However, neither of them worked.
I've tried: curl and curl -u which gave me the following error:
{"$id":"1","innerException":null,"message":"TF400813: Resource not available for anonymous access. Client authentication required.","typeName":"Microsoft.TeamFoundation.Framework.Server.UnauthorizedRequestException, Microsoft.TeamFoundation.Framework.Server","typeKey":"UnauthorizedRequestException","errorCode":0,"eventId":3000}%
I'v also tried with a proxy and a cert. However, with the same results.
This is a bit old, not sure if you are still stuck, but for Windows you can use this npm package: https://www.npmjs.com/package/vsts-npm-auth:
npm install -g vsts-npm-auth
vsts-npm-auth -config path-to-your\.npmrc
Here is a more complete article from Azure DevOps that walks you through setup, .npmrc and auth and publishing: https://learn.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops&tabs=windows:
If you are developing on Linux or Mac, vsts-npm-auth is not supported and we recommend generating a token in the following manner for your $HOME/.npmrc
The Connect to feed dialog box generates an appropriately formatted token that you can place into your .npmrc file with a lifespan of 90 days.
From Azure Artifacts, select Connect to feed.
Select npm.
Select Generate npm credentials. Copy the credentials to add them to your user .npmrc file manually. For Windows this is in %USERPROFILE%.npmrc and can be useful if the above method doesn't work. For Linux it is in $HOME/.npmrc.