We have an issue with devops upstream npm feed. We built and deployed our angular app a few month ago with no issues. We are now trying to build the project but its complaining the below error. This seems to be because the locally stored version of the upstream npm package, ajv version 6.12.6 has been deleted; due to a retention policy. We are unable to get it back, does anyone know how we get it back?
npm ERR! code E404
npm ERR! 404 Not Found - GET https://tfs.pkgs.visualstudio.com/_packaging/myproject/npm/registry/ajv/-/ajv-6.12.6.tgz - Cannot find the package 'ajv' in feed 'npm_***'
npm ERR! 404
npm ERR! 404 'ajv#https://tfs.pkgs.visualstudio.com/_packaging/myproject/npm/registry/ajv/-/ajv-6.12.6.tgz' is not in this registry.
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.
The artifact view shows the below:
Update:
It would seem that there is no way to correct this! as mentioned here https://developercommunity.visualstudio.com/t/deleted-versions-of-upstream-packages-cannot-be-re/1620100
Fairly poor thought on DevOps part! So currently we are now pointing to the standard npm package url (https://registry.npmjs.org/) and not the dev ops artifact one; in order to get it to build again!
As you know, Azure DevOps feed has a immutability constraint. So there doesn't have any way to add the same version in the true sense at this moment.
Currently available methods are:
1, Create a new feed and download the package via this feed.
2, Download the package change the package info and publish to Artifact feed(As another package.)
3, Directly use the public source.
There is a suggestion ticket about this feature in official community:
Relax Azure Artifacts restriction on re-publishing versions
You can upvote this suggestion ticket and share your ideas in it.
Related
Since 2 days a download/install (no changes to the env at all) fails in my azure devops pipeline:
npm ERR! code E404 npm ERR! 404 Not Found - GET
https://pkgs.dev.azure.com/.../NPM-Mirror/npm/registry/esprima-fb/-/esprima-fb-15001.1001.0-dev-harmony-fb.tgz
Cannot find the package 'esprima-fb' in feed '' npm ERR! 404 npm ERR! 404
'esprima-fb#https://pkgs.dev.azure.com/.../NPM-Mirror/npm/registry/esprima-fb/-/esprima-fb-15001.1001.0-dev-harmony-fb.tgz'
is not in this registry. npm ERR! 404 You should bug the author to
publish it (or use the name yourself!) npm ERR! 404 npm ERR! 404 Note
that you can also install from a npm ERR! 404 tarball, folder, http
url, or git URL.
which basically points to that package here:
https://www.npmjs.com/package/esprima-fb/v/15001.1001.0-dev-harmony-fb
The package was last changed 7 years ago and worked fine until Wednesday of this week (even the underlying source was not changed for a while).
I tried Node version 16 and 17, both have same effect.
I tried to install the package by
- task: Npm#1
displayName: 'NPM: install'
inputs:
command: 'ci'
workingDir: 'path/'
and
- task: Npm#1
displayName: 'NPM: install'
inputs:
command: 'install'
workingDir: 'path/'
none worked.
I've recognized that this version of the package is not downloaded to the Azure Artifacts Upstream Archive (only an older version is available).
Tried to recreate another Feed, also no effect. Tried to add the dependency directly into the package.json file - all showed no effect at all.
Do you have any idea how to fix that problem? I ran out of ideas.
Azure DevOps feed for npm package sometimes throws incorrect HTTP status code when encountering authentication issue (it should be 401 instead of 404).
Typically npm install issues with Azure DevOps relies on an expired PAT token into your project's .npmrc file.
To be sure npm is always hitting npmjs.org for public packages, I'd suggest scoping your private packages (the ones coming from Azure DevOps feed) and defining the npmjs.org registry for all the others.
#[your_private_package_scope]:registry=[your_feed_url]
registry=https://registry.npmjs.org/
I'm trying to npm install a package from a heavily proxied Enterprise computer. One of the dependencies of this package tries to install via GitHub, which throws a 407 error.
npm ERR! /usr/bin/git ls-remote -h -t https://github.com/ethereumjs/ethereumjs-abi.git
npm ERR!
npm ERR! fatal: unable to access 'https://github.com/ethereumjs/ethereumjs-abi.git/': Received HTTP code 407 from proxy after CONNECT
I've discussed this with the team that manages these servers- even with proper proxy authentication, access to sites outside of internal registries are very uncommon as a security measure.
As an alternative, I found the package on the NPM registry, and successfully installed it using the internal mirror we have. However, it still tries to install the GitHub version as a dependency of the first package, and then the installation fails. Is there any way to prevent this? I think I've seen something about editing the package's package.json to remove the dependency, but I don't have any experience with this, so I'll only do it if it's the only option to prevent this. Thank you
I published several npm packages to a private npm registry hosted in GCP and I can see all versions in the registry. Yet I can't download the package since the install command throws an E404.
NPM throws a generic 404 Error when trying to download the page:
npm ERR! code E404
npm ERR! 404 Not Found - GET https://europe-north1-npm.pkg.dev/[Project]/#[scope]/[packagename]/-/#[scope]/[packagename]-0.1.0.tgz
npm ERR! 404
npm ERR! 404 '#[scope]/[packagename]#https://europe-north1-npm.pkg.dev/[Project]/#[scope]/[packagename]/-/#[scope]/[packagename]-0.1.0.tgz' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.
Opening the url that is tring to resolve returns this error:
{"error":"Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource \"projects/[Project]/locations/europe-north1/repositories/#[scope]\" (or it may not exist)"}
{"error":"Unmatched scope name: \"\" != \"#[scope]\""}
Additonally, I checked and my account does in fact have to downloadArtifacts role.
I can't figure out why it's unable to resolve the URL since with the same config I'm able to both publish and even run npm view the package.
I've tried creating brand new packages, specifying a version, resetting the auth token and yet for some reason I'm still unable to download packages.
If it helps, my current .npmrc file looks like this:
registry:https://registry.npmjs.org/=true
#[scope]:registry=https://europe-north1-npm.pkg.dev/[project]/[repo name]/
//europe-north1-npm.pkg.dev/[project]/[repo name]/:_authToken=[gcp auth token]
//europe-north1-npm.pkg.dev/[project]/[repo name]/:always-auth=true
//registry.npmjs.org/:_authToken=[npm auth token]
Turns out this was not a GCP specific issue, but a Yarn Berry issue when publishing to GCP.
The tarballURL that is being generated uses a truncated registry URL which is why i was getting a 404 when trying to install the package.
A fix for this was proposed here.
https://github.com/yarnpkg/berry/pull/3513
I had an error similar to this one.
I did the entire repository creation process and created a new project to test the package.
When I tried npm install my-package I got a 404 error.
After some tests I discovered that I need the .npmrc file also in the project that will consume the package. It may seem kind of obvious to some people but I didn't know.
Add the returned configuration settings to the .npmrc configuration file in your Node.js projects. This file is usually in the same directory as package.json.
Make sure that you include these settings in Node.js projects for packages that you publish as well as projects that will install dependencies from your npm repository.
Set up authentication for npm
I'm doing a proof-of-concept for my organization using Azure DevOps Pipelines to handle our front-end CI builds.
I've created two Angular projects: a library project and an application project that consumes that library. I've each to its own DevOps Project within my Organization, each of which has its own Repo. (e.g., the library Angular code is in My-Org/My-Library's Project's Repo, and the application that consumes that library is in My-Org/My-Application's Project's Repo.)
I've successfully gotten DevOps to publish that library's package to its Artifacts. I've successfully installed that package from Artifacts for my application from the CLI using npm install.
When I try to build the same application using an Azure Pipeline, things start out looking good but then I get warnings:
...
2020-09-25T01:40:22.9633584Z npm verb npm-session b9c6c5c07bc27d0f
2020-09-25T01:40:22.9634637Z npm info lifecycle #<myorganization>/<my-application-package-name>#0.0.0~preinstall: #<myorganization>/<my-application-package-name>#0.0.0
...
2020-09-25T01:40:22.9652940Z npm http fetch GET 200 https://registry.npmjs.org/ansi-colors/-/ansi-colors-4.1.1.tgz 888ms
2020-09-25T01:40:22.9653589Z npm http fetch GET 200 https://registry.npmjs.org/path-exists/-/path-exists-4.0.0.tgz 885ms
...
2020-09-25T01:40:22.9696448Z npm http fetch GET 200 https://registry.npmjs.org/tar/-/tar-6.0.5.tgz 256ms
2020-09-25T01:40:22.9697172Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/source-map-655ef13e/dist/source-map.js'
2020-09-25T01:40:22.9697948Z npm http fetch GET 200 https://registry.npmjs.org/minizlib/-/minizlib-2.1.2.tgz 254ms
2020-09-25T01:40:22.9698728Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/#angular/cli-095a8231/commands/build-impl.js'
2020-09-25T01:40:22.9699541Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/engine.io-client-a00fe2c5/LICENSE'
...
culminating with the unhappy finale:
...
2020-09-25T01:40:23.9366311Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/rxjs-77a83855/LICENSE.txt'
2020-09-25T01:40:23.9367111Z npm WARN tar ENOENT: no such file or directory, open '/home/vsts/work/1/s/node_modules/.staging/rxjs-77a83855/src/LICENSE.txt'
2020-09-25T01:40:23.9367879Z npm verb unlock done using /home/vsts/.npm/_locks/staging-b6ade8de5fa1f467.lock for /home/vsts/work/1/s/node_modules/.staging
2020-09-25T01:40:23.9369511Z npm verb stack Error: 404 Not Found - GET https://pkgs.dev.azure.com/<MyOrganization>/<My-Library-Project>/_packaging/<My-Library-Project-Artifact-Feed>/npm/registry/#<myorganization>/<my-library-package-name>/-/<my-library-package-name>-0.0.6.tgz
...
2020-09-25T01:40:23.9371625Z npm verb statusCode 404
2020-09-25T01:40:23.9372024Z npm verb pkgid #<myorganization>/<my-library-package-name>#0.0.6
2020-09-25T01:40:23.9372260Z npm verb cwd /home/vsts/work/1/s
2020-09-25T01:40:23.9372620Z npm verb Linux 5.4.0-1025-azure
2020-09-25T01:40:23.9373034Z npm verb argv "/opt/hostedtoolcache/node/12.18.4/x64/bin/node" "/opt/hostedtoolcache/node/12.18.4/x64/bin/npm" "install"
2020-09-25T01:40:23.9373392Z npm verb node v12.18.4
2020-09-25T01:40:23.9373573Z npm verb npm v6.14.6
2020-09-25T01:40:23.9373749Z npm ERR! code E404
2020-09-25T01:40:23.9374536Z npm ERR! 404 Not Found - GET https://pkgs.dev.azure.com/<MyOrganization>/<My-Library-Project>/_packaging/<My-Library-Project-Artifact-Feed>/npm/registry/#<myorganization>/<my-library-package-name>/-/<my-library-package-name>-0.0.6.tgz
2020-09-25T01:40:23.9375074Z npm ERR! 404
2020-09-25T01:40:23.9375516Z npm ERR! 404 '#<myorganization>/<my-library-package-name>#0.0.6' is not in the npm registry.
...
But here's the weird part: if I click on that url in the browser for which the Pipeline is reporting a 404---https://pkgs.dev.azure.com/<MyOrganization>/<My-Library-Project>/_packaging/<My-Library-Project-Artifact-Feed>/npm/registry/#<myorganization>/<my-library-package-name>/-/<my-library-package-name>-0.0.6.tgz---the browser downloads my package!
So to summarize:
When doing an npm install from the command-line for my application project, I'm able to install the package from the library's DevOps Artifacts, and
When clicking on the very URL that the application Pipeline log says is failing, the browser downloads the package from the library's Artifacts, but
The application's Pipeline that runs inside my DevOps Organization can't find it.
I suspect some kind of permissions or authorization issue, but I'm not sure where to go from here.
I've tried the npmAuthenticate#0 task, and indeed the logs say encouraging things like:
2020-09-25T01:40:04.2511306Z ##[debug]Got auth token
..
2020-09-25T01:40:04.2540281Z ##[debug]Created webApi client for https://dev.azure.com/<MyOrganization>/; options: {"proxy":null,"allowRetries":true,"maxRetries":5,"ignoreSslError":false}
2020-09-25T01:40:04.2581233Z ##[debug]Getting URI for area ID <some GUID> from https://dev.azure.com/<MyOrganization>/
2020-09-25T01:40:04.3973124Z ##[debug]Found resource area with locationUrl: https://pkgs.dev.azure.com/<MyOrganization>/
2020-09-25T01:40:04.3976465Z ##[debug]Found serviceUri: https://pkgs.dev.azure.com/<MyOrganization>/
2020-09-25T01:40:04.3978178Z ##[debug]Getting credentials for local feeds
2020-09-25T01:40:04.3978962Z SYSTEMVSSCONNECTION exists true
2020-09-25T01:40:04.3979926Z ##[debug]SYSTEMVSSCONNECTION exists true
2020-09-25T01:40:04.4003325Z ##[debug]Got auth token
2020-09-25T01:40:04.4004250Z ##[debug]Agent.ProxyUrl=undefined
2020-09-25T01:40:04.4005572Z ##[debug]Created webApi client for https://pkgs.dev.azure.com/<MyOrganization>/; options: {"proxy":null,"allowRetries":true,"maxRetries":5,"ignoreSslError":false}
2020-09-25T01:40:04.4007252Z ##[debug]Acquiring Packaging endpoints...
2020-09-25T01:40:04.6490830Z ##[debug]Successfully acquired the connection data
2020-09-25T01:40:04.6502681Z ##[debug]Acquired location
2020-09-25T01:40:04.6503915Z ##[debug]{"PackagingUris":["https://dev.azure.com/<MyOrganization>/","https://pkgs.dev.azure.com/<MyOrganization>/","https://pkgsprodcus1.pkgs.visualstudio.com/","https://pkgs.dev.azure.com/<MyOrganization>/","https://<myorganization>.pkgs.visualstudio.com/","https://pkgs.dev.azure.com/<MyOrganization>/"],"DefaultPackagingUri":"https://pkgs.dev.azure.com/<MyOrganization>/"}
Yet it still fails.
Any suggestions on what I can try?
Thanks!!
P.S. I should add that I have a lot more information I can share including my package.json, .npmrc, and the entire log but I wanted to keep this brief. If you need me to share additional details, please let me know and I'll add them. Thank you!
When connecting to a private project scoped feed from an Azure DevOps pipeline that is in the same organization but in a different project, the project that the feed is scoped to must allow access to the other project's build service. The build service must also be separately added to the feed permissions, regardless of the scope of the feed.
This is a problem with project scoped feed permissions. In short, to access a project scoped feed that is scoped to a project that is different than the project that the pipeline is running in, the project that the pipeline is running in must have access to BOTH the project that the feed is scoped to and the feed itself.
Here's how to set the proper permissions.
Check the project that the pipeline is running in. The build service permission that needs to be added to the feed permission and the feed's project permissions is going to look like something like [Project name] Build Service ([Organization name]).
In the project that the feed is scoped to, go to the permission settings to add the pipeline's project build service ([Project name] Build Service ([Organization name])) to a the Contributors group, or some other group your project may have that allows contributor access to its users.
In the feed permission page, add the [Project name] Build Service ([Organization name]) at least Collaborator access, so packages can be ingested from upstream sources. If you only give read permissions, packages cannot be ingested from upstream sources.
I have an app I'm creating a build pipeline for in Azure DevOps. It uses npm packages which are in a private npm registry (with code created from a different Azure DevOps organisation). When I run npm ci (or npm install) it fails with the following error:
npm ERR! code E401
npm ERR! Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/b2d01466-6e2c-4b55-8b90-e3ed41afca4a, Basic realm="https://pkgsproduks1.pkgs.visualstudio.com/", TFS-Federated
The specific packages which fail are the ones from the other organisation, which return a 401 when trying to get them.
I thought the best practice to authenticate this was to create a Service Connection within Azure DevOps. I've created a Personal Access Token within the organisation which hosts the npm packages, and used it to create a Service Connection in the organisation which contains my build pipeline. I then included it in my build pipeline yaml as follows:
- task: Npm#1
displayName: Install npm packages
inputs:
command: 'ci'
workingDir: 'Path/To/Working/Directory'
customEndpoint: 'Custom npm registry'
I've also tried using the npm authenticate build step before this (both with and without the customEndpoint: 'Custom npm registry' in the install step) and while the npm authenticate runs successfully it doesn't make any difference to the error I'm getting. I've also tried setting up the Service Connection to use my username and password rather than a PAT, but that made no difference either.
The .npmrc within my project is as follows (modified slightly):
registry=https://registry.npmjs.org/
#{scope}:registry=https://pkgs.dev.azure.com/{organisation}/_packaging/{feedName}/npm/registry/
#{scope}:always-auth=true
Can anyone see what's wrong with the authentication, or link to an article giving an example of doing this across multiple Azure DevOps organisations?
I just killed a few hours troubleshooting a similar NPM authentication issue with a hosted build agent for Azure DevOps.
I did have the NPM Authenticate job in the pipline, and I was still experiencing this error:
npm ERR! code E401
npm ERR! Unable to authenticate, your authentication token seems to be invalid.
npm ERR! To correct this please trying logging in again with:
npm ERR! npm login
As it turned, the project had a mixture of https://pkgs.dev.azure.com/<myorg>/_packaging and https://<myorg>.pkgs.visualstudio.com/_packaging (the legacy URL for the same NPM registry) in scattered .npmrc files.
Making them all to consistently use https://pkgs.dev.azure.com/<myorg>/_packaging had solved my problem.
The .npmrc should look like:
registry=https://pkgs.dev.azure.com/{organization}/{project}/_packaging/{feed}/npm/registry/
#{scope}:registry=https://pkgs.dev.azure.com/{otherorganization}/_packaging/{feed}/npm/registry/
#{otherscope}:registry=https://{thirdPartyRepository}/npm/registry/
always-auth=true