How to publish a scoped npm package to gitlab's package registry in a group namespace via a CI/CD pipeline - npm

I have a private dummy project in gitlab which I want to publish to gitlab's package registry. My dummy project contains four files:
package.json
{
"name": "#<my-group>/<my-project>",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": "git+https://gitlab.com/<my-group>/<my-project>.git"
},
"keywords": [],
"author": "",
"license": "ISC",
"bugs": {
"url": "https://gitlab.com/<my-group>/<my-project>/issues"
},
"homepage": "https://gitlab.com/<my-group>/<my-project>#readme"
}
.gitlab-ci.yml
image: node:latest
stages:
- deploy
deploy:
stage: deploy
script:
- echo "#<my-group>:registry=https://${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/">.npmrc
- echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}">>.npmrc
- npm publish
index.js
console.log('success');
README.md
<my-project>
When I commit my project to gitlab, the job fails with the following output
npm notice package: #<my-group>/<my-project>#1.0.0
npm notice === Tarball Contents ===
npm notice 8B README.md
npm notice 20B index.js
npm notice 612B package.json
npm notice === Tarball Details ===
npm notice name: #<my-group>/<my-project>
npm notice version: 1.0.0
npm notice filename: <my-group>-<my-project>-1.0.0.tgz
npm notice package size: 475 B
npm notice unpacked size: 640 B
npm notice shasum: 7b3db...
npm notice integrity: sha512-xDv0dl9A86...
npm notice total files: 3
npm notice
npm notice Publishing to https://gitlab.com/api/v4/projects/<my-project-id>/packages/npm/ with tag latest and default access
npm ERR! code E403
npm ERR! 403 403 Forbidden - PUT https://gitlab.com/api/v4/projects/<my-project-id>/packages/npm/#<my-group>%2f<my-project> - insufficient_scope
npm ERR! 403 In most cases, you or one of your dependencies are requesting
npm ERR! 403 a package version that is forbidden by your security policy, or
npm ERR! 403 on a server you do not have access to.
NOTE:
I have replaced the actual group name, project name and project id with <my-group>, <my-project>, and <my-project-id> in the code sections above.
I have followed gitlab's official documentation on setting this up (see https://docs.gitlab.com/ee/user/packages/npm_registry/) and believe that I can safely rule out the following:
I have made sure that Package registryis enabled in the project setup
I followed naming convention as described in the documentation
I am using a CI_JOB_TOKEN which should always be valid and should have appropriate permissions.
I made sure that there is no other package with the same name or version within the given scope.
I have made sure that the scoped package's URL includes a trailing slash (see gitlab-ci.yml above)
I have confirmed the path of the <my-group> namespace querying https://gitlab.com/api/v4/groups (just to make sure that the root namespace is correct)
I have used npm init --scope=#<my-group> --yes for initialization
The url of the repository is indeed: https://gitlab.com/<my-group>/<my-project>/
Any help on getting this to work would be much appreciated.

After changing from a job token ${CI_JOB_TOKEN} to a deploy token ${CI_DEPLOY_PASSWORD}, I was able to publish to the registry.
See Predefined variables reference for more info on predefined variables.
Gitlab deploy tokens can be created for projects under Settings > Repository > Deploy tokens with the following scopes: read_repository, read_package_registry, write_package_registry

Related

GitlabCI - Package Registry - NPM Package can't publish

My goal is to publish a npm package (private if possible) on the Gitlab Registry.
Here is my files :
Package.json :
{
"name": "#sushislasher/sushislasher-package",
"version": "0.0.1",
"description": "Package for game",
"main": "index.ts",
"repository": "https://gitlab.com/sushislasher/sushislasher-package",
"author": "Gildraen",
"license": "MIT",
"private": false
}
Gitlab-CI.yml :
stages:
- deploy
image: $CI_REGISTRY_IMAGE/docker-in-docker
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay2
COMPOSE_FILE: "docker-compose.yml:docker-compose.gitlab-ci.yml"
.template-load-gitlab-image: &internal-image
before_script:
- docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY
- docker-compose pull js || true
deploy:
<<: *internal-image
stage: deploy
script:
- |
{
echo "#${CI_PROJECT_ROOT_NAMESPACE}:registry=${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/npm/"
echo "${CI_API_V4_URL#https?}/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}"
} | tee -a app/.npmrc
- make install
- make npm-publish
make install build container, up them and do a yarn install.
make npm publish do basically a npm publish.
All in container.
I'm using node:15.7-alpine
But for now, despite all my tries and research, can't make this work, i have this error :
npm ERR! need auth This command requires you to be logged in.
npm ERR! need auth You need to authorize this machine using `npm adduser`
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2021-02-03T11_51_48_233Z-debug.log
1
I try to follow gitlab doc.. but i'm kind of stuck right now.. What am i doing wrong ?
I managed to do it. With a lot of try and try, here are the key points.
Package name. It is very important to keep an eye on it. What the documentation does not say it's that the "package.json" need the same name as the registry.
If your registry is #foo, your registry need #foo/something as project name. With the code I provide in the script, it takes the root project, and this is the registry.
For me it's https://gitlab.com/gildraen/sushislasher-package
so my package.json name is #gildraen/sushislasher-package (though the name seems not important)
So I did it all over again from scratch, with paying more attention to this, and it worked.
I really think this was my real mistake: the registry name.

Why does jsonlines package get resolved to registry.npm.taobao.org?

When I install the npm package jsonlines, it gets resolved to a mirrored registry registry.npm.taobao.org rather than registry.npmjs.org. It only does this for jsonlines. What causes this?
Here's the diff on my package-lock.json. The original "resolved" value was created when another developer installed the package:
"jsonlines": {
"version": "0.1.1",
- "resolved": "https://registry.npmjs.org/jsonlines/-/jsonlines-0.1.1.tgz",
+ "resolved": "https://registry.npm.taobao.org/jsonlines/download/jsonlines-0.1.1.tgz",
"integrity": "sha1-T80kbcXQ44aRkHxEqwAveC0dlMw="
},
I confirmed my configured registry is npmjs.org:
$ npm config get registry
https://registry.npmjs.org/
The developer's npm registry was likely set to registry.npm.taobao.org when they ran npm install jsonlines. Some users have npm configured to use the taobao registry for geographic proximity.
Deleting node_modules and package-lock.json and re-running npm install fixes it.
Tip: Use lockfile-lint to prevent it from happening again.
npm install --save-dev lockfile-lint
Run lockfile-lint to your lint script, ideally in a pre-push git hook.
Add this config to your package.json:
"lockfile-lint": {
"allowed-schemes": [
"https:"
],
"allowed-hosts": [
"npm"
],
"empty-hostname": false,
"type": "npm ",
"path": "package-lock.json"
},

Can "NPM -i" (local) install a global package without i know it?

I'm ok with local dependencies that packages install. But now I have a huge concerns about if a local installed package can install other global packages as dependencies.
as example:
npm install nunjucks
npm install sqlite
or
npm install botkit
It is not possible to mark a dependency as global in package.json, so that it is installed system-wide when you run npm i.
Here's an old comment by Isaac Schlueter stating that this will never be implemented.
Hooowever, it would be really simple to write a preinstall script to install arbitary dependencies globally.
{
"name": "Project",
"version": "1.0.0",
"description": "Preinstall script to install global deps",
"main": "index.js",
"scripts": {
"preinstall": "node -e \"const {execSync} = require('child_process'); JSON.parse(fs.readFileSync('package.json')).globalDependencies.forEach(globalDep => execSync('npm i -g ' + globalDep));\""
},
"dependencies": {
"react": "16.13.1"
},
"globalDependencies": [
"lodash"
],
"license": "ISC"
}
Copy this code into a package.json file in a folder on your PC. Then, in the folder run npm i. It will install React locally (in a node_modules folder) and it will install lodash globally.
You can verify this using: npm i ls -g --depth=0.
Reference: Install dependencies globally and locally using package.json
As to your question:
Can npm -i (local) install a global package without me knowing it?
It's not entirely silent. When running npm i for the above package.json file, you would see the following output:
> Project#1.0.0 preinstall /home/jim/Desktop/Project
> node -e "const {execSync} = require('child_process'); JSON.parse(fs.readFileSync('package.json')).globalDependencies.forEach(globalDep => execSync('npm i -g ' + globalDep));"
npm WARN Project#1.0.0 No repository field.
audited 6 packages in 1.113s
found 0 vulnerabilities
But whether you would catch this when running npm i on a large project is debatable.

package-lock.json resolved link points to local artifactory URL instead of globally set registry

I have a .npmrc file with:
registy=http://artifacts.sample.com/artifactory/api/npm/
package-lock.json contains resolved field for dependencies which looks something like:
"dependencies": {
"acorn": {
"version": "5.7.3",
"resolved": "http://artifacts.sample.com/artifactory/api/npm/acorn/-/acorn-5.7.3.tgz",
"integrity": "sha512-T/zvzYRfbVojPWahDsE5evJdHb3oJoQfFbsrKM7w5Zcs++Tr257tia3BmMP8XYVjp1S9RZXQMh7gao96BlqZOw==",
"dev": true
}
Now, when I do npm install, it changes the package-lock.json resolved field to point to a local artifactory instead of the one specified in .npmrc like:
"dependencies": {
"acorn": {
"version": "5.7.3",
"resolved": "http://localrepo.sample.com/artifactory/api/npm/acorn/-/acorn-5.7.3.tgz",
"integrity": "sha512-T/zvzYRfbVojPWahDsE5evJdHb3oJoQfFbsrKM7w5Zcs++Tr257tia3BmMP8XYVjp1S9RZXQMh7gao96BlqZOw==",
"dev": true
}
We have localized code and this will give access denied to someone living on the other side of the world, where their artifacts URL points somewhere else, or it will just change the package-lock.json file for them to point to their local URL instead, and we will always have changing package-lock.json file.
I had issues with "dev" and "integrity" too, but they were fixed using https://github.com/npm/npm/issues/16938#issuecomment-339863980
I'm on the latest versions of node and npm.
node -v
v10.15.3
npm -v
6.9.0
Is there anyway we can avoid this?
From my own experience. If you have setup default registry, the "resolved" field in package-lock.json will have no effect. I just followed Artifactory npm registry official doc.
npm config set registry https://artifactory.my.com/artifactory/api/npm/npm-repo/
npm login
npm config set //artifactory.my.com/artifactory/api/npm/npm-repo/:always-auth=true
Alternatively, a simple trick is to update the host file (eg. /etc/hosts) to ensure the "resolved" field url is never reachable, this will help you to observe npm behaviour.

How to make a forked dependency in package.json run its prepare script during npm install?

I forked a NPM module and modified it in my fork. In my fork, the command gulp needs to be run to compile the module. Running this command will output the file ical-expander-dist.js into the dist folder. This file should become part of the fork package.
The adjusted package.json in my fork looks like this:
{
"name": "ical-expander",
"version": "1.1.1",
"description": "ICS / iCal / iCalendar parser / expander",
"main": "dist/ical-expander-dist.js", // adjusted path
"scripts": {
"prepare": "gulp" // added script
},
// [SNIP] - excluded irrelevant lines
"files": [
"dist/ical-expander-dist.js" // adjusted path
]
}
Now I want to make my fork a dependency of another project.
{
"name": "otherproject",
"title": "Other Project",
// [SNIP] - excluded irrelevant lines
"dependencies": {
"ical-expander": "github:haukepribnow/ical-expander"
}
// [SNIP] - excluded irrelevant lines
}
After executing npm install in otherproject's root folder, the path ./node_modules/ical-expander will contain the files LICENSE, README.md and package.json. It will however not contain dist/ical-expander-dist.js.
It looks like the prepare script of my ical-expander fork is not being run during npm install in otherproject.
So my question boils down to: How can I make sure that preparation scripts for compiling my fork are being run when executing npm install in a project that has my fork as one of its dependencies?
Sometimes it's very simple: Make sure to use npm version 5.0.0 or higher.
According to the npm changelog, npm install will run dependencies' prepare scripts correctly from 5.0.0 on.