Error in .gitlab-ci while trying to package - npm

So I'm trying to make conditions in my .gitlab-ci.yml if there is no package then, npm publish to pack the library in the GitLab registry
I gave my pipeline the permission to the registry and the npm access token, but I still get the unauthorized error
this is the part of the .gitlab-ci.yml where I create the .npmrc file and set the configurations.
script:
- |
if [[ ! -f .npmrc ]]; then
echo 'No .nmprc found! Creating one now.'
echo "#${CI_PROJECT_ROOT_NAMESPACE}:registry=${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/npm/">.npmrc
echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}">>.npmrc
echo "//registry.npmjs.org/:_authToken=${NPM_ACCESS_TOKEN}">>.npmrc
echo "Created the following .npmrc:"; cat .npmrc
fi
The pipeline get me this when I try to find if there is a package with the name of : $NPM_PACKAGE_NAME
#scope:registry = "https://gitlab.example.com/api/v4/projects/project_id/packages/npm/"
//gitlab.example.com/api/v4/projects/projet_id/packages/npm/:_authToken = (protected)
//registry.npmjs.org/:_authToken = (protected)
; "cli" config from command line options
long = true
$ npm config set always-auth true
$ echo $(npm view "${NPM_PACKAGE_NAME}" )
npm ERR! code E401
npm ERR! 401 Unauthorized - GET https://gitlab.example.com/api/v4/projects/project_id/packages/npm/#scope%2fmy-package
Where :
- NPM_PACKAGE_NAME=$(node -p "require('./my-package/package.json').name")

Instead of echo to the .npmrc, you could try the npm commands directly
npm config set -- '//gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken' "${NPM_TOKEN}"
npm config set -- '//gitlab.example.com/api/v4/packages/npm/:_authToken' "${NPM_TOKEN}"
That way, you are sure the .npmrc is properly updated.

Could you try to create the .npmrc in your home folder instead of locally?
We do this in our pipeline and it works without any problem:
publish:
stage: publish
script:
- echo "#<scope>:registry=https://${CI_SERVER_HOST}/api/v4/projects/${REGISTRY_PROJECT_ID}/packages/npm/" > ~/.npmrc
- echo "//${CI_SERVER_HOST}/api/v4/projects/${REGISTRY_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}" >> ~/.npmrc
- npm version --no-git-tag-version "$(<.next-version)" --allow-same-version
- npm publish --tag ${NPM_TAG_NAME}
As you see, other than the npm version and npm publish commands, the only difference is the .npmrc file location.

Related

Bitbucket pipelines - SCP Deployment- Failing "Identity file /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp not accessible: No such file or directory."

I am trying to add bitbucket pipelines to deploy my angular application to one of our servers.
I configured SSH on my server and I could fetch the Host's finger print under Known hosts in Bitbucket
Below is my YAML file.
image: node:14
pipelines:
branches:
master:
- step:
name: Building Test angular application
caches:
- node
script:
- echo "npm install in progress.."
- npm install
- echo "Installing angular/cli..."
- npm install -g #angular/cli
- echo "Starting the Build process.."
- ng build
artifacts:
- dist/** # Save build for next steps
- step:
name: "Deployment"
script:
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: $USER
SERVER: $SERVER
REMOTE_PATH: '/c/testscp/'
LOCAL_PATH: 'dist/*'
SSH_KEY: $MY_SSH_KEY
The first step is running fine without any issues and I can see dist folder being added to the artifact however the second step is failing with the below error.
scp -rp -i /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp dist/TestPipelineApplication <<USER>>#<<SERVERIP>>:/c/testscp/
Warning: Identity file /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp not accessible: No such file or directory.
Load key "/root/.ssh/pipelines_id": invalid format
Permission denied, please try again.
Permission denied, please try again.
<<USER>>#<<SERVERIP>>: Permission denied (publickey,password,keyboard-interactive).
I never configured pipelines before so I am not completely sure what I am missing here.
Also, I looked into the below documentation but of no luck
https://bitbucket.org/atlassian/scp-deploy/src/1.0.1/README.md
Any help or suggestions are greatly appreciated.

Azure DevOps private Linux agent - YAML pipeline checkout failing - can fix with "git config --global --unset http.extraHeader" but not early enough

We have an issue regarding the following:
Azure DevOps Linux Private Agent
Possible issue with corrupt / stale bearer token
Can be fixed with this command by logging onto the box but this is not convenient: git config --global --unset http.extraHeader
Can be fixed with this command as part of script in YAML: git config --global --unset http.extraHeader but not early enough. (See next comment below).
I can't run this command early enough in the pipeline YAML to clear the header as checkout is not controlled by me.
It generally only happens if a previous run fails at some point on the same private agent
Syncing repository: test-project-azure-workspace (Git)
git version
git version 2.26.0
git lfs version
git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.4)
git config --get remote.origin.url
git clean -ffdx
git reset --hard HEAD
HEAD is now at 5f9fd24 sql mi
git config gc.auto 0
git config --get-all http.https://xxxxxxx#dev.azure.com/xxxxxxx/xxxxxxx/_git/test-project-azure-workspace.extraheader
git config --get-all http.proxy
git config http.version HTTP/1.1
git -c http.extraheader="AUTHORIZATION: bearer ***" -c http.proxy="http://10.XXX.XXX.XX:80" fetch --force --tags --prune --progress --no-recurse-submodules --unshallow origin
* Couldn't find host dev.azure.com in the .netrc file; using defaults
Here is the code that can remedy the symptom:
- script: |
echo '======================================================================'
echo 'list all of git config values for your convenience:'
echo '======================================================================'
git config --list
echo '======================================================================'
existing_header=$(git config --get http.extraHeader)
if [ ${#existing_header} -gt 0 ]
then
echo 'We found the http.extraHeader'
echo 'un-setting extra header: http.extraHeader 🔥'
git config --global --unset http.extraHeader
else
echo 'no extra header: http.extraHeader was not found. Nothing to unset 👍'
fi
condition: always()
workingDirectory: '$(Agent.BuildDirectory)/s'
displayName: 'Remove Git Authentication'
Obviously, I am treating the symptom and not the cause so any pointers as to what is causing this would be a great help.
You can create a "cleaning" job at the start of your pipeline that uses the checkout task with none as the repo to checkout. This will allow you to then run the cleaning script.
- job: cleanAgent
steps:
- checkout: none
- script: |
echo "Put your git cleaner here"
A better option is really to clean the agent after you are done by running a cleaning task with a condition of always() at the end of your pipeline. This can be hard if other people also use the agents and don't clean-up though.
- job: cleanupAgents
condition: always()
steps:
- script: |
echo "Put your git cleaner here"

How do we verify "npm login" succeeded and the token has not expired?

I've got a script that verifies a user has logged into a private npm registry (via "npm login") by greping for:
//registry-sub-url:_authToken=
... in:
~/.npmrc
However, this breaks down over time as the user's credentials expire
(due to standard password expiration rules).
What's more, the helper scripts I've created have cannot differentiate between successful/failed npm login calls, since the script always exits with 0 status.
Q: (1) How do we verify that npm login succeeded? (2) How do identify when the npm token has expired?
You might use npm whoami command.
$> npm whoami
${username}
$> npm logout
npm whoami
npm ERR! code ENEEDAUTH
npm ERR! need auth This command requires you to be logged in.
npm ERR! need auth You need to authorize this machine using `npm adduser`
npm ERR! A complete log of this run can be found in:
npm ERR! /xxxxx/.npm/_logs/2019-02-06T10_21_10_780Z-debug.logged
In the context of GitHub Packages: if you logged in using npm login --registry=https://npm.pkg.github.com, you can use npm whoami --registry=https://npm.pkg.github.com (same registry) to verify the currently logged in user.
$ npm login --registry=https://npm.pkg.github.com
Username: <your-user>
Password: <your-password-or-your-personal-access-token>
Email: (this IS public) <your#email.com>
Logged in as <your-user> on https://npm.pkg.github.com/.
$ npm whoami --registry=https://npm.pkg.github.com
<your-user>
If I only run npm whoami I get the error mentioned in the question.
I'm posting the workaround I came up with, but I'd love a better solution.
I've got Jenkins running this bash script periodically to test/verify my npm login against a private registry:
#/bin/bash
# Suppress commands (Jenkins turns this on)
set +x
# Suppress color codes from NPM output (for proper grepping)
export TERM=dumb
# Stop on any error
set -e
NPM_USERNAME=...
NPM_PASSWORD=...
NPM_URL=...
NPM_EMAIL=...
WORKSPACE=... (in my case, set by Jenkins)
echo "========"
echo "Looking for previous failed login (cached credentials)..."
echo ""
# NOTE: A previous failed login can result in an ".npmrc" containing
# a username/password in lieu of an auth token. We look for this and
# remove it (logout) if it exists so that the cached credentials are
# not applied when we run "expect" to login
# (which would see different prompts from cached credentials).
# Chop off "http:"/"https:" prefix from URL
NPM_REPO_PREFIX=`sed -e 's~https\{0,1\}:\(.*\)~\1~' <<< "$NPM_URL"`
# NOTE: piping to /dev/null so the password isn't printed
set +e
grep -F "${NPM_REPO_PREFIX}:_password=" ~/.npmrc > /dev/null
GREP_EXIT="$?"
set -e
if [[ "$GREP_EXIT" == "0" ]]; then
echo "========"
echo "Logging out of repo..."
echo ""
npm logout --registry "$NPM_URL"
fi
echo "========"
echo "Logging into repo..."
echo ""
(/usr/bin/expect <<EOF
set timeout 10
spawn npm login --verbose --registry "$NPM_URL"
match_max 100000
expect "Username"
send "$NPM_USERNAME\r"
expect "Password"
send "$NPM_PASSWORD\r"
expect "Email"
send "$NPM_EMAIL\r"
expect {
timeout exit 1
expect eof
}
EOF
) | tee "$WORKSPACE/npm-login.out"
echo "========"
echo "Verifying output of login..."
echo ""
# NOTE: If the login fails, the npm command still exits with status "0",
# so we read the verbose output to see that the http server confirms
# successful with "http 201".
set +e
grep "npm http 201" "$WORKSPACE/npm-login.out"
GREP_EXIT="$?"
set -e
if [[ "$GREP_EXIT" != "0" ]]; then
>&2 echo "========"
>&2 echo "ERROR: Failed to login to repo [$NPM_REPO]"
exit 1
else
echo "========"
echo "SUCCESS: Logged into [$NPM_REPO]"
fi
this problem is probably due to the fact that there is an .npmrc file in the npm whoami invocation file. if the token in that file is corrupt, this error appears. you can correct or delete the token value in the file and use the global token value.

Why is npm passing cmd line flags directly to my script?

I'm using npm to run a build and I'm trying to override my .npmrc config options using cmd line flags. However npm insists on passing these flags directly to my script rather than reading them as config options as described in the docs. What am I doing wrong?
From the cmd line I try to build a Tizen pacakge like so:
npm run package --tizen_profile myprofile
inside my package.json I have:
"package": "tizen package -t wgt --sign $npm_package_config_tizen_profile -- .buildResult/wgt -o .buildResult/wgt"
The result from running the command is:
package: `tizen package -t wgt --sign $npm_package_config_tizen_profile -- .buildResult/wgt -o .buildResult/wgt "myprofile"`
Where it should be:
package: `tizen package -t wgt --sign "myprofile"_tizen_profile -- .buildResult/wgt -o .buildResult/wgt`
It's like npm is merely appending the cmd line argument to the script command instead of plugging it in like a variable as described in the docs: https://docs.npmjs.com/misc/config
Has there been a recent update to npm which deprecates and removes this ability? I just updated to npm 6.x but it was working this way on 5.x as well.
you can try to rewrite your script within package.json without --sign flag like:
"package": "tizen package -t wgt -- .buildResult/wgt -o .buildResult/wgt"
and then pass it when you run npm command:
npm run package -- --sign myprofile
I assume that you can change the order of arguments, because --sign myprofile now will be at the very end of your command
UPDATE
here is another way to pass variables and place them in any place in your CLI command (without using npm config set). In my package.json:
"aaa": "ls $myoptionalflag && $mycmd"
this way I can pass any flag to la command (or not to pass at all) and I can pass any CLI command as mycmd variable. So, now I can run:
myoptionalflag=-la mycmd=pwd npm run aaa
which will execute
ls -la && pwd
or
mycmd=pwd npm run aaa
which will execute
ls && pwd
I FIGURED IT OUT!
The documentation is flawed as it doesn't tell you the correct syntax for passing npm config properties. I had to use:
npm run package --mypackagename:tizen_profile="myprofile"
where mypackagename is the name property used in package.json. Also note the key value syntax is --key=value and not --key value as described in the docs. Again, --key would be packagename:key using the name specified at the top level of your package.json.

How to enable nvm in steps in circleci 2.0?

Here are my steps in my
steps:
-run:
name: Setup nvm and npm
command: |
wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | bash
export NVM_DIR=$HOME/.nvm
source $NVM_DIR/nvm.sh
nvm install 8.9 && nvm alias default 8.9
-run: npm install && npm run lint && npm test
The second step always fails with this error message
/bin/bash: npm: command not found
I checked .bashrc and I can see the following lines are added to the end of the file
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" # This loads nvm bash_completion
Circleci 2.0 invokes the step command by starting a new shell with #!/bin/bash -eo pipefail
If I starts a docker (docker run -i -t buildpack-deps:xenial) and apply the first step, and then start a new shell via #!/bin/bash -eo pipefail, I can see npm is available on the path
I am using docker for this project
version: 2
jobs:
test_main:
docker:
- image: buildpack-deps:xenial
So why does it fail in circleci 2.0 environment? How can I ensure npm will be available to step 2 from step 1?
I have tried to add [ -s "$HOME/.bashrc" ] && \. "$HOME/.bashrc" to ~/.bash_profile (in case .bashrc is not executed due to the non-interactive/non-login shell)
To reproduce the issue you can run circleci build with this .circleci/config.yml file
version: 2
jobs:
build:
docker:
- image: buildpack-deps:xenial
steps:
- run:
name: Setup nvm and npm
command: |
wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | bash
# Activate nvm
export NVM_DIR=$HOME/.nvm
touch $HOME/.nvmrc
source $NVM_DIR/nvm.sh
# Use node 8.9
nvm install 8.9 && nvm alias default 8.9
echo 8.9 > $HOME/.nvmrc
# Enable nvm in following steps
echo '[ -s "$HOME/.bashrc" ] && \. "$HOME/.bashrc"' >> $HOME/.bash_profile
# To fix npm install : "node-pre-gyp: Permission denied"
npm config set user 0
npm config set unsafe-perm true
npm install -g npx webpack webpack-cli jest
node --version
npm --version
- run: npm install
You will see the following error message:
====>> npm install
#!/bin/bash -eo pipefail
npm install
/bin/bash: npm: command not found
Error: Exited with code 127
Step failed
Task failed
The problem lies with these lines:
# Enable nvm in following steps
echo '[ -s "$HOME/.bashrc" ] && \. "$HOME/.bashrc"' >> $HOME/.bash_profile
I was hoping to source .bashrc from .bash_profile. However since the shell of circleci is non-interactive, the environment variable PS1 is blank. Hence .bashrc basically quits immediately once it is sourced, because of this line in .bashrc
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
I have to put the following lines directly in the file specified by $BASH_ENV
echo 'export NVM_DIR=$HOME/.nvm' >> $BASH_ENV
echo 'source $NVM_DIR/nvm.sh' >> $BASH_ENV
I found that changing default node by nvm is not working for my steps.
Solved by:
- run:
name: 'Install Project Node'
command: |
set +x
source ~/.bashrc
nvm install 12
NODE_DIR=$(dirname $(which node))
echo "export PATH=$NODE_DIR:\$PATH" >> $BASH_ENV
Just source /opt/circleci/.nvm/nvm.sh in the beginning of every step.