gitlab-ci: installing multiple gitlab repos as npm modules with different ssh keys - npm

I have a project on gitlab that must install two other gitlab projects as npm packages. When there was only one package, I had my .gitlab-ci.yml set up like this:
stages:
- lint
variables:
PROJECT_1_KEY: $PROJECT_1_KEY
lint:
stage: lint
image: node-chrome:latest
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- ssh-add <(echo "$PROJECT_1_KEY")
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- cd app
- npm install
This worked just fine.
However, trying to add in a second project, which requires its own deploy key, has been unsuccessful so far.
I've added a second env variable PROJECT_2_KEY to the variables section.
Thing I've tried:
Using ssh-add to add both keys
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- ssh-add <(echo "$PROJECT_1_KEY")
- ssh-add <(echo "$PROJECT_2_KEY")
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- cd app
- npm install
Building separate files, one for each deploy key, and adding them to an .ssh/config file
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
- echo "$PROJECT_1_KEY" > ~/.ssh/project_1
- echo "$PROJECT_2_KEY" > ~/.ssh/project_2
- echo -e "Host project_1\n\tHostName gitlab.com\n\tIdentityFile $HOME/.ssh/project_1" > ~/.ssh/config
- echo -e "Host project_2\n\tHostName gitlab.com\n\tIdentityFile $HOME/.ssh/project_2" >> ~/.ssh/config
- cd app
- npm install
Adding both keys to the same id_rsa file and adding gitlab.com to known_hosts
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
- echo "$PROJECT_1_KEY" >> ~/.ssh/id_rsa
- echo "$PROJECT_2_KEY" >> ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- touch ~/.ssh/known_hosts
- ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
- cd app
- npm install
I'm flying a bit blind. Is there a correct technique for what I'm trying to do?

I fought a similar battle just in a different context (PHP Composer requiring a private gitlab repo). I couldn't get the SSH based example to work in a way I was satisfied with so I opted to take advantage of composer setting that used a custom url as the reference for a defined dependency.
In PHP it looked like this:
"require": {
"foo/bar": "dev-master",
...
"repositories": [
{
"type": "vcs",
"url": "https://gitlab+deploy-token-1234:abc-def-ghijk#gitlab.com/path/to/repo.git"
}
]
So given the npm context, can you use the dependencies keyword to define the projects using git urls that contain the token data to authenticate?
"dependencies" : {
"foo/bar" : "https://gitlab+deploy-token-1234:abc-def-ghijk#gitlab.com/path/to/repo.git",
}
If you don't like include auth data in the committed package.json you might try omitting that and just use the raw git url. In some projects during the build I just do a straight git clone of another private project and it appears the build process has permission to clone without configuring anything. (I'm not entirely sure "who" the build process is authed as, but presumably the user who triggered the build?)

Related

Gitlab CI/CD issue with SSH config file

I am trying to deploy my first project to my production server. Here is the script for the deployment stage:
deploy_production:
stage: deploy
script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "ssh -p 69" "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
- ./vendor/bin/envoy run deploy
environment:
name: production
when: manual
only:
- main
When I run the stage, I get this error :
[myServer#xxx.xxx.x.x]: /home/php/.ssh/config: line 1: Bad configuration option: ssh
[myServer#xxx.xxx.x.x]: /home/php/.ssh/config: terminating, 1 bad configuration options
[✗] This task did not complete successfully on one of your servers.
Why is it trying to access the SSH on this path :
/home/php/.ssh/config
Why is it trying to access the SSH on this path :
This should be related to the account used by gitlab-ci: it is supposed to look for SSH settings in $HOME/.ssh: display first what $HOME is.
If you look at the official documentation, you will see an SSH setup relies on proper rights associated to SSH folders/files:
efore_script:
##
## Install ssh-agent if not already installed, it is required by Docker.
## (change apt-get to yum if you use an RPM-based image)
##
- 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
##
## Run ssh-agent (inside the build environment)
##
- eval $(ssh-agent -s)
##
## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
## We're using tr to fix line endings which makes ed25519 keys work
## without extra base64 encoding.
## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
##
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
##
## Create the SSH directory and give it the right permissions
##
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
I mention before a chmod 400 my_private_key if you store a key in ~/.ssh.
And to be safe, I would add a chmod 600 ~/.ssh/config.
The point is: if the rights are to opened, SSH will refuse to operate.

The problem when retrieving data when I try to perform an automatic deployment through gitlab

I made a deploy script via ssh and gitlab but when the git pull script is executed, everything appears as already up to date and I can't even run the composer commands
before_script:
- apt-get update -qq
- apt-get install -qq git
# Setup SSH deploy keys
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
deploy_staging:
type: deploy
environment:
name: staging
url: test.ro
script:
- ssh -p 28785 test#test "git checkout development && git pull"
- cd server
- composer i
- composer optimize
- php artisan migrate
- cd ..
- cd client
- npm i
- npm run dev
- exit
only:
- development
$ apt-get update -qq
$ apt-get install -qq git
$ which ssh-agent || ( apt-get install -qq openssh-client )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 266
$ ssh-add <(echo "$SSH_PRIVATE_KEY")
Identity added: /dev/fd/63 (/dev/fd/63)
$ mkdir -p ~/.ssh
$ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ ssh -p 28785
$ git pull
$ Already up to date
Don't I make the connection via ssh ok?
Try first to replace your ssh step with:
ssh -p 28785 test#test "id -a && ls -alrth && pwd && git status && git remote -v"
That way, you can make sure you are:
in the right path (which would be by default /home/test, an odd path for a Git view)
in an actual local Git repository, and the right branch (hence the git status)
referencing the right remote (meaning the remote repository where you have pushed new commits, which should be pulled)

Gitlab CI - Composer not found with ssh

When connecting with ssh to my webserver I try to install composer dependencies after git pull. But the pipeline fails saying bash: composer: command not found. But there is definitely composer installed on the server.
I can also mount my docker image and try it directly with no problem!
Does anyone has an Idea why composer is not found?
Here the .gitlab-ci.yml:
stages:
- deploy
deploy_stage:
stage: deploy
image: parallactic/php-node:php8.0-node14.x
before_script:
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d)
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- ssh ${DEPLOY_USER}#${DEPLOY_SERVER} "cd ${DEPLOY_DIR} && git pull origin main"
- ssh ${DEPLOY_USER}#${DEPLOY_SERVER} "cd ${DEPLOY_DIR} && composer install --no-interaction --prefer-dist --optimize-autoloader"
only:
- main
Thanks for any suggestions!
it may be that your DEPLOY_USER doesn't have composer in the PATH. you can check that with echo $PATH in this deploy script, and then manually connect with your user and compare the two.
but also you can specify full path to the composer (eg. /usr/local/bin/composer install ...)
btw, why separate ssh connections? you can do it in one line
ssh ${DEPLOY_USER}#${DEPLOY_SERVER} "cd ${DEPLOY_DIR} && git pull origin main && /usr/local/bin/composer install --no-interaction --prefer-dist --optimize-autoloader"

Deploy Vue.js build with Gitlab CI

This is my gitlab pipeline. The Vue.js artifacts are build on the runner. How can I deploy the to my testserver? FYI: Fab pull does a git pull on the repo.
deploy_staging:
image: python:3.6
stage: deploy
only:
- master
before_script:
- curl -sL https://deb.nodesource.com/setup_13.x | bash -
- apt-get update -y
- apt-get install -y curl git gnupg nodejs
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- |
cat >~/.ssh/config <<EOF
Host testserver
ForwardAgent yes
HostName dev.testserver.ts
User testuser
EOF
- cat ~/.ssh/config
script:
- pip install -r requirements.txt
- npm install
- npm run production
- fab pull
Since you want to copy files from GitLab runner into your server, This will be possible using scp command.
For example:
⋮
script:
- pip install -r requirements.txt
- npm install
- npm run production
- scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no /PATH/TO/BUILD_ARTIFACTS testserver:~/PATH/TO/DESTINATION
- fab pull
UserKnownHostsFile and StrictHostKeyChecking are SSH options that prevent error Host key verification failed. So they should be used with scp command in your case.
Also, destination path of artifact files must be started from testuser's home directory (Tilde character ~). Otherwise you may face Permission denied error.

run npm install on private gitlab project from gitlab-ci

I have a private(with ssh) project in gitlab(that wasn't published to npm)
what is the right way to install this project from gitlab-CI?
I get this error:'Host key verification failed.' from gitlab-ci stdout
If the repo you try to access is not the one the deployment runs...
Add a ssh public-key to your deploy-docker-container via gitlab-ci.yml like this
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- mkdir -p ~/.ssh
- eval $(ssh-agent -s)
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
- ssh-add <(echo "$SSH_PRIVATE_KEY")
With that you can put your private ssh key into a variable in GitLab Project Configuration.
The public key you place inside the private repo Deploy Keys (you find this in the Repo Settings depending on the GitLab version you have).