Gitlab CI/CD issue with SSH config file - ssh

I am trying to deploy my first project to my production server. Here is the script for the deployment stage:
deploy_production:
stage: deploy
script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "ssh -p 69" "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
- ./vendor/bin/envoy run deploy
environment:
name: production
when: manual
only:
- main
When I run the stage, I get this error :
[myServer#xxx.xxx.x.x]: /home/php/.ssh/config: line 1: Bad configuration option: ssh
[myServer#xxx.xxx.x.x]: /home/php/.ssh/config: terminating, 1 bad configuration options
[✗] This task did not complete successfully on one of your servers.
Why is it trying to access the SSH on this path :
/home/php/.ssh/config

Why is it trying to access the SSH on this path :
This should be related to the account used by gitlab-ci: it is supposed to look for SSH settings in $HOME/.ssh: display first what $HOME is.
If you look at the official documentation, you will see an SSH setup relies on proper rights associated to SSH folders/files:
efore_script:
##
## Install ssh-agent if not already installed, it is required by Docker.
## (change apt-get to yum if you use an RPM-based image)
##
- 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
##
## Run ssh-agent (inside the build environment)
##
- eval $(ssh-agent -s)
##
## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
## We're using tr to fix line endings which makes ed25519 keys work
## without extra base64 encoding.
## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
##
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
##
## Create the SSH directory and give it the right permissions
##
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
I mention before a chmod 400 my_private_key if you store a key in ~/.ssh.
And to be safe, I would add a chmod 600 ~/.ssh/config.
The point is: if the rights are to opened, SSH will refuse to operate.

Related

GitLab pipelines pushes to remote server but cannot ssh

My .gitlab-ci.yml looks like this:
build app:
stage: build
only:
- feature/ci-pipeline-job-v2
before_script:
- echo "before script"
- 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$GIT_URL" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- echo "HOST *" > ~/.ssh/config
- echo "StrictHostKeyChecking no" >> ~/.ssh/config
- git config user.email "user.villiers#main.com"
- git config user.name "user-main"
- git remote add acquia $GIT_URL
script:
- echo "running the script"
- git checkout -b feature/ci-pipeline-job-v2
- git push acquia feature/ci-pipeline-job-v2
after_script:
- echo "time to ssh"
- ssh maindecoupled.dev#maindecoupleddev.ssh.prod.acquia-sites.com "cd /var/www/html && ls -la && composer install && exit"
My pipeline gives a success, but when I look at the job result, I see a permission denied certainly from the after script.
The full result of the job is as follows:
$ echo "before script"
before script
$ command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )
$ eval $(ssh-agent -s)
Agent pid 12
$ echo "$SSH_PRIVATE_KEY" | ssh-add -
Identity added: (stdin) (userdevilliers#Norton-MacBook-Pro.local)
$ mkdir -p ~/.ssh
$ chmod 700 ~/.ssh
$ echo "$GIT_URL" >> ~/.ssh/known_hosts
$ chmod 644 ~/.ssh/known_hosts
$ echo "HOST *" > ~/.ssh/config
$ echo "StrictHostKeyChecking no" >> ~/.ssh/config
$ git config user.email "user.villiers#main.com"
$ git config user.name "user-main"
$ git remote add acquia $GIT_URL
$ echo "running the script"
running the script
$ git checkout -b feature/ci-pipeline-job-v2
Switched to a new branch 'feature/ci-pipeline-job-v2'
$ git push acquia feature/ci-pipeline-job-v2
Warning: Permanently added 'svn-23449.prod.hosting.acquia.com,22.222.22.222' (RSA) to the list of known hosts.
To svn-23449.prod.hosting.acquia.com:maindecoupled.git
b1f2c6ca..622cab3b feature/ci-pipeline-job-v2 -> feature/ci-pipeline-job-v2
Running after_script
00:02
Running after script...
$ echo "time to ssh"
time to ssh
$ ssh maindecoupled.dev#maindecoupleddev.ssh.prod.acquia-sites.com "cd /var/www/html && ls -la && composer install && exit"
Warning: Permanently added 'maindecoupleddev.ssh.prod.acquia-sites.com,11.11.111.111' (ECDSA) to the list of known hosts.
maindecoupled.dev#maindecoupleddev.ssh.prod.acquia-sites.com: Permission denied (publickey).
Cleaning up project directory and file based variables
00:01
Job succeeded
How am I able to push to the Acquia repo but have a public key error when its time to ssh?
Not sure how to go about from here.
How can I ssh into the remote server and cd into the intended directories?

Trouble with OpenVPN in Gitlab Pipeline

So I'm trying to connect to a VPN to my server in order to pull the project trough gitlab, this is my gitlab-ci.yml file:
image: cypress/base:10
stages:
- deployment
deploy:
stage: deployment
before_script:
## VPN
- which openvpn || (apt-get update -y -qq && apt-get install -y -qq openvpn) # Install openvpn if not available.
- cat <<< $CLIENT_OVPN > /etc/openvpn/client.conf # Move vpn config from gitlab variable to config file.
- cat <<< $VPN_U > /etc/openvpn/pass.txt # Move vpn user from gitlab variable to pass file.
- cat <<< $VPN_P >> /etc/openvpn/pass.txt # Move vpn password from gitlab variable to pass file.
- cat <<< "auth-user-pass /etc/openvpn/pass.txt" >> /etc/openvpn/client.conf # Tell vpn config to use password file.
- cat <<< "log /etc/openvpn/client.log" >> /etc/openvpn/client.conf # Tell vpn config to use log file.
- openvpn --config /etc/openvpn/client.conf --daemon # Start openvpn with config as a deamon.
- sleep 30s # Wait for some time so the vpn can connect before doing anything else.
- cat /etc/openvpn/client.log # Print the vpn log.
- ping -c "server IP" <IP> # Ping the server I want to deploy to. If not available this stops the deployment process.
##
## SSH
## Inspiration for gitlab from https://docs.gitlab.com/ee/ci/ssh_keys/
## Inpsiration for new key from https://www.thomas-krenn.com/de/wiki/OpenSSH_Public_Key_Authentifizierung_unter_Ubuntu
##
- which ssh-agent || (apt-get update -y -qq && apt-get install openssh-client -y -qq) # Install ssh-agent if not available.
- eval $(ssh-agent -s) # Run ssh-agent.
- mkdir -p ~/.ssh # Create ssh directory.
- cat <<< $SSH_PRIVATE_KEY > ~/.ssh/id_rsa # Move ssh key from gitlab variable to file.
- chmod 700 ~/.ssh/id_rsa # Set permissions so only I am allowed to access my ssh key.
- ssh-add # Add the key (no params -> default file name assumed).
- cat <<< $SSH_KNOWN_HOSTS_DMS > ~/.ssh/known_hosts # Add the servers SSH Key to known_hosts prevent man in the middle attack.
script:
- ssh root#"server IP" "cd ../var/www/html/"projetct-name" && git checkout master && git pull && exit"
only:
- main
I've been going around in circles for two days now, but right now I'm getting this feedback on the deploy job
$ cat <<< $CLIENT_OVPN > /etc/openvpn/client.conf
$ cat <<< $VPN_U > /etc/openvpn/pass.txt
$ cat <<< $VPN_P >> /etc/openvpn/pass.txt
$ cat <<< "auth-user-pass /etc/openvpn/pass.txt" >> /etc/openvpn/client.conf
$ cat <<< "log /etc/openvpn/client.log" >> /etc/openvpn/client.conf
$ openvpn --config /etc/openvpn/client.conf --daemon
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

how to execute commands via ssh shell runner from private gitlab to private server

Situation:
shell gitlab runner, certificate configured, ssh connected as follows:
ssh-keygen --> id_rsa & id_rsa.pub
ssh-copy-id <user>#<remotehost>
ssh <user>#<remotehost> works as designed
id_rsa -> gitlab cicd variable called 'SSH_PRIVATE_KEY'
gitlab-ci as follows:
before_script:
- echo "Before script section"
# Install ssh-agent if not already installed, it is required by Docker.
# (change apt-get to yum if you use a CentOS-based image)
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add < ~/.ssh/id_rsa
- ssh-add -l
build1:
stage: build
script:
- echo "Pulling on Dev\n"
- ssh -A <user>#<remotehost>
- hostname
- ssh-agent bash -c 'hostname'
- ssh-agent bash -c 'awk "NR==1{print;exit}" /etc/php7/php.ini'
Complication:
when executing commands via gitlab-ci after the ssh connection, it seems to be executed on the gitlab machine. (php is installed on the ssh'ed system, not on gitlab)
See gitlab job output below:
...
eval $(ssh-agent -s)
Agent pid 1234
$ ssh-add < ~/.ssh/id_rsa
Identity added: /home/gitlab-runner/.ssh/id_rsa (/home/gitlab-runner/.ssh/id_rsa)
$ ssh-add -l
4096 SHA256:<KEY> /home/gitlab-runner/.ssh/id_rsa (RSA)
# same behaviour with ssh -T <user>#<ipaddress> -p <portnumber>
$ ssh -A <user>#<ipaddress> -p <portnumber>
Pseudo-terminal will not be allocated because stdin is not a terminal.
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
$ hostname
gitlab
$ ssh-agent bash -c 'hostname'
gitlab
$ ssh-agent bash -c 'awk "NR==1{print;exit}" /etc/php7/php.ini'
awk: cannot open /etc/php7/php.ini (No such file or directory)
In what way do I need to configure the system, so that the commands are actually run on the ssh'ed system?
I'm currently working with a solution which seems a bit too dirty for me.
In the gitlab-ci I'm pulling and running phpunit as follows
ssh -T <user>#<remotehost> "cd /var/www/projectfolder; git pull https://<gitlabUser>:$GITLAB_TOKEN#<privateGitlab>/<gitRepo>.git;"
ssh -T <user>#<remotehost> "cd /var/www/projectfolder/tests; phpunit;"
ie, I'm using a new ssh each time I'd like to run a command, which doesnt quite seem right to me. Any suggestions are welcome!
#til As per your suggestion request, single ssh command...
ssh -T <user>#<remotehost> "cd /var/www/projectfolder; git pull https://<gitlabUser>:$GITLAB_TOKEN#<privateGitlab>/<gitRepo>.git; cd /var/www/projectfolder/tests; phpunit;"

gitlab-ci: installing multiple gitlab repos as npm modules with different ssh keys

I have a project on gitlab that must install two other gitlab projects as npm packages. When there was only one package, I had my .gitlab-ci.yml set up like this:
stages:
- lint
variables:
PROJECT_1_KEY: $PROJECT_1_KEY
lint:
stage: lint
image: node-chrome:latest
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- ssh-add <(echo "$PROJECT_1_KEY")
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- cd app
- npm install
This worked just fine.
However, trying to add in a second project, which requires its own deploy key, has been unsuccessful so far.
I've added a second env variable PROJECT_2_KEY to the variables section.
Thing I've tried:
Using ssh-add to add both keys
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- ssh-add <(echo "$PROJECT_1_KEY")
- ssh-add <(echo "$PROJECT_2_KEY")
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- cd app
- npm install
Building separate files, one for each deploy key, and adding them to an .ssh/config file
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
- echo "$PROJECT_1_KEY" > ~/.ssh/project_1
- echo "$PROJECT_2_KEY" > ~/.ssh/project_2
- echo -e "Host project_1\n\tHostName gitlab.com\n\tIdentityFile $HOME/.ssh/project_1" > ~/.ssh/config
- echo -e "Host project_2\n\tHostName gitlab.com\n\tIdentityFile $HOME/.ssh/project_2" >> ~/.ssh/config
- cd app
- npm install
Adding both keys to the same id_rsa file and adding gitlab.com to known_hosts
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
- echo "$PROJECT_1_KEY" >> ~/.ssh/id_rsa
- echo "$PROJECT_2_KEY" >> ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- touch ~/.ssh/known_hosts
- ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
- cd app
- npm install
I'm flying a bit blind. Is there a correct technique for what I'm trying to do?
I fought a similar battle just in a different context (PHP Composer requiring a private gitlab repo). I couldn't get the SSH based example to work in a way I was satisfied with so I opted to take advantage of composer setting that used a custom url as the reference for a defined dependency.
In PHP it looked like this:
"require": {
"foo/bar": "dev-master",
...
"repositories": [
{
"type": "vcs",
"url": "https://gitlab+deploy-token-1234:abc-def-ghijk#gitlab.com/path/to/repo.git"
}
]
So given the npm context, can you use the dependencies keyword to define the projects using git urls that contain the token data to authenticate?
"dependencies" : {
"foo/bar" : "https://gitlab+deploy-token-1234:abc-def-ghijk#gitlab.com/path/to/repo.git",
}
If you don't like include auth data in the committed package.json you might try omitting that and just use the raw git url. In some projects during the build I just do a straight git clone of another private project and it appears the build process has permission to clone without configuring anything. (I'm not entirely sure "who" the build process is authed as, but presumably the user who triggered the build?)

Gitlab CI - SSH Permission denied (publickey,password)

I've been trying to setup CD for my project. My Gitlab CI runner and my project will be on same server. I've followed https://docs.gitlab.com/ee/ci/examples/deployment/composer-npm-deploy.html but I keep getting SSH Permission denied (publickey,password). error. All my variables, private key and other variables set correctly in project settings.
I've created my ssh key with ssh-keygen -t rsa -C "my.email#example.com" -b 4096 command with no passphrase and set my PRODUCTION_PRIVATE_KEY variable with content of ~/.ssh/id_rsa file.
This is my gitlab-ci.yml:
stages:
- deploy
deploy_production:
stage: deploy
image: tetraweb/php
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$PRODUCTION_PRIVATE_KEY")
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- apt-get install rsync
script:
- ssh $PRODUCTION_SERVER_USER#$PRODUCTION_SERVER
- hostname
only:
- master
And this is output from Gitlab CI runner:
Running with gitlab-ci-multi-runner 9.2.0 (adfc387)
on ci-test (1eada8d0)
Using Docker executor with image tetraweb/php ...
Using docker image sha256:17692e06e6d33d8a421441bbe9adfda5b65c94831c6e64d7e69197e0b51833f8 for predefined container...
Pulling docker image tetraweb/php ...
Using docker image tetraweb/php ID=sha256:474f639dc349f36716fb98b193e6bae771f048cecc9320a270123ac2966b98c6 for build container...
Running on runner-1eada8d0-project-3287351-concurrent-0 via lamp-512mb-ams2-01...
Fetching changes...
HEAD is now at dfdb499 Update .gitlab-ci.yml
Checking out dfdb4992 as master...
Skipping Git submodules setup
$ which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 12
$ ssh-add <(echo "$PRODUCTION_PRIVATE_KEY")
Identity added: /dev/fd/63 (rsa w/o comment)
$ mkdir -p ~/.ssh
$ echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ apt-get install rsync
Reading package lists...
Building dependency tree...
Reading state information...
rsync is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
$ ssh $PRODUCTION_SERVER_USER#$PRODUCTION_SERVER
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '{MY_SERVER_IP}' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
ERROR: Job failed: exit code 1
Thanks in advance.
You need to add the public key to the server so it would be recognized as an authentication key. This is, paste the content of the public key corresponding to the private key you are using to the ~/.ssh/authorized_keys on the $PRODUCTION_SERVER.
This is the script that worked to me:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- mkdir -p ~/.ssh
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
- chmod 700 ~/.ssh/id_rsa
- eval "$(ssh-agent -s)"
- ssh-add ~/.ssh/id_rsa
- ssh-keyscan -t rsa 64.227.1.160 > ~/.ssh/known_hosts
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- chmod 644 ~/.ssh/known_hosts
And I had to unprotect the variable as well.
The following can be used alternatively
some_stage:
- eval $(ssh-agent -s)
- cd ~
- touch id.rsa
- echo "$SSH_PRIVATE_KEY" > id.rsa
- chmod 700 id.rsa
- ssh -o StrictHostKeyChecking=no -i id.rsa $SSH_USER#$SERVER
Something important too...
The permissions of the ~/.ssh/authorized_keys file should be 600.
It can also be due to restrictions on users you can ssh into.
In my case, on the server, I got the following tail -f /var/log/auth.log:
..
Sep 6 19:25:59 server-name sshd[7943]: User johndoe from WW.XX.YY.ZZ not allowed because none of user's groups are listed in AllowGroups
..
The solution consists in updating the AllowGroups directive on the server's file /etc/ssh/sshd_config:
AllowGroups janesmith johndoe
In our case, we were clueless until we add the flag -v to the SSH command (we knew the public key setup was OK because we were able to connect to this instance from our laptop using the private key).
We saw this :
debug1: Offering public key: ... RSA SHA256:... agent
95debug1: send_pubkey_test: no mutual signature algorithm
And understood the situation thanks to the two links below : our key was generated with RSA format which is considered legacy on up-to-date openssh versions.
https://confluence.atlassian.com/bitbucketserverkb/ssh-rsa-key-rejected-with-message-no-mutual-signature-algorithm-1026057701.html
https://transang.me/ssh-handshake-is-rejected-with-no-mutual-signature-algorithm-error/
So you have two solutions :
generate a new key using ed25519 format and setup the public key on your instance
use this extra flag below in your ssh command
It should be a temporary workaround :
ssh -o PubkeyAcceptedKeyTypes=+ssh-rsa -o StrictHostKeyChecking=no your_user#your_instance_url "your command"
I hope it can help you if you are reading this.
Regards!
Add the public key (corresponding to the private key) to authorized keys.
Just a new line with you pub key:
cat /root/.ssh/id_rsa.pub.pub >> /root/.ssh/authorized_keys
And also add the pub key to gitlab ssh keys section Profile > Keys