Gitlab.com CI cannot logon to SSH server - ssh

Most of the repositories of my private projects are hosted on a private repository on gitlab.com (the hosted solution, not a privately hosted gitlab server). The sites are hosted on a digitalocean VPS.
I want to use gitlab CI to have every commit on the develop branch automatically deployed on the test server. Since I already have a clone of the repository on this test server the easiest way to automatically deploy seems to have gitlab-ci connect to the ssh server, and trigger a git pull.
The gitlab-ci.yml I have now (ssh before_script copied from http://docs.gitlab.com/ce/ci/ssh_keys/README.html).
deploy to test:
environment: test
only:
- develop
before_script:
# Install ssh-agent if not already installed, it is required by Docker.
# (change apt-get to yum if you use a CentOS-based image)
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
# disable host key checking (NOTE: makes you susceptible to man-in-the-middle attacks)
# WARNING: use only in docker container, if you use it with shell you will overwrite your user's ssh config
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
script:
# Try and connect to the test server
- ssh [myname]#[mydomain.com] "cd /var/www/test.[projectname].com/ && git pull"
The result of a commit on develop in the gitlab pipelines:
$ ssh [myname]#[mydomain.com] "cd /var/www/test.[projectname].com/ && git pull"
Warning: Permanently added '[mydomain.com],[255.255.255.255]' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
ERROR: Build failed: exit code 1
I have the private key of my local user on my laptop added to the SSH_PRIVATE_KEY variable on gitlab. The private key should work since I can connect to the server from my laptop without providing a password.
Does anyone have this working, how can the gitlab.com worker connect to the ssh server?

AFAIK, you can't do this:
# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
The ssh-agent is not getting the key context, nor the FD. You should store the key in some temporary file and then add it to the agent (and potentially remove the file, if it is not needed anymore):
# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
- echo "$SSH_PRIVATE_KEY" > key
- chmod 600 key
- ssh-add key
- rm key

Related

How do I resolve Invalid SSH Key Entry error when starting App with GCE

I'm trying to launch my app on Google Compute Engine, and I get the following error:
Sep 26 22:46:09 debian google_guest_agent[411]: ERROR non_windows_accounts.go:199 Invalid ssh key entry - unrecognized format: ssh-rsa AAAAB...
I'm having a hard time interpreting it. I have the following startup script:
# Talk to the metadata server to get the project id
PROJECTID=$(curl -s "http://metadata.google.internal/computeMetadata/v1/project/project-id" -H "Metadata-Flavor: Google")
REPOSITORY="github_sleepywakes_thunderroost"
# Install logging monitor. The monitor will automatically pick up logs sent to
# syslog.
curl -s "https://storage.googleapis.com/signals-agents/logging/google-fluentd-install.sh" | bash
service google-fluentd restart &
# Install dependencies from apt
apt-get update
apt-get install -yq ca-certificates git build-essential supervisor
# Install nodejs
mkdir /opt/nodejs
curl https://nodejs.org/dist/v16.15.0/node-v16.15.0-linux-x64.tar.gz | tar xvzf - -C /opt/nodejs --strip-components=1
ln -s /opt/nodejs/bin/node /usr/bin/node
ln -s /opt/nodejs/bin/npm /usr/bin/npm
# Get the application source code from the Google Cloud Repository.
# git requires $HOME and it's not set during the startup script.
export HOME=/root
git config --global credential.helper gcloud.sh
git clone https://source.developers.google.com/p/${PROJECTID}/r/${REPOSITORY} /opt/app/github_sleepywakes_thunderroost
# Install app dependencies
cd /opt/app/github_sleepywakes_thunderroost
npm install
# Create a nodeapp user. The application will run as this user.
useradd -m -d /home/nodeapp nodeapp
chown -R nodeapp:nodeapp /opt/app
# Configure supervisor to run the node app.
cat >/etc/supervisor/conf.d/node-app.conf << EOF
[program:nodeapp]
directory=/opt/app/github_sleepywakes_thunderroost
command=npm start
autostart=true
autorestart=true
user=nodeapp
environment=HOME="/home/nodeapp",USER="nodeapp",NODE_ENV="production"
stdout_logfile=syslog
stderr_logfile=syslog
EOF
supervisorctl reread
supervisorctl update
# Application should now be running under supervisor
My instance shows I have 2 public SSH keys. The second begins like this one in the error, but after about 12 characters it is different.
Any idea why this might be occurring?
Thanks in advance.
Once you deployed your VM instance, its a default setting that the SSH key isn't
configure yet, but you can also configure the SSH key upon deploying the VM instance.
To elaborate the answer of #JohnHanley, I tried to test in my environment.
Created a VM instance, verified the SSH configuration. As a default configuration there's no SSH key configured as I said earlier you can configure SSH key upon deploying the VM
Created a SSH key pair via CLI, you can use this link for instruction details
Navigate your VM instance, Turn off > EDIT > Security > Add Item > SSH key 1 - copy+paste generated SSH key pair > Save > Power ON VM instance
Then test the VM instance if accessible.
Documentation link How to Add SSH keys to project metadata.

Gitlab CI cannot connect with SSH to remote server using SSH key

So, I want to deploy my Gitlab pipelines onto a server with SSH. This is my script .gitlab-ci :
test_job:
stage: test
variables:
GIT_STRATEGY: none # Disable Gitlab auto clone
before_script:
- 'command -v ssh-agent > /dev/null || ( apk add --update openssh )'
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "${SSH_PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- ssh-add ~/.ssh/id_rsa
# Add server to known hosts
- ssh-keyscan ${VM_IPADDRESS} >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
# Verify that key has been registered
- ls ~/.ssh -al
# Verify server connection
- echo "Ping server"
- ping ${VM_IPADDRESS} -c 5
script:
# Pull Git project on remote server
- echo "Git clone from repository"
- ssh -o PreferredAuthentications=publickey ${SSH_USER}#${VM_IPADDRESS} "
rm -rf /tmp/src/${CI_PROJECT_NAME}/ &&
git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}#gitlab.my-domain.fr/user/project.git /tmp/src/${CI_PROJECT_NAME}/
"
$SSH_PRIVATE_KEY contains my private SSH key I use daily to connect on that server. It works perfectly in normal time. ${SSH_USER} and ${VM_IPADDRESS} contain my username and the server address. I already checked that all the values in these parameters are correct on worker.
This is the message I have when trying this script :
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
I'm quite stuck with this actually :(. Any help :) ?
Adding my public key id_rsa.pub to ssh authorized_keys file in the server has solved the problem for me. And you need to make sure of adding your public key to your SSH keys in your Gitlab profile.
Also, it's good to note that:
"Add the public key to the services that you want to have an access to from within the build environment. If you are accessing a private GitLab repository you must add it as a deploy key."

GitLab CI failing with SSH error loading key invalid format

I am starting to get into GitLab CI for my company. We have a PrestaShop, and I want automatic deployment to the web server after a Git push.
Unit testing will come later. At the moment I just need it to deal with putting a copy of the "/app" folder in the web root of the web server.
So this is what I have got...
before_script:
- apt-get update -qq
- apt-get install -qq git
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
deploy_test:
type: deploy
environment:
name: test
url: [test server domain]
script:
- ssh [user]#[server] -p [port] "cd [repo folder] && git checkout master && git pull origin master && exit"
- ssh [user]#[server] -p [port] "rsync -rzvh [repo /app folder] [web server root path]"
only:
- master
Recently, gitlab-runner has started failing with the error Error loading key "/dev/fd/63": invalid format.
Can you help me to solve that error?
FYI, I have my personal private key set as $SSH_PRIVATE_KEY environment var in GitLab - the public on the web server of course. SSH is enabled on the web server which has WHM and cPanel. And I pre-checked out a copy of master via cPanel on the web server into the [repo folder].
Originally, OpenSSH used the PKCS #1 format for RSA private keys. This format is not very secure, so newer versions have moved to a different format for storing private keys which is specific to OpenSSH. This is more secure, but it's not backwards compatible.
While it is possible to convert the keys with ssh-keygen, it would be far better for you to create a new key that you used only for deployments. That's a best practice because it separates your personal key from the deployments and means that if one is compromised, the other is not affected.
Since you'd need to create a new key anyway, you'd be better off using an Ed25519 key. Mozilla and others recommend this format of key because it is fast, secure, and easy to make constant time. You can create such a key with ssh-keygen -t ed25519 -f deployment-key, where deployment-key and deployment-key.pub will be the private and public keys.
If you're using CentOS 7 on the server, it does indeed support Ed25519 keys if it have been appropriately updated with patches, and whatever you're using on GitLab should also support it. You'll need to add the new public key to the remote server as with your personal key.
If you really want to continue to use this key, you should be able to export it with ssh-keygen -e -m PEM.
Did you check the $SSH_PRIVATE_KEY run on protected branches and tags pipeline only? if so you need to add your branch into protected. Setting->Repository->Protected Branch. or unchecked the option in Setting->CI/CD->Variables

How to make gitlab CI use ssh to clone a repository?

The company I work for has a private gitlab server that only supports ssh protocol when cloning a repository.
Inside this server, I have a gitlab-ci.yml file that uses docker executor to run some scripts.
The script's execution fails because it pulls the repository with https at its early stage. It generates this error message: fatal: unable to access 'https://gitlab.mycompany.com/path/to/the/repository/my_repo.git/': SSL certificate problem: unable to get local issuer certificate.
Where can I configure gitlab runner so that it uses ssh to clone the repository?
Here's the full execution log.
Running with gitlab-runner 12.7.1 (003fe500)
on my Group Runner Yh_yL3A2
Using Docker executor with image www.mycompany.com/path/to/the/image:1.0 ...
Pulling docker image www.mycompany.com/path/to/the/image:1.0 ...
Using docker image sha256:474e110ba44ddfje8ncoz4c44e91f2442547281192d4a82b88capmi9047cd8cb for www.mycompany.com/path/to/the/image:1.0 ...
Running on runner-Yh_yL3A2-project-343-concurrent-0 via b55d8c5ba21f...
Fetching changes...
Initialized empty Git repository in /path/to/the/repository/.git/
Created fresh repository.
fatal: unable to access 'https://gitlab.mycompany.com/path/to/the/repository/my_repo.git/': SSL certificate problem: unable to get local issuer certificate
ERROR: Job failed: exit code 1
Here's my .gitlab-ci.yml
image: www.mycompany.com/path/to/the/image:1.0
before_script:
- eval $(ssh-agent -s)
# Reference: https://docs.gitlab.com/ee/ci/ssh_keys/
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
# We're using tr to fix line endings which makes ed25519 keys work
# without extra base64 encoding.
# https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
#
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
#
# Create the SSH directory and give it the right permissions
#
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
stages:
- deploy
deploy:
stage: deploy
tags:
- infra
only:
refs:
- master
script:
- /bin/sh run.sh
I cannot find an option to specify whether the docker executor should use ssh or https to clone the repository.

Getting gitlab-runner 10.0.2 cloning repo using ssh

I have a gitlab installation and I am trying to setup a gitlab-runner using a docker executor. All ok until tests start running and then since my projects are private and they have no http access enabled, they fail at clone time with:
Running with gitlab-runner 10.0.2 (a9a76a50)
on Jupiter-docker (5f4ed288)
Using Docker executor with image fedora:26 ...
Using docker image sha256:1f082f05a7fc20f99a4ccffc0484f45e6227984940f2c57d8617187b44fd5c46 for predefined container...
Pulling docker image fedora:26 ...
Using docker image fedora:26 ID=sha256:b0b140824a486ccc0f7968f3c6ceb6982b4b77e82ef8b4faaf2806049fc266df for build container...
Running on runner-5f4ed288-project-5-concurrent-0 via 2705e39bc3d7...
Cloning repository...
Cloning into '/builds/pmatos/tob'...
remote: Git access over HTTP is not allowed
fatal: unable to access 'https://gitlab.linki.tools/pmatos/tob.git': The requested URL returned error: 403
ERROR: Job failed: exit code 1
I have looked into https://docs.gitlab.com/ee/ci/ssh_keys/README.html
and decided to give it a try so my .gitlab-ci.yml starts with:
image: fedora:26
before_script:
# Install ssh-agent if not already installed, it is required by Docker.
# (change apt-get to yum if you use a CentOS-based image)
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
# For Docker builds disable host key checking. Be aware that by adding that
# you are suspectible to man-in-the-middle attacks.
# WARNING: Use this only with the Docker executor, if you use it with shell
# you will overwrite your user's SSH config.
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
... JOBS...
I setup the SSH_PRIVATE_KEY correctly, etc but the issue is that the cloning of the project happens before before_script. I then tried to start the container with -v /home/pmatos/gitlab-runner_ssh:/root/.ssh but still the cloning is trying to use HTTP. How can I force the container to clone through ssh?
Due to the way gitlab CI works, CI requires https access to the repository. Therefore if you enable CI, you need to have https repo access enabled as well.
This is however, not an issue privacy wise as making the container https accessible doesn't stop gitlab from checking if you're authorized to access it.
I then tried to start the container with -v /home/pmatos/gitlab-runner_ssh:/root/.ssh but still the cloning is trying to use HTTP
Try at least if possible within your container to add a
git config --global url.ssh://git#.insteadOf https://
(assuming the ssh user is git)
That would make any clone of any https URL use ssh.