Windows 10 OpenSSH ssh-add "agent refused operation" (GitLab Windows Shared Runner) - ssh

I'm having a significant amount of trouble adding an SSH Key to ssh-agent on GitLab's Windows Shared Runner, to subsequently allow me to access private packages on GitLab.
The actual SSH private key is set using a "file" environment variable, but directly adding this via ssh-add states that the permissions are too open, so I researched methods to try rectify this - none of which work.
Instead, I echo the key into a manually created file. Only now, I receive the "agent refused operation" when adding the key.
To add to the fire, the ssh agent even refuses keys generated inside the VM for testing purposes.
The SSH keys are all ed25519 type.
Here is my gitlab-ci.yml script - what am I missing?
stages:
- test
test_windows:
stage: test
tags:
- shared-windows
- windows
- windows-1809
variables:
CI_ARTIFACTS_PATH_TMP: '"$CI_PROJECT_DIR"'
script:
# ------------------------------------------------------------------------
# Print the contents of some key directories
# ------------------------------------------------------------------------
- ls -l "C:\"
- ls -l "C:\Users\"
- ls -l "C:\Git\"
- ls -l "C:\Git\usr\bin\"
- ls -l "C:\Program Files\"
- ls -l "C:\Windows\System32\OpenSSH\"
# ------------------------------------------------------------------------
# Install dependencies/helpers
# ------------------------------------------------------------------------
# No steps required
# ------------------------------------------------------------------------
# Configure SSH (OpenSSH variant)
# ------------------------------------------------------------------------
- Get-Command ssh-keygen
- Get-Command ssh-add
- Get-Command ssh
- Set-Service -Name ssh-agent -StartupType Manual
- Start-Service ssh-agent
- Get-Service ssh-agent | select * # Check if it has started
# ------------------------------------------------------------------------
# Copy the DK_FILE ssh private key env variable into the id_ed25519 file (create if nonexistent)
# ------------------------------------------------------------------------
- $dk_file_txt = Get-Content $DK_FILE -Raw
- echo $dk_file_txt
- New-Item -ItemType "file" -Force -Path C:\Users\$env:UserName\.ssh\id_ed25519
- echo "$dk_file_txt" > C:\Users\$env:UserName\.ssh\id_ed25519
- (Get-Content C:\Users\$env:UserName\.ssh\id_ed25519 -Raw).Replace("`r`n","`n") | Set-Content C:\Users\$env:UserName\.ssh\id_ed25519 -Force
# ------------------------------------------------------------------------
# Set Key Permissions
# ------------------------------------------------------------------------
# :: Remove Inheritance ::
# - cmd /c icacls "C:\Users\gitlab_runner\.ssh" /c /t /inheritance:d
- cmd /c icacls C:\Users\$env:UserName\.ssh\id_ed25519 /c /t /inheritance:d
# :: Set Ownership to Owner ::
# - cmd /c icacls "C:\Users\gitlab_runner\.ssh" /c /t /grant %username%:F
- cmd /c icacls C:\Users\$env:UserName\.ssh\id_ed25519 /c /t /grant %username%:F
# :: Remove All Users, except for Owner ::
# - cmd /c icacls "C:\Users\gitlab_runner\.ssh" /c /t /remove Administrator BUILTIN\Administrators BUILTIN Everyone System Users
- cmd /c icacls C:\Users\$env:UserName\.ssh\id_ed25519 /c /t /remove Administrator BUILTIN\Administrators BUILTIN Everyone System Users
# :: Verify ::
# - cmd /c icacls "C:\Users\gitlab_runner\.ssh"
- cmd /c icacls C:\Users\$env:UserName\.ssh\id_ed25519
# ------------------------------------------------------------------------
# Add the SSH key via ssh-add
# ------------------------------------------------------------------------
- ssh-add C:\Users\$env:UserName\.ssh\id_ed25519
# ------------------------------------------------------------------------
# Test if it worked
# ------------------------------------------------------------------------
# - ssh -Tvvv git#gitlab.com
# - git config --global core.sshCommand "'C:\Windows\System32\OpenSSH\ssh.exe'"
# - npm install
# - npm run package-win
# ------------------------------------------------------------------------
# Upload artifacts (if any)
# ------------------------------------------------------------------------
# - gitlab-runner artifacts-uploader --verbose --id "${CI_JOB_ID}" --token "${CI_JOB_TOKEN}" --url "${CI_SERVER_URL}" --artifact-format zip --artifact-type archive --path $CI_ARTIFACTS_PATH_TMP || exit 1
# ------------------------------------------------------------------------
# Finish...
# ------------------------------------------------------------------------
- exit 0
artifacts:
paths:
- .ssh
exclude:
- node_modules
Note that I also tried a variant which used poshgit and set different aliases for ssh commands pointing to this installation.
When using the poshgit variant, the ssh private key was successfully added. However, when actually trying to run an ssh session I receive a “host key check failed” error. This then kills my npm install (using private repos as dependencies).
stages:
- test
test_windows:
stage: test
tags:
- shared-windows
- windows
- windows-1809
variables:
CI_ARTIFACTS_PATH_TMP: '"$CI_PROJECT_DIR"'
script:
# ------------------------------------------------------------------------
# Print the contents of some key directories
# ------------------------------------------------------------------------
- ls -l "C:\"
- ls -l "C:\Users\"
- ls -l "C:\Git\"
- ls -l "C:\Git\usr\bin\"
- ls -l "C:\Program Files\"
- ls -l "C:\Windows\System32\OpenSSH\"
# ------------------------------------------------------------------------
# Install dependencies/helpers
# ------------------------------------------------------------------------
# ------------------------------------------------------------------------
# Configure SSH (Poshgit variant)
# ------------------------------------------------------------------------
- choco install poshgit -y --limit-output --no-progress
- refreshenv
- Import-Module 'C:\tools\poshgit\dahlbyk-posh-git-9bda399\src\posh-git.psd1'
- refreshenv
- $env:PATH+=";C:\Git\usr\bin"
- Set-Alias ssh-keygen "$env:ProgramFiles\git\usr\bin\ssh-keygen.exe" # Provided by poshgit
- Set-Alias ssh-agent "C:\Git\usr\bin\ssh-agent.exe"
- Set-Alias ssh-add "C:\Git\usr\bin\ssh-add.exe"
- Set-Alias ssh "C:\Git\usr\bin\ssh.exe"
- Start-SshAgent -Quiet
- echo "$env"
- gci env:SSH_AUTH_SOCK
- mkdir .ssh
- Get-WindowsCapability -Online | ? Name -like 'OpenSSH*'
# - ssh-keygen -f "$CI_PROJECT_DIR\.ssh\test_rsa" -t rsa -N '""'
# - ssh-keygen -f "$CI_PROJECT_DIR\.ssh\test_ed25519" -t ed25519 -N '""'
- New-Item -ItemType "directory" -Force -Path C:\Users\$env:UserName\.ssh\
- ssh-keygen -a 100 -t ed25519 -f "C:\Users\$env:UserName\.ssh\id_ed25519" -C "MY_SSH_KEY" -N '""'
- Remove-Item -Path "C:\Users\$env:UserName\.ssh\id_ed25519.pub"
# - Set-Content -Path "C:\Users\$env:UserName\.ssh\id_ed25519" -Value $dk_file_txt -Force
- ssh-add "C:\Users\$env:UserName\.ssh\id_ed25519"
# ------------------------------------------------------------------------
# Test if it worked
# ------------------------------------------------------------------------
# - ssh -o StrictHostKeyChecking=no git#gitlab.com uptime
- ls -l "$env:ProgramFiles\"
- ls -l "$env:ProgramFiles\git\"
- ls -l "$env:ProgramFiles\git\bin"
# - Set-Alias -Name git -Value "$env:ProgramFiles\git\bin\git.exe"
# - npm config set git "$env:ProgramFiles\git\bin\git.exe"
# - git config --global core.sshCommand "$env:ProgramFiles\git\usr\bin\ssh.exe"
- ssh -Tvvv git#gitlab.com
# - npm install # Fails with error code 128 (ssh failure)
# - npm run package-win
# ------------------------------------------------------------------------
# Upload artifacts (if any)
# ------------------------------------------------------------------------
- gitlab-runner artifacts-uploader --verbose --id "${CI_JOB_ID}" --token "${CI_JOB_TOKEN}" --url "${CI_SERVER_URL}" --artifact-format zip --artifact-type archive --path $CI_ARTIFACTS_PATH_TMP || exit 1
# ------------------------------------------------------------------------
# Finish...
# ------------------------------------------------------------------------
- exit 0
artifacts:
paths:
- .ssh
exclude:
- node_modules
Ideally, I’d prefer to work with openssh and not have to add external dependencies. I’d appreciate any and all help as I’ve lost so much time on this matter.

are private key permissions 600, and public key 644?
chmod 600 ~/.ssh/id_ed25519 && chmod 644 ~/.ssh/id_ed25519.pub

Related

GitLab pipelines pushes to remote server but cannot ssh

My .gitlab-ci.yml looks like this:
build app:
stage: build
only:
- feature/ci-pipeline-job-v2
before_script:
- echo "before script"
- 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$GIT_URL" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
- echo "HOST *" > ~/.ssh/config
- echo "StrictHostKeyChecking no" >> ~/.ssh/config
- git config user.email "user.villiers#main.com"
- git config user.name "user-main"
- git remote add acquia $GIT_URL
script:
- echo "running the script"
- git checkout -b feature/ci-pipeline-job-v2
- git push acquia feature/ci-pipeline-job-v2
after_script:
- echo "time to ssh"
- ssh maindecoupled.dev#maindecoupleddev.ssh.prod.acquia-sites.com "cd /var/www/html && ls -la && composer install && exit"
My pipeline gives a success, but when I look at the job result, I see a permission denied certainly from the after script.
The full result of the job is as follows:
$ echo "before script"
before script
$ command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )
$ eval $(ssh-agent -s)
Agent pid 12
$ echo "$SSH_PRIVATE_KEY" | ssh-add -
Identity added: (stdin) (userdevilliers#Norton-MacBook-Pro.local)
$ mkdir -p ~/.ssh
$ chmod 700 ~/.ssh
$ echo "$GIT_URL" >> ~/.ssh/known_hosts
$ chmod 644 ~/.ssh/known_hosts
$ echo "HOST *" > ~/.ssh/config
$ echo "StrictHostKeyChecking no" >> ~/.ssh/config
$ git config user.email "user.villiers#main.com"
$ git config user.name "user-main"
$ git remote add acquia $GIT_URL
$ echo "running the script"
running the script
$ git checkout -b feature/ci-pipeline-job-v2
Switched to a new branch 'feature/ci-pipeline-job-v2'
$ git push acquia feature/ci-pipeline-job-v2
Warning: Permanently added 'svn-23449.prod.hosting.acquia.com,22.222.22.222' (RSA) to the list of known hosts.
To svn-23449.prod.hosting.acquia.com:maindecoupled.git
b1f2c6ca..622cab3b feature/ci-pipeline-job-v2 -> feature/ci-pipeline-job-v2
Running after_script
00:02
Running after script...
$ echo "time to ssh"
time to ssh
$ ssh maindecoupled.dev#maindecoupleddev.ssh.prod.acquia-sites.com "cd /var/www/html && ls -la && composer install && exit"
Warning: Permanently added 'maindecoupleddev.ssh.prod.acquia-sites.com,11.11.111.111' (ECDSA) to the list of known hosts.
maindecoupled.dev#maindecoupleddev.ssh.prod.acquia-sites.com: Permission denied (publickey).
Cleaning up project directory and file based variables
00:01
Job succeeded
How am I able to push to the Acquia repo but have a public key error when its time to ssh?
Not sure how to go about from here.
How can I ssh into the remote server and cd into the intended directories?

Trouble with OpenVPN in Gitlab Pipeline

So I'm trying to connect to a VPN to my server in order to pull the project trough gitlab, this is my gitlab-ci.yml file:
image: cypress/base:10
stages:
- deployment
deploy:
stage: deployment
before_script:
## VPN
- which openvpn || (apt-get update -y -qq && apt-get install -y -qq openvpn) # Install openvpn if not available.
- cat <<< $CLIENT_OVPN > /etc/openvpn/client.conf # Move vpn config from gitlab variable to config file.
- cat <<< $VPN_U > /etc/openvpn/pass.txt # Move vpn user from gitlab variable to pass file.
- cat <<< $VPN_P >> /etc/openvpn/pass.txt # Move vpn password from gitlab variable to pass file.
- cat <<< "auth-user-pass /etc/openvpn/pass.txt" >> /etc/openvpn/client.conf # Tell vpn config to use password file.
- cat <<< "log /etc/openvpn/client.log" >> /etc/openvpn/client.conf # Tell vpn config to use log file.
- openvpn --config /etc/openvpn/client.conf --daemon # Start openvpn with config as a deamon.
- sleep 30s # Wait for some time so the vpn can connect before doing anything else.
- cat /etc/openvpn/client.log # Print the vpn log.
- ping -c "server IP" <IP> # Ping the server I want to deploy to. If not available this stops the deployment process.
##
## SSH
## Inspiration for gitlab from https://docs.gitlab.com/ee/ci/ssh_keys/
## Inpsiration for new key from https://www.thomas-krenn.com/de/wiki/OpenSSH_Public_Key_Authentifizierung_unter_Ubuntu
##
- which ssh-agent || (apt-get update -y -qq && apt-get install openssh-client -y -qq) # Install ssh-agent if not available.
- eval $(ssh-agent -s) # Run ssh-agent.
- mkdir -p ~/.ssh # Create ssh directory.
- cat <<< $SSH_PRIVATE_KEY > ~/.ssh/id_rsa # Move ssh key from gitlab variable to file.
- chmod 700 ~/.ssh/id_rsa # Set permissions so only I am allowed to access my ssh key.
- ssh-add # Add the key (no params -> default file name assumed).
- cat <<< $SSH_KNOWN_HOSTS_DMS > ~/.ssh/known_hosts # Add the servers SSH Key to known_hosts prevent man in the middle attack.
script:
- ssh root#"server IP" "cd ../var/www/html/"projetct-name" && git checkout master && git pull && exit"
only:
- main
I've been going around in circles for two days now, but right now I'm getting this feedback on the deploy job
$ cat <<< $CLIENT_OVPN > /etc/openvpn/client.conf
$ cat <<< $VPN_U > /etc/openvpn/pass.txt
$ cat <<< $VPN_P >> /etc/openvpn/pass.txt
$ cat <<< "auth-user-pass /etc/openvpn/pass.txt" >> /etc/openvpn/client.conf
$ cat <<< "log /etc/openvpn/client.log" >> /etc/openvpn/client.conf
$ openvpn --config /etc/openvpn/client.conf --daemon
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

Gitlab CI/CD issue with SSH config file

I am trying to deploy my first project to my production server. Here is the script for the deployment stage:
deploy_production:
stage: deploy
script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "ssh -p 69" "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
- ./vendor/bin/envoy run deploy
environment:
name: production
when: manual
only:
- main
When I run the stage, I get this error :
[myServer#xxx.xxx.x.x]: /home/php/.ssh/config: line 1: Bad configuration option: ssh
[myServer#xxx.xxx.x.x]: /home/php/.ssh/config: terminating, 1 bad configuration options
[✗] This task did not complete successfully on one of your servers.
Why is it trying to access the SSH on this path :
/home/php/.ssh/config
Why is it trying to access the SSH on this path :
This should be related to the account used by gitlab-ci: it is supposed to look for SSH settings in $HOME/.ssh: display first what $HOME is.
If you look at the official documentation, you will see an SSH setup relies on proper rights associated to SSH folders/files:
efore_script:
##
## Install ssh-agent if not already installed, it is required by Docker.
## (change apt-get to yum if you use an RPM-based image)
##
- 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
##
## Run ssh-agent (inside the build environment)
##
- eval $(ssh-agent -s)
##
## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
## We're using tr to fix line endings which makes ed25519 keys work
## without extra base64 encoding.
## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
##
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
##
## Create the SSH directory and give it the right permissions
##
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
I mention before a chmod 400 my_private_key if you store a key in ~/.ssh.
And to be safe, I would add a chmod 600 ~/.ssh/config.
The point is: if the rights are to opened, SSH will refuse to operate.

gitlab runner ssh private key 644 file permission error

When running a gitlab ci/cd pipeline, ssh gives 0644 bad permission error. Variable is stored as a file type in the settings>variable section in gitlab.
.gitlab-ci.yml file looks like:
stages:
- deploy
before_script:
- apt-get update -qq
- apt-get install -qq git
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
deploy_1:
stage: deploy
only:
- master
tags:
- master
script:
- ssh -i $SSH_KEY user#ip "mkdir -p runner_test"
deploy_2:
stage: deploy
only:
- master
tags:
- master
script:
- ssh -i $SSH_KEY user#ip "mkdir -p runner_test"
Error:
$ ssh -i $SSH_KEY host#ip "mkdir -p runner_test"
###########################################################
# WARNING: UNPROTECTED PRIVATE KEY FILE! #
###########################################################
Permissions 0644 for '/home/user/builds/gPnQDT8L/0/username/server.tmp/SSH_KEY' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/home/user/builds/gPnQDT8L/0/username/server.tmp/SSH_KEY": bad permissions
user#ip: Permission denied (publickey).
Cleaning up file based variables
How do I change the private key permissions from 644 to 600 or 400.
You can see the same error in this deploy process for this gitlab-ci.yml
The fixed version of that file:
server:
stage: deploy
script:
- apt-get install -y openssh-client rsync
- chmod 400 $SSH_KEY
- scp -o StrictHostKeyChecking=no -P $SSH_PORT -i $SSH_KEY public/server.zip $SSH_URI:modpack/server.zip
A simple chmod 400 $SSH_KEY should be enough.

GitLab CI / CD deploy script via SSH not working -> UNPROTECTED PRIVATE KEY FILE

I've a problem with my GitLab CI / CD pipeline: It's not connecting to my server during the deployment.
I've followed the instructions on the GitLab page and created a key pair for my server locally and tried it out - works perfectly.
Now I've switched to GitLab and created a file variable with the content of my private key file:
After that I've added a deployment section to my .gitlab-ci.yml file:
stages:
- deploy
deploy:
stage: deploy
before_script:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- touch ~/.ssh/known_hosts
- ssh-keyscan 136.xxx.xxx.xx >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- echo "Deploying to server..."
- ssh -i $IDENTITY ftp#136.xxx.xxx.xx "echo Hallo"
only:
- master
But when I execute the script, I'm getting this error:
$ ssh -i $IDENTITY ftp#136.xxx.xxx.xx "echo Hallo"
###########################################################
# WARNING: UNPROTECTED PRIVATE KEY FILE! #
###########################################################
Permissions 0666 for '/builds/john/test-website.tmp/IDENTITY' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/builds/john/test-website.tmp/IDENTITY": bad permissions
Permission denied, please try again.
Permission denied, please try again.
ftp#136.xxx.xxx.xx: Permission denied (publickey,password).
ERROR: Job failed: exit code 1
What I'm doing wrong here? I don't get it.
Thanks to VonC. This is how I solved the problem with his help:
First I've changed the variable from file to variable. After that I've modified my deploy script:
deploy:
stage: deploy
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- touch ~/.ssh/known_hosts
- ssh-keyscan 136.xxx.xxx.xx >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- echo "Deploying to server..."
- cd /builds/john/test-website/frontend/
- ls
- ssh ftp#136.xxx.xxx.xx "ls"
only:
- master
You might want to consider a custom variable of type Variable instead of type file.
That way, GitLab won't create a temporary file with the wrong permission.
But your pipeline can:
create the relevant file (with the right permission 600),
use it in ssh -i, and
delete it immediately.