I'm sure this is not the first question for BitBucket Pipeline and Digital Ocean, but I have gone through several similar posts without any luck.
pipelines:
default:
- step:
name: SSH to Digital Ocean and update docker image
script:
- ssh -i ~/.ssh/config root#xxx.xxx.xxx.xxx
- docker rm -f mycontainer
- docker image rm -f myrepo/imagename:tag
- docker pull myrepo/imagename:tag
- docker run --name mycontainer -p 12345:80 -d=true --restart=always myrepo/imagename:tag
services:
- docker
Here is the SSH Key in my BitBucket repository
Here is what the BitBucket Pipeline shows to me:
How can I resolve this?
This is not a key problem - it's that the Pipelines container does not act as a normal terminal, but ssh expects a terminal under normal operation. You should be able to pass the command(s) to be run as arguments to the SSH command: ssh -i /path/to/key user#host "docker rm -f mycontainer && docker image rm -f myrepo/imagename:tag" etc.
Related
So, I want to deploy my Gitlab pipelines onto a server with SSH. This is my script .gitlab-ci :
test_job:
stage: test
variables:
GIT_STRATEGY: none # Disable Gitlab auto clone
before_script:
- 'command -v ssh-agent > /dev/null || ( apk add --update openssh )'
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "${SSH_PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- ssh-add ~/.ssh/id_rsa
# Add server to known hosts
- ssh-keyscan ${VM_IPADDRESS} >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
# Verify that key has been registered
- ls ~/.ssh -al
# Verify server connection
- echo "Ping server"
- ping ${VM_IPADDRESS} -c 5
script:
# Pull Git project on remote server
- echo "Git clone from repository"
- ssh -o PreferredAuthentications=publickey ${SSH_USER}#${VM_IPADDRESS} "
rm -rf /tmp/src/${CI_PROJECT_NAME}/ &&
git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}#gitlab.my-domain.fr/user/project.git /tmp/src/${CI_PROJECT_NAME}/
"
$SSH_PRIVATE_KEY contains my private SSH key I use daily to connect on that server. It works perfectly in normal time. ${SSH_USER} and ${VM_IPADDRESS} contain my username and the server address. I already checked that all the values in these parameters are correct on worker.
This is the message I have when trying this script :
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
I'm quite stuck with this actually :(. Any help :) ?
Adding my public key id_rsa.pub to ssh authorized_keys file in the server has solved the problem for me. And you need to make sure of adding your public key to your SSH keys in your Gitlab profile.
Also, it's good to note that:
"Add the public key to the services that you want to have an access to from within the build environment. If you are accessing a private GitLab repository you must add it as a deploy key."
Using bitbucket pipelines to push to our remote from the build process that you get from the pipeline.
This is a snippet of the bitbucket-pipelines.yml file
- pipe: atlassian/ssh-run:0.2.2
variables:
SSH_USER: $PRODUCTION_USER
SERVER: $PRODUCTION_SERVER
COMMAND: '''rsync -zrSlh -e "ssh -p 22007" --stats --max-delete=0 $BITBUCKET_CLONE_DIR/ $PRODUCTION_USER#$PRODUCTION_SERVER:home/$PRODUCTION_USER'''
PORT: '22007'
The connection itself works, and it does run the command correctly once it is remoted onto the server...
INFO: Executing the pipe...
INFO: Using default ssh key
INFO: Executing command on {HOST}
ssh -A -tt -i /root/.ssh/pipelines_id -o StrictHostKeyChecking=no -p 22007 {USER}#{HOST} 'rsync -zrSlh -e "ssh -p 22007" --stats --max-delete=0 /opt/atlassian/pipelines/agent/build/ {USER}#{HOST}:home/{USER}'
bash: rsync -zrSlh -e "ssh -p 22007" --stats --max-delete=0 /opt/atlassian/pipelines/agent/build/ {USER}#{HOST}:home/{USER}: No such file or directory
Connection to {HOST} closed.
I've tried to run the same command locally from the directory on my machine
ssh -A -tt -i /root/.ssh/pipelines_id -o StrictHostKeyChecking=no -p 22007 {USER}#{HOST} 'rsync -zrSlh -e "ssh -p 22007" --stats --max-delete=0 "$PWD" {USER}#{HOST}:/home/{USER}'
but it just duplicates the home directory on the remote.
It looks to me like it's looking for the source directory on the server and not looking at the docker container from bitbucket (or the files on my local machine with pwd).
If I try to run the command without the '' then it fails because it's using port 22 by default. I've also tried offsetting the command into a bash script and using MODE: 'Script' which is an acceptable pattern for the plugin, but I can't use my environment variables in the sh file.
If all you wan't to do is to copy the files from the pipeline to the production server, you should you the rsync-deploy pipe, instead of the ssh-run. Your pipe configuration is gonna look pretty much like the following:
script:
- pipe: atlassian/rsync-deploy:0.3.2
variables:
USER: $PRODUCTION_USER
SERVER: $PRODUCTION_USER
REMOTE_PATH: 'home/$PRODUCTION_USER'
LOCAL_PATH: 'build'
SSH_PORT: '22007'
Make sure to configure your SSH keys in pipelines properly (here is a link to our docs for configuring SSH keys https://confluence.atlassian.com/bitbucket/use-ssh-keys-in-bitbucket-pipelines-847452940.html)
I've found another way around this instead of needing a plugin, instead I'm running an rsync as a script step
image: atlassian/default-image:latest
- rsync -rltDvzCh --max-delete=0 --stats --exclude-from=excludes -e 'ssh -e none -p 22007' $BITBUCKET_CLONE_DIR/ $PRODUCTION_USER#$PRODUCTION_SERVER:/home/$PRODUCTION_USER
It seems the -e none is an important addition, as is loading in the atlassian image, as fails to find the rsync function, otherwise. I found this info on this post on Atlassian Community.
This seems to work pretty well for me
image: node:10.15.3
pipelines:
default:
- step:
name: <project-path>
script:
- apt-get update && apt-get install -y rsync
- ssh-keyscan -H $SSH_HOST >> ~/.ssh/known_hosts
- cd $BITBUCKET_CLONE_DIR
- rsync -r -v -e ssh . $SSH_USER#$SSH_HOST:/<project-path>
- ssh $SSH_USER#$SSH_HOST 'cd <project-path> && npm install'
- ssh $SSH_USER#$SSH_HOST 'pm2 restart 0'
Note: Avoid using sudo cmd in pipeline scripts
same issue with atlassian/default-image:3
rsync -azv ./project_path/*
bash: rsync: command not found
Solution:
apt-get update && apt-get install -y rsync
Hello everyone i need your help plz, i'm using gitlab ci/cd and trying to deploy my .jar application to an ubuntu server, i configured my gitlab project with a windows runner with shell executor. i configured a key based access on the runner to avoid being prompt for a password;
the following command runs successfully when i login to the runner machine and use it's powershell :
scp -i C:\Users\Administrators\ssh\id_rsa myapp-0.0.1-SNAPSHOT.jar username#myubuntuserver:/
but when i'm using the above commande in my .yml file to copy the .jar on the server, it doesn't give any response until the job fail due to timeout
i tried also the solution proposed here https://docs.gitlab.com/ee/ci/ssh_keys/ by setting an SSH_PRIVATE_KEY variable on my project but i'm unable to adapt the given 'before_script' to my windows runner.
this is the before_script proposed in the documentation (above link):
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
when the previous scp command is replaced by this:
ssh -iv C:\Users\Administrators\ssh\id_rsa username#myubuntuserver
i get the following output:
the image
Thanks in advance
It works after doing the following steps:
1) configuring the runner (shell executor) on ubuntu 18.04
2) Then from the terminal login as the gitlab-runner user: sudo su - gitlab-runner
3) run ssh-keygen -t rsa
4) run ssh -i ~/.ssh/id_rsa username#myubuntuserver:
5) run cat ~/.ssh/id_rsa.pub | ssh username#myubuntuserver "mkdir -p ~/.ssh && touch ~/.ssh/authorized_keys && chmod -R go= ~/.ssh && cat >> ~/.ssh/authorized_keys"
5) now you can add the following to your job script (yml file) and it should work:
- scp -i ~/.ssh/id_rsa fileToCopy username#myubuntuserver:/mydirectory
#you can execute multiple commands at a time, for ex:
- ssh username#myubuntuserver " mv /mydirectory/myapp-0.0.1-SNAPSHOT.jar /mydirectory/myapp.jar "
Hope it will help
If ssh -iv C:\Users\Administrators\ssh\id_rsa username#myubuntuserver does not work, that may be because of the C: part, which confuses ssh into thinkig C is the name of the server!
A Unix-like path would work:
ssh -iv /C/Users/Administrators/ssh/id_rsa username#myubuntuserver
But, as the OP Medmahmoud comments, this supposes the public key has been published on the server:
Configure the runner on ubuntu18.04.
Then from the terminal login as the gitlab-runner user:
sudo su - gitlab-runner - run ssh-keygen -t rsa
ssh -i ~/.ssh/id_rsa username#myubuntuserver
cat ~/.ssh/id_rsa.pub | ssh username#myubuntuserver \
"mkdir -p ~/.ssh && touch ~/.ssh/authorized_keys && chmod -R go= ~/.ssh && cat >> ~/.ssh/authorized_keys"
Now from your yml file the following should work:
- scp -i ~/.ssh/id_rsa pom.xml username#myubuntuserver:/mydirectory
Situation:
shell gitlab runner, certificate configured, ssh connected as follows:
ssh-keygen --> id_rsa & id_rsa.pub
ssh-copy-id <user>#<remotehost>
ssh <user>#<remotehost> works as designed
id_rsa -> gitlab cicd variable called 'SSH_PRIVATE_KEY'
gitlab-ci as follows:
before_script:
- echo "Before script section"
# Install ssh-agent if not already installed, it is required by Docker.
# (change apt-get to yum if you use a CentOS-based image)
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# Run ssh-agent (inside the build environment)
- eval $(ssh-agent -s)
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add < ~/.ssh/id_rsa
- ssh-add -l
build1:
stage: build
script:
- echo "Pulling on Dev\n"
- ssh -A <user>#<remotehost>
- hostname
- ssh-agent bash -c 'hostname'
- ssh-agent bash -c 'awk "NR==1{print;exit}" /etc/php7/php.ini'
Complication:
when executing commands via gitlab-ci after the ssh connection, it seems to be executed on the gitlab machine. (php is installed on the ssh'ed system, not on gitlab)
See gitlab job output below:
...
eval $(ssh-agent -s)
Agent pid 1234
$ ssh-add < ~/.ssh/id_rsa
Identity added: /home/gitlab-runner/.ssh/id_rsa (/home/gitlab-runner/.ssh/id_rsa)
$ ssh-add -l
4096 SHA256:<KEY> /home/gitlab-runner/.ssh/id_rsa (RSA)
# same behaviour with ssh -T <user>#<ipaddress> -p <portnumber>
$ ssh -A <user>#<ipaddress> -p <portnumber>
Pseudo-terminal will not be allocated because stdin is not a terminal.
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
$ hostname
gitlab
$ ssh-agent bash -c 'hostname'
gitlab
$ ssh-agent bash -c 'awk "NR==1{print;exit}" /etc/php7/php.ini'
awk: cannot open /etc/php7/php.ini (No such file or directory)
In what way do I need to configure the system, so that the commands are actually run on the ssh'ed system?
I'm currently working with a solution which seems a bit too dirty for me.
In the gitlab-ci I'm pulling and running phpunit as follows
ssh -T <user>#<remotehost> "cd /var/www/projectfolder; git pull https://<gitlabUser>:$GITLAB_TOKEN#<privateGitlab>/<gitRepo>.git;"
ssh -T <user>#<remotehost> "cd /var/www/projectfolder/tests; phpunit;"
ie, I'm using a new ssh each time I'd like to run a command, which doesnt quite seem right to me. Any suggestions are welcome!
#til As per your suggestion request, single ssh command...
ssh -T <user>#<remotehost> "cd /var/www/projectfolder; git pull https://<gitlabUser>:$GITLAB_TOKEN#<privateGitlab>/<gitRepo>.git; cd /var/www/projectfolder/tests; phpunit;"
I added a deploy key with write access to my GitLab repository. My .gitlab-ci.yml file contains:
- git clone git#gitlab.domain:user/repo.git
- git checkout master
- git add myfile.pdf
- git commit -m "Generated PDF file"
- git push origin master
The deploy key works when cloning the repository.
Pushing is not possible, even if the deploy key has write access.
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#domain/user/repo.git/': The requested URL returned error: 403
I just encountered the same problem and saw this question without answer, so there is my solution.
Problem
The problem is caused by the fact that the remote url used by git to push the code is in the form http(s)://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#git.mydomain.com/group/project.git.
This url is using http(s) protocol so git doesn't use the ssh deploy key that you setup.
Solution
The solution is to change the push url of the remote origin so it matches ssh://git#git.mydomain.com/group/project.git.
The easiest way to do so is to use the predefined variable CI_REPOSITORY_URL.
Here is an example of code doing so by using sed:
# Change url from http(s) to ssh
url_host=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#|ssh://git#|g')
echo "${url_host}"
# ssh://git#git.mydomain.com/group/project.git
# Set the origin push url to the new one
git remote set-url --push origin "${url_host}"
Also, those using docker executor may want to verify the SSH host key as suggested by the gitlab documentation on deploy keys for docker executor.
So I give a more complete example for docker executor.
The code is mainly from gitlab documentation on ssh deploy keys.
In this example, the private deploy key is stored inside a variable named SSH_PRIVATE_KEY.
create:push:pdf:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "${SSH_PRIVATE_KEY}" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- git config --global user.email "email#example.com"
- git config --global user.name "User name"
- gitlab_hostname=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#||g' | sed -e 's|/.*||g')
- ssh-keyscan "${gitlab_hostname}" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- git checkout master
- git add myfile.pdf
- git commit -m "Generated PDF file"
- url_host=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#|ssh://git#|g')
- git remote set-url --push origin "${url_host}"
- git push origin master