I have CI setup, that deploy changes to the server. Everything works perfect, changes are pulled to server, but when all tasks are ended, runner still waiting:
What is wrong? It should be finished with success.
Here is the .gitlab-ci.yml:
stages:
- deploy
before_script:
# Setup SSH deploy keys
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 --decode)
deploy_staging:
type: deploy
environment:
name: staging
url: example.com
script:
- ssh -o StrictHostKeyChecking=no root#example.com "cd public_html/gitlab-test && git checkout master && git pull origin master && exit"
only:
- master
Update:
Output:
Running with gitlab-runner 11.6.1 (8d829975)
on Shared heeGPy6w
Using Shell executor...
Running on demeter...
Fetching changes...
HEAD is now at 4eaccda Update .gitlab-ci.yml
From https://git.example.com/user/ssh-test
4eaccda..ce1729c master -> origin/master
Checking out ce1729c4 as master...
Skipping Git submodules setup
$ which ssh-agent || ( apt-get install -qq openssh-client )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 14151
$ ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 --decode)
Identity added: /dev/fd/63 (/dev/fd/63)
$ ssh -o StrictHostKeyChecking=no root#example.com "cd public_html/gitlab-test && git checkout master && git pull origin master"
Already on 'master'
From https://git.example.com/user/ssh-test
* branch master -> FETCH_HEAD
4eaccda..ce1729c master -> origin/master
Updating 4eaccda..ce1729c
Fast-forward
.gitlab-ci.yml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
$ exit 0
and after this, still waiting...
Finally, I resolved my problem.
Reason was in line - eval $(ssh-agent -s) - when I commented it, the job could be finished (but of course, connection didn't work). So, I attempted add killing command at the end of script:
- eval $(ssh-agent -k)
It was a solution. Now everything works excellent.
Finally code:
stages:
- deploy
before_script:
# Setup SSH deploy keys
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent)
- ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 --decode)
deploy_staging:
type: deploy
environment:
name: staging
url: example.com
script:
- ssh -o StrictHostKeyChecking=no root#example.com "cd public_html/gitlab-test && git checkout master && git pull origin master && exit 0"
- eval $(ssh-agent -k)
only:
- master
Related
I made a deploy script via ssh and gitlab but when the git pull script is executed, everything appears as already up to date and I can't even run the composer commands
before_script:
- apt-get update -qq
- apt-get install -qq git
# Setup SSH deploy keys
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
deploy_staging:
type: deploy
environment:
name: staging
url: test.ro
script:
- ssh -p 28785 test#test "git checkout development && git pull"
- cd server
- composer i
- composer optimize
- php artisan migrate
- cd ..
- cd client
- npm i
- npm run dev
- exit
only:
- development
$ apt-get update -qq
$ apt-get install -qq git
$ which ssh-agent || ( apt-get install -qq openssh-client )
/usr/bin/ssh-agent
$ eval $(ssh-agent -s)
Agent pid 266
$ ssh-add <(echo "$SSH_PRIVATE_KEY")
Identity added: /dev/fd/63 (/dev/fd/63)
$ mkdir -p ~/.ssh
$ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ ssh -p 28785
$ git pull
$ Already up to date
Don't I make the connection via ssh ok?
Try first to replace your ssh step with:
ssh -p 28785 test#test "id -a && ls -alrth && pwd && git status && git remote -v"
That way, you can make sure you are:
in the right path (which would be by default /home/test, an odd path for a Git view)
in an actual local Git repository, and the right branch (hence the git status)
referencing the right remote (meaning the remote repository where you have pushed new commits, which should be pulled)
When connecting with ssh to my webserver I try to install composer dependencies after git pull. But the pipeline fails saying bash: composer: command not found. But there is definitely composer installed on the server.
I can also mount my docker image and try it directly with no problem!
Does anyone has an Idea why composer is not found?
Here the .gitlab-ci.yml:
stages:
- deploy
deploy_stage:
stage: deploy
image: parallactic/php-node:php8.0-node14.x
before_script:
- 'which ssh-agent || ( apt-get install -qq openssh-client )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY" | base64 -d)
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
script:
- ssh ${DEPLOY_USER}#${DEPLOY_SERVER} "cd ${DEPLOY_DIR} && git pull origin main"
- ssh ${DEPLOY_USER}#${DEPLOY_SERVER} "cd ${DEPLOY_DIR} && composer install --no-interaction --prefer-dist --optimize-autoloader"
only:
- main
Thanks for any suggestions!
it may be that your DEPLOY_USER doesn't have composer in the PATH. you can check that with echo $PATH in this deploy script, and then manually connect with your user and compare the two.
but also you can specify full path to the composer (eg. /usr/local/bin/composer install ...)
btw, why separate ssh connections? you can do it in one line
ssh ${DEPLOY_USER}#${DEPLOY_SERVER} "cd ${DEPLOY_DIR} && git pull origin main && /usr/local/bin/composer install --no-interaction --prefer-dist --optimize-autoloader"
This is my gitlab pipeline. The Vue.js artifacts are build on the runner. How can I deploy the to my testserver? FYI: Fab pull does a git pull on the repo.
deploy_staging:
image: python:3.6
stage: deploy
only:
- master
before_script:
- curl -sL https://deb.nodesource.com/setup_13.x | bash -
- apt-get update -y
- apt-get install -y curl git gnupg nodejs
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- |
cat >~/.ssh/config <<EOF
Host testserver
ForwardAgent yes
HostName dev.testserver.ts
User testuser
EOF
- cat ~/.ssh/config
script:
- pip install -r requirements.txt
- npm install
- npm run production
- fab pull
Since you want to copy files from GitLab runner into your server, This will be possible using scp command.
For example:
⋮
script:
- pip install -r requirements.txt
- npm install
- npm run production
- scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no /PATH/TO/BUILD_ARTIFACTS testserver:~/PATH/TO/DESTINATION
- fab pull
UserKnownHostsFile and StrictHostKeyChecking are SSH options that prevent error Host key verification failed. So they should be used with scp command in your case.
Also, destination path of artifact files must be started from testuser's home directory (Tilde character ~). Otherwise you may face Permission denied error.
I have a project on gitlab that must install two other gitlab projects as npm packages. When there was only one package, I had my .gitlab-ci.yml set up like this:
stages:
- lint
variables:
PROJECT_1_KEY: $PROJECT_1_KEY
lint:
stage: lint
image: node-chrome:latest
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- ssh-add <(echo "$PROJECT_1_KEY")
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- cd app
- npm install
This worked just fine.
However, trying to add in a second project, which requires its own deploy key, has been unsuccessful so far.
I've added a second env variable PROJECT_2_KEY to the variables section.
Thing I've tried:
Using ssh-add to add both keys
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- ssh-add <(echo "$PROJECT_1_KEY")
- ssh-add <(echo "$PROJECT_2_KEY")
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- cd app
- npm install
Building separate files, one for each deploy key, and adding them to an .ssh/config file
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
- echo "$PROJECT_1_KEY" > ~/.ssh/project_1
- echo "$PROJECT_2_KEY" > ~/.ssh/project_2
- echo -e "Host project_1\n\tHostName gitlab.com\n\tIdentityFile $HOME/.ssh/project_1" > ~/.ssh/config
- echo -e "Host project_2\n\tHostName gitlab.com\n\tIdentityFile $HOME/.ssh/project_2" >> ~/.ssh/config
- cd app
- npm install
Adding both keys to the same id_rsa file and adding gitlab.com to known_hosts
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
- mkdir -p ~/.ssh
- echo "$PROJECT_1_KEY" >> ~/.ssh/id_rsa
- echo "$PROJECT_2_KEY" >> ~/.ssh/id_rsa
- chmod 600 ~/.ssh/id_rsa
- touch ~/.ssh/known_hosts
- ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
- cd app
- npm install
I'm flying a bit blind. Is there a correct technique for what I'm trying to do?
I fought a similar battle just in a different context (PHP Composer requiring a private gitlab repo). I couldn't get the SSH based example to work in a way I was satisfied with so I opted to take advantage of composer setting that used a custom url as the reference for a defined dependency.
In PHP it looked like this:
"require": {
"foo/bar": "dev-master",
...
"repositories": [
{
"type": "vcs",
"url": "https://gitlab+deploy-token-1234:abc-def-ghijk#gitlab.com/path/to/repo.git"
}
]
So given the npm context, can you use the dependencies keyword to define the projects using git urls that contain the token data to authenticate?
"dependencies" : {
"foo/bar" : "https://gitlab+deploy-token-1234:abc-def-ghijk#gitlab.com/path/to/repo.git",
}
If you don't like include auth data in the committed package.json you might try omitting that and just use the raw git url. In some projects during the build I just do a straight git clone of another private project and it appears the build process has permission to clone without configuring anything. (I'm not entirely sure "who" the build process is authed as, but presumably the user who triggered the build?)
Here is how the pipeline works
when a push on master (work as expected): build project && push jar to dev
when a tag is created (woesn't work as expected):
build project
increment pom.xml version and push pom.xml to
push jar to server
But when I do step 2, it retrigger another build in the CI.
How can I push and avoid triggering the job in this case ?
Here is the full gitlab-ci.yml:
image: maven:3.6.0-jdk-10
variables:
APP_NAME: demo
MAVEN_OPTS: -Dmaven.repo.local=/cache/maven.repository
stages:
- build
- deploy_dev
- deploy_prod
build:
stage: build
script:
- mvn package -P build
- mv target/*.jar target/$APP_NAME.jar
artifacts:
untracked: true
deploy_dev:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- '[[ -f /.dockerenv ]] && mkdir -p ~/.ssh && echo "$KNOWN_HOST" > ~/.ssh/known_hosts'
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
stage: deploy_dev
environment:
name: dev
url: http://devsb01:9999
dependencies:
- build
only:
- master
except:
- tags
script:
- ssh root#devsb01 "service $APP_NAME stop"
- scp target/$APP_NAME.jar root#devsb01:/var/apps/$APP_NAME/
- ssh root#devsb01 "service $APP_NAME start"
deploy_prod:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$SSH_PRIVATE_KEY")
- '[[ -f /.dockerenv ]] && mkdir -p ~/.ssh && echo "$KNOWN_HOST" > ~/.ssh/known_hosts'
- mkdir -p ~/.ssh
- '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
stage: deploy_prod
environment:
name: production
dependencies:
- build
only:
- tags
except:
- branches
script:
- mvn versions:set -DnewVersion=$CI_COMMIT_REF_NAME
- git config --global user.name "gitlab-ci"
- git config --global user.email "gitlab-ci#unc.nc"
- git --version
- git status
- git add pom.xml
- git commit -m "increment pom version"
- git push http://gitlab-ci:${GITLABCI_PWD}#gitlab.unc.nc/dsi-infogestion/demo.git HEAD:master
- git status
- ssh root#prodsb01 "service $APP_NAME stop"
- scp target/$APP_NAME.jar root#prodsb01:/var/apps/$APP_NAME/
- ssh root#prodsb01 "service $APP_NAME start"
I had the string '[ci skip]' in the commit message and it works:
git commit -m "increment pom version [ci skip]"