My drone.yml file is straightforward:
build:
image: node
commands:
- echo $${BRANCH}
deploy:
ssh:
host: my-domain
user: admin
port: 22
commands:
- touch /home/admin/testdrone
But in the output it seems like the ssh plugin is never even pulled:
[info] Pulling image plugins/drone-git:latest
$ git init
Initialized empty Git repository in /drone/src/github.com/.../.git/
$ git remote add origin https://github.com/....git
$ git fetch --no-tags --depth=50 origin +refs/pull/782/merge:
From https://github.com/...
* branch refs/pull/782/merge -> FETCH_HEAD
$ git checkout -qf FETCH_HEAD
$ echo drone-deploy
drone-deploy
How can I investigate what's going wrong?
Turns out that deploy steps are only executed if the hook is not a Pull Request. That was what was going wrong with my setup
Related
I am trying to add bitbucket pipelines to deploy my angular application to one of our servers.
I configured SSH on my server and I could fetch the Host's finger print under Known hosts in Bitbucket
Below is my YAML file.
image: node:14
pipelines:
branches:
master:
- step:
name: Building Test angular application
caches:
- node
script:
- echo "npm install in progress.."
- npm install
- echo "Installing angular/cli..."
- npm install -g #angular/cli
- echo "Starting the Build process.."
- ng build
artifacts:
- dist/** # Save build for next steps
- step:
name: "Deployment"
script:
- pipe: atlassian/scp-deploy:0.3.3
variables:
USER: $USER
SERVER: $SERVER
REMOTE_PATH: '/c/testscp/'
LOCAL_PATH: 'dist/*'
SSH_KEY: $MY_SSH_KEY
The first step is running fine without any issues and I can see dist folder being added to the artifact however the second step is failing with the below error.
scp -rp -i /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp dist/TestPipelineApplication <<USER>>#<<SERVERIP>>:/c/testscp/
Warning: Identity file /opt/atlassian/pipelines/agent/ssh/id_rsa_tmp not accessible: No such file or directory.
Load key "/root/.ssh/pipelines_id": invalid format
Permission denied, please try again.
Permission denied, please try again.
<<USER>>#<<SERVERIP>>: Permission denied (publickey,password,keyboard-interactive).
I never configured pipelines before so I am not completely sure what I am missing here.
Also, I looked into the below documentation but of no luck
https://bitbucket.org/atlassian/scp-deploy/src/1.0.1/README.md
Any help or suggestions are greatly appreciated.
We have an issue regarding the following:
Azure DevOps Linux Private Agent
Possible issue with corrupt / stale bearer token
Can be fixed with this command by logging onto the box but this is not convenient: git config --global --unset http.extraHeader
Can be fixed with this command as part of script in YAML: git config --global --unset http.extraHeader but not early enough. (See next comment below).
I can't run this command early enough in the pipeline YAML to clear the header as checkout is not controlled by me.
It generally only happens if a previous run fails at some point on the same private agent
Syncing repository: test-project-azure-workspace (Git)
git version
git version 2.26.0
git lfs version
git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.4)
git config --get remote.origin.url
git clean -ffdx
git reset --hard HEAD
HEAD is now at 5f9fd24 sql mi
git config gc.auto 0
git config --get-all http.https://xxxxxxx#dev.azure.com/xxxxxxx/xxxxxxx/_git/test-project-azure-workspace.extraheader
git config --get-all http.proxy
git config http.version HTTP/1.1
git -c http.extraheader="AUTHORIZATION: bearer ***" -c http.proxy="http://10.XXX.XXX.XX:80" fetch --force --tags --prune --progress --no-recurse-submodules --unshallow origin
* Couldn't find host dev.azure.com in the .netrc file; using defaults
Here is the code that can remedy the symptom:
- script: |
echo '======================================================================'
echo 'list all of git config values for your convenience:'
echo '======================================================================'
git config --list
echo '======================================================================'
existing_header=$(git config --get http.extraHeader)
if [ ${#existing_header} -gt 0 ]
then
echo 'We found the http.extraHeader'
echo 'un-setting extra header: http.extraHeader 🔥'
git config --global --unset http.extraHeader
else
echo 'no extra header: http.extraHeader was not found. Nothing to unset 👍'
fi
condition: always()
workingDirectory: '$(Agent.BuildDirectory)/s'
displayName: 'Remove Git Authentication'
Obviously, I am treating the symptom and not the cause so any pointers as to what is causing this would be a great help.
You can create a "cleaning" job at the start of your pipeline that uses the checkout task with none as the repo to checkout. This will allow you to then run the cleaning script.
- job: cleanAgent
steps:
- checkout: none
- script: |
echo "Put your git cleaner here"
A better option is really to clean the agent after you are done by running a cleaning task with a condition of always() at the end of your pipeline. This can be hard if other people also use the agents and don't clean-up though.
- job: cleanupAgents
condition: always()
steps:
- script: |
echo "Put your git cleaner here"
The company I work for has a private gitlab server that only supports ssh protocol when cloning a repository.
Inside this server, I have a gitlab-ci.yml file that uses docker executor to run some scripts.
The script's execution fails because it pulls the repository with https at its early stage. It generates this error message: fatal: unable to access 'https://gitlab.mycompany.com/path/to/the/repository/my_repo.git/': SSL certificate problem: unable to get local issuer certificate.
Where can I configure gitlab runner so that it uses ssh to clone the repository?
Here's the full execution log.
Running with gitlab-runner 12.7.1 (003fe500)
on my Group Runner Yh_yL3A2
Using Docker executor with image www.mycompany.com/path/to/the/image:1.0 ...
Pulling docker image www.mycompany.com/path/to/the/image:1.0 ...
Using docker image sha256:474e110ba44ddfje8ncoz4c44e91f2442547281192d4a82b88capmi9047cd8cb for www.mycompany.com/path/to/the/image:1.0 ...
Running on runner-Yh_yL3A2-project-343-concurrent-0 via b55d8c5ba21f...
Fetching changes...
Initialized empty Git repository in /path/to/the/repository/.git/
Created fresh repository.
fatal: unable to access 'https://gitlab.mycompany.com/path/to/the/repository/my_repo.git/': SSL certificate problem: unable to get local issuer certificate
ERROR: Job failed: exit code 1
Here's my .gitlab-ci.yml
image: www.mycompany.com/path/to/the/image:1.0
before_script:
- eval $(ssh-agent -s)
# Reference: https://docs.gitlab.com/ee/ci/ssh_keys/
# Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
# We're using tr to fix line endings which makes ed25519 keys work
# without extra base64 encoding.
# https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
#
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
#
# Create the SSH directory and give it the right permissions
#
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
stages:
- deploy
deploy:
stage: deploy
tags:
- infra
only:
refs:
- master
script:
- /bin/sh run.sh
I cannot find an option to specify whether the docker executor should use ssh or https to clone the repository.
I want to change a file and commit changes inside a gitlab-ci pipeline
I tried writing normal git commands in script
script:
- git clone git#gitlab.url.to.project.git
- cd project file
- touch test.txt
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git add .
- git commit -m "testing autocommit"
- git push
I get cannot find command git or something along those lines, I know it has something to do with tags, but if I try add a git tag it says no active runner. anyone has an idea how to run git commands on gitlab-ci ?
First you need to make sure you can actually use git, so either run your jobs on a shell executor located on a system that has git or use a docker executor and use an image that has git installed.
Next problem you will encounter is that you can't push to Git(lab) since you can't enter credentials.
So the solution is to create a ssh keypair and load the ssh private key into your CI environment through CI/CD variables, also add the corresponding public key to you your Git(lab) account.
Source: https://about.gitlab.com/2017/11/02/automating-boring-git-operations-gitlab-ci/
Your .gitlab-ci.yml will then look like this:
job-name:
stage: touch
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$GIT_SSH_PRIV_KEY")
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- mkdir -p ~/.ssh
- cat gitlab-known-hosts >> ~/.ssh/known_hosts
script:
- git clone git#gitlab.url.to.project.git
- cd project file
- touch test.txt
- git add .
- git commit -m "testing autocommit"
- git push
Gitlab CI/CD will clone the repository inside the running job automatically. What you need is the git command installed. You could use bitnami/git image to run the job in a container having the command installed.
This worked for me (trying to verify if a tag is available):
tag-available:
stage: .pre
image: bitnami/git:2.37.1
script:
# list all tags
- git tag -l
# check existence of tag "v1.0.0"
- >
if [ $(git tag -l "v1.0.0") ]; then
echo "yes"
else
echo "no."
If you need to authorize the job agains some gitlab registry (or api) please note that there are some predefined variables for user and password (tokens). For this kind of actions you might be most interested in these variables:
$CI_REGISTRY_PASSWORD
$CI_REGISTRY_USER
$CI_REGISTRY
$CI_REPOSITORY_URL
$CI_DEPLOY_PASSWORD
$CI_DEPLOY_USER
The CI_DEPLOY_USER must have a deploy user created and named "gitlab-deploy-token" to have his password loaded in the CI/CD. Read here more.
I added a deploy key with write access to my GitLab repository. My .gitlab-ci.yml file contains:
- git clone git#gitlab.domain:user/repo.git
- git checkout master
- git add myfile.pdf
- git commit -m "Generated PDF file"
- git push origin master
The deploy key works when cloning the repository.
Pushing is not possible, even if the deploy key has write access.
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#domain/user/repo.git/': The requested URL returned error: 403
I just encountered the same problem and saw this question without answer, so there is my solution.
Problem
The problem is caused by the fact that the remote url used by git to push the code is in the form http(s)://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#git.mydomain.com/group/project.git.
This url is using http(s) protocol so git doesn't use the ssh deploy key that you setup.
Solution
The solution is to change the push url of the remote origin so it matches ssh://git#git.mydomain.com/group/project.git.
The easiest way to do so is to use the predefined variable CI_REPOSITORY_URL.
Here is an example of code doing so by using sed:
# Change url from http(s) to ssh
url_host=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#|ssh://git#|g')
echo "${url_host}"
# ssh://git#git.mydomain.com/group/project.git
# Set the origin push url to the new one
git remote set-url --push origin "${url_host}"
Also, those using docker executor may want to verify the SSH host key as suggested by the gitlab documentation on deploy keys for docker executor.
So I give a more complete example for docker executor.
The code is mainly from gitlab documentation on ssh deploy keys.
In this example, the private deploy key is stored inside a variable named SSH_PRIVATE_KEY.
create:push:pdf:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "${SSH_PRIVATE_KEY}" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- git config --global user.email "email#example.com"
- git config --global user.name "User name"
- gitlab_hostname=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#||g' | sed -e 's|/.*||g')
- ssh-keyscan "${gitlab_hostname}" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- git checkout master
- git add myfile.pdf
- git commit -m "Generated PDF file"
- url_host=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#|ssh://git#|g')
- git remote set-url --push origin "${url_host}"
- git push origin master