Azure DevOps private Linux agent - YAML pipeline checkout failing - can fix with "git config --global --unset http.extraHeader" but not early enough - authentication

We have an issue regarding the following:
Azure DevOps Linux Private Agent
Possible issue with corrupt / stale bearer token
Can be fixed with this command by logging onto the box but this is not convenient: git config --global --unset http.extraHeader
Can be fixed with this command as part of script in YAML: git config --global --unset http.extraHeader but not early enough. (See next comment below).
I can't run this command early enough in the pipeline YAML to clear the header as checkout is not controlled by me.
It generally only happens if a previous run fails at some point on the same private agent
Syncing repository: test-project-azure-workspace (Git)
git version
git version 2.26.0
git lfs version
git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.4)
git config --get remote.origin.url
git clean -ffdx
git reset --hard HEAD
HEAD is now at 5f9fd24 sql mi
git config gc.auto 0
git config --get-all http.https://xxxxxxx#dev.azure.com/xxxxxxx/xxxxxxx/_git/test-project-azure-workspace.extraheader
git config --get-all http.proxy
git config http.version HTTP/1.1
git -c http.extraheader="AUTHORIZATION: bearer ***" -c http.proxy="http://10.XXX.XXX.XX:80" fetch --force --tags --prune --progress --no-recurse-submodules --unshallow origin
* Couldn't find host dev.azure.com in the .netrc file; using defaults
Here is the code that can remedy the symptom:
- script: |
echo '======================================================================'
echo 'list all of git config values for your convenience:'
echo '======================================================================'
git config --list
echo '======================================================================'
existing_header=$(git config --get http.extraHeader)
if [ ${#existing_header} -gt 0 ]
then
echo 'We found the http.extraHeader'
echo 'un-setting extra header: http.extraHeader 🔥'
git config --global --unset http.extraHeader
else
echo 'no extra header: http.extraHeader was not found. Nothing to unset 👍'
fi
condition: always()
workingDirectory: '$(Agent.BuildDirectory)/s'
displayName: 'Remove Git Authentication'
Obviously, I am treating the symptom and not the cause so any pointers as to what is causing this would be a great help.

You can create a "cleaning" job at the start of your pipeline that uses the checkout task with none as the repo to checkout. This will allow you to then run the cleaning script.
- job: cleanAgent
steps:
- checkout: none
- script: |
echo "Put your git cleaner here"
A better option is really to clean the agent after you are done by running a cleaning task with a condition of always() at the end of your pipeline. This can be hard if other people also use the agents and don't clean-up though.
- job: cleanupAgents
condition: always()
steps:
- script: |
echo "Put your git cleaner here"

Related

gitlab-ci testing build failed

I am new to gitlab-ci and I am trying to test my build phase but it keeps failing before last step. here is the section
build:
stage: build
before_script:
- git config user.name "$GITLAB_USER_NAME"
- git config user.email "$GITLAB_USER_EMAIL"
script:
- git add .
- git commit -m "[skip ci] migrations"
- git push "https://${GITLAB_USER_NAME}:${CI_GIT_TOKEN}#${CI_REPOSITORY_URL}" "HEAD:qa" -o ci.skip
dependencies:
- migrations
only:
- qa
This is the log, last few lines
Using docker image sha256:5b3b4504ff1f7b859dbc5d7fb86f4afc644be62f99b8ced636fbca64c8a6c2de for python:latest with digest python#sha256:73cc381fa0fe5e6d5dd38f1397da28c70096519d8818c2249f2bc8e1158ba015 ...
$ git config user.name "$GITLAB_USER_NAME"
$ git config user.email "$GITLAB_USER_EMAIL"
$ git add .
$ git commit -m "[skip ci] migrations"
HEAD detached at bf2a8e4
nothing to commit, working tree clean
Cleaning up file based variables
00:01
ERROR: Job failed: exit code 1
How can I find out what ERROR: Job failed: exit code 1 is? I am using shared runners
The error ERROR: Job failed: exit code 1 is coming from git not finding anything to commit. In other words, the below message is causing the error:
nothing to commit, working tree clean
Cleaning up file based variables
You can either may sure there is something to commit, or you can adjust your script as outlined in this related thread: How to git commit nothing without an error?

how to config lfs.fetchinclude in gitlabci

I want to git lfs fetch only in some dir in the gitlab CI. but failed
the gitlab-runner was 11.8.0~beta.1077
i config like this:
variables:
# Please edit to your GitLab project
GIT_STRATEGY: clone
GIT_CHECKOUT: "false"
script:
- git config lfs.fetchinclude "xxx/xxx/, test/"
but ci erro:
root config contains unknown keys: script
how to fix it?
The first part is your syntax error - the script key must be part of a job, e.g. like this:
build:
stage: build
script:
- git config ...
However, I think the CI runner will fetch LFS files automatically, and the script is only run after cloning.
So I think you have to disable automatic fetching before, & then this might work:
build:
stage: build
before_script:
- git config lfs.fetchinclude "xxx/xxx/, test/"
- git lfs pull

how to execute git commands in gitlab-ci scripts

I want to change a file and commit changes inside a gitlab-ci pipeline
I tried writing normal git commands in script
script:
- git clone git#gitlab.url.to.project.git
- cd project file
- touch test.txt
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git add .
- git commit -m "testing autocommit"
- git push
I get cannot find command git or something along those lines, I know it has something to do with tags, but if I try add a git tag it says no active runner. anyone has an idea how to run git commands on gitlab-ci ?
First you need to make sure you can actually use git, so either run your jobs on a shell executor located on a system that has git or use a docker executor and use an image that has git installed.
Next problem you will encounter is that you can't push to Git(lab) since you can't enter credentials.
So the solution is to create a ssh keypair and load the ssh private key into your CI environment through CI/CD variables, also add the corresponding public key to you your Git(lab) account.
Source: https://about.gitlab.com/2017/11/02/automating-boring-git-operations-gitlab-ci/
Your .gitlab-ci.yml will then look like this:
job-name:
stage: touch
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$GIT_SSH_PRIV_KEY")
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- mkdir -p ~/.ssh
- cat gitlab-known-hosts >> ~/.ssh/known_hosts
script:
- git clone git#gitlab.url.to.project.git
- cd project file
- touch test.txt
- git add .
- git commit -m "testing autocommit"
- git push
Gitlab CI/CD will clone the repository inside the running job automatically. What you need is the git command installed. You could use bitnami/git image to run the job in a container having the command installed.
This worked for me (trying to verify if a tag is available):
tag-available:
stage: .pre
image: bitnami/git:2.37.1
script:
# list all tags
- git tag -l
# check existence of tag "v1.0.0"
- >
if [ $(git tag -l "v1.0.0") ]; then
echo "yes"
else
echo "no."
If you need to authorize the job agains some gitlab registry (or api) please note that there are some predefined variables for user and password (tokens). For this kind of actions you might be most interested in these variables:
$CI_REGISTRY_PASSWORD
$CI_REGISTRY_USER
$CI_REGISTRY
$CI_REPOSITORY_URL
$CI_DEPLOY_PASSWORD
$CI_DEPLOY_USER
The CI_DEPLOY_USER must have a deploy user created and named "gitlab-deploy-token" to have his password loaded in the CI/CD. Read here more.

Drone ssh plugin not triggered

My drone.yml file is straightforward:
build:
image: node
commands:
- echo $${BRANCH}
deploy:
ssh:
host: my-domain
user: admin
port: 22
commands:
- touch /home/admin/testdrone
But in the output it seems like the ssh plugin is never even pulled:
[info] Pulling image plugins/drone-git:latest
$ git init
Initialized empty Git repository in /drone/src/github.com/.../.git/
$ git remote add origin https://github.com/....git
$ git fetch --no-tags --depth=50 origin +refs/pull/782/merge:
From https://github.com/...
* branch refs/pull/782/merge -> FETCH_HEAD
$ git checkout -qf FETCH_HEAD
$ echo drone-deploy
drone-deploy
How can I investigate what's going wrong?
Turns out that deploy steps are only executed if the hook is not a Pull Request. That was what was going wrong with my setup

Getting GitLab CI to clone private repositories

I have GitLab & GitLab CI set up to host and test some of my private repos. For my composer modules under this system, I have Satis set up to resolve my private packages.
Obviously these private packages require an ssh key to clone them, and I have this working in the terminal - I can run composer install and get these packages, so long as I have the key added with ssh-add in the shell.
However, when running my tests in GitLab CI, if a project has any of these dependencies the tests will not complete as my GitLab instance needs authentication to get the deps (obviously), and the test fails saying Host key verification failed.
My question is how do I set this up so that when the runner runs the test it can authenticate to gitlab without a password? I have tried putting a password-less ssh-key in my runners ~/.ssh folder, however the build wont even add the key, "eval ssh-agent -s" followed by ssh-add seems to fail saying the agent isn't running...
See also other solutions:
git submodule permission (see Marco A.'s answer)
job token and override repo in git config (see a544jh's answer)
Here a full howto with SSH keys:
General Design
generating a pair of SSH keys
adding the private one as a secure environment variable of your project
making the private one available to your test scripts on GitLab-CI
adding the public one as a deploy key on each of your private dependencies
Generating a pair of public and private SSH keys
Generate a pair of public and private SSH keys without passphrase:
ssh-keygen -b 4096 -C "<name of your project>" -N "" -f /tmp/name_of_your_project.key
Adding the private SSH key to your project
You need to add the key as a secure environment variable to your project as
following:
browse https://<gitlab_host>/<group>/<project_name>/variables
click on "Add a variable"
fill the text field Key with SSH_PRIVATE_KEY
fill the text field Value with the private SSH key itself
click on "Save changes"
Exposing the private SSH key to your test scripts
In order to make your private key available to your test scripts you need to add
the following to your .gitlab-ci.yml file:
before_script:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_PRIVATE_KEY")
# disable host key checking (NOTE: makes you susceptible to man-in-the-middle attacks)
# WARNING: use only in docker container, if you use it with shell you will overwrite your user's ssh config
- mkdir -p ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
Code Snippet comes from GitLab documentation
Adding the public SSH key as a deploy key to all your private dependencies
You need to register the public SSH key as deploy key to all your private
dependencies as following:
browse https://<gitlab_host>/<group>/<dependency_name>/deploy_keys
click on "New deploy key"
fill the text field Title with the name of your project
fill the text field Key with the public SSH key itself
click on "Create deploy key"
If you don't want to fiddle around with ssh keys or submodules, you can override the repo in git's configuration to authenticate with the job token instead (in gitlab-ci.yml):
before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.example.com/group/repo.git".insteadOf git#gitlab.example.com:group/repo.git
I'm posting this as an answer since others weren't completely clear and/or detailed IMHO
Starting from GitLab 8.12+, assuming the submodule repo is in the same server as the one requesting it, you can now:
Set up the repo with git submodules as usual (git submodule add git#somewhere:folder/mysubmodule.git)
Modify your .gitmodules file as follows
[submodule "mysubmodule"]
path = mysubmodule
url = ../../group/mysubmodule.git
where ../../group/mysubmodule.git is a relative path from your repository to the submodule's one.
Add the following lines to gitlab-ci.yml
variables:
GIT_SUBMODULE_STRATEGY: recursive
to instruct the runner to fetch all submodules before the build.
Caveat: if your runner seems to ignore the GIT_SUBMODULE_STRATEGY directive, you should probably consider updating it.
(source: https://docs.gitlab.com/ce/ci/git_submodules.html)
The currently accepted answer embeds Gitlab-specific requirements into my .gitmodules file. This forces a specific directory layout for local development and would complicate moving to another version control platform.
Instead, I followed the advice in Juddling's answer. Here's a more complete answer.
My .gitmodules files has the following contents:
[submodule "myproject"]
url = git#git.myhost.com:mygroup/myproject.git
In my gitlab-ci.yml I have the following:
build:
stage: build
before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#git.myhost.com/".insteadOf "git#git.myhost.com:"
- git submodule sync && git submodule update --init
The trailing / and : are critical in the git config line, since we are mapping from SSH authentication to HTTPS. This tripped me up for a while with "Illegal port number" errors.
I like this solution because it embeds the Gitlab-specific requirements in a Gitlab-specific file, which is ignored by everything else.
I used deploy tokens to solve this issue, as setting up SSH keys for a test runner seems a little long winded.
git clone http://<username>:<deploy_token>#gitlab.example.com/tanuki/awesome_project.git
The deploy tokens are per project and are read only.
One way to solve this without changing the git repository's structure is to perform the following steps:
1. get ssh host keys
Get the ssh host keys of the server that you are running on. For gitlab.com:
run ssh-keyscan gitlab.com > known_hosts
check that ssh-keygen -lf known_hosts agrees with the fingerprints reported here.
copy the content of the known_hosts and paste it on a variable called SSH_KNOWN_HOSTS on the repository.
This step is only needed once.
2. configure the job to use ssh
before_script:
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com".insteadOf "git#gitlab.com:"
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
The "ssh://git#gitlab.com" bit may be different if you are trying to do git clone git#gitlab.com: or pip install -e git+ssh://git#gitlab.com/...; adjust it accordingly to your needs.
At this point, your CI is able to use ssh to fetch from another (private) repository.
3. [Bonus DRY]
Use this trick to write it generically:
.enable_ssh: &enable_ssh |-
git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.com".insteadOf "ssh://git#gitlab.com"
mkdir -p ~/.ssh
chmod 700 ~/.ssh
echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
chmod 644 ~/.ssh/known_hosts
and enable it on jobs that need it
test:
stage: test
before_script:
- *enable_ssh
script:
- ...
If your CI runner is running on a container model, you need to use the deploy key. doc: https://docs.gitlab.com/ee/user/project/deploy_tokens/#git-clone-a-repository
git clone https://<username>:<deploy_token>#gitlab.example.com/tanuki/awesome_project.git
Create your deploy token
Add your token in CI pipeline Variable
make sure your container has the git and change the git URL by insteadOf
image: docker:latest
before_script:
- apk add --no-cache curl jq python3 py3-pip git
- git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.example.come/".insteadOf 'git#gitlab.example.come:'
for replace URL: https://docs.gitlab.com/ee/user/project/working_with_projects.html#authenticate-git-fetches
I had a scenario where I had to use my ssh key in 3 different scripts, so I put the ssh key stuff in a single shell script and called it first, before the other 3 scripts. This ended up not working, I think due to the ssh-agent not persisting between shell scripts, or something to that effect. I ended up actually just outputting the private key into the ~/.ssh/id_rsa file, which will for sure persist to other scripts.
.gitlab-ci.yml
script:
- ci/init_ssh.sh
- git push # or whatever you need ssh for
ci/init_ssh.sh
# only run in docker:
[[ ! -e /.dockerenv ]] && exit 0
mkdir -p ~/.ssh
echo "$GITLAB_RUNNER_SSH_KEY" > ~/.ssh/id_rsa
chmod 400 ~/.ssh/id_rsa
echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > /.ssh/config
It works like a charm!
If you are using an alpine-based image (maybe docker:latest or docker:dind), your before_script might look like this:
before_script:
- apk add --no-cache openssh-client git
- mkdir -p /.ssh && touch /.ssh/known_hosts
- ssh-keyscan gitlab.com >> /.ssh/known_hosts
- echo $SSH_KEY | base64 -d >> /.ssh/id_rsa && chmod 600 /.ssh/id_rsa
- git clone git#git.myhost.com:mygroup/myproject.git
Adding this to .gitlab-ci.yml did the trick for me.
(as mentioned here: https://docs.gitlab.com/ee/user/project/new_ci_build_permissions_model.html#dependent-repositories)
before_script:
echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc
(I tried setting up SSH_PRIVATE_KEY as mentioned in one of the answers above, it won't work)
Gitlab 15.9.0 introduces an update to the pre-defined variable CI_JOB_TOKEN. Now you can control other projects' access to your private repository, see the release note and documentation.
Once access is granted, you can clone private repositories by adding this line to your job's scripts or before_scripts.
git clone https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.example.com/<namespace>/<project>
Unfortunately, this still does not play nicely with the submodule integration with Gitlab CI/CD. Instead, I do this in my projects.
# .gitlab-ci.yml
default:
before_script:
- |
cat << EOF > ~/.gitconfig
[url "https://gitlab-ci-token:${CI_JOB_TOKEN}#gitlab.example.com/<namespace>/<project>.git"]
insteadOf = git#gitlab.example.com/<namespace>/<project>.git
EOF
- git submodule update --init --recursive
And this is what my .gitmodules would look like
[submodule "terraform-eks"]
path = modules/<project>
url = git#gitlab.example.com/<namespace>/<project>.git
branch = main
Hope this help!
Seems there is finally a reasonable solution.
In short as of GitLab 8.12 all you need to do is use relative paths in the .submodules, and the git submodule update --init will simply work