Auto deploy mkdocs with versionning using mike on gitlab pages - gitlab-ci

I'm willing to deploy a new version of the doc each time I commit a tag on gitlab.
I'm using mike for the mkdocs versionning.
I want the versionning to be handle directly on the remote repository not locally.
Locally, I perfom my changes but each time I send a new tag, I would like the get a new added version of the doc on my gitlab page

Finally, I found a solution here.
Here is my adaptation:
pages:
stage: deploy
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
PAGES_BRANCH: gl-pages
HTTPS_REMOTE: https://gitlab-ci-token:${ACCESS_TOKEN}#${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git
before_script:
- pip install mkdocs mike
- git config user.name $GITLAB_USER_NAME
- git config user.email $GITLAB_USER_EMAIL
- git fetch origin $PAGES_BRANCH && git checkout -b $PAGES_BRANCH origin/$PAGES_BRANCH || echo "Pages branch not deployed yet."
- git checkout $CI_COMMIT_SHA
script:
- mike deploy --rebase --prefix public -r $HTTPS_REMOTE -p -b $PAGES_BRANCH -u $CI_COMMIT_TAG latest
- mike set-default --rebase --prefix public -r $HTTPS_REMOTE -p -b $PAGES_BRANCH latest
- git checkout $PAGES_BRANCH -- public/
artifacts:
paths:
- public/
only:
- tags

Related

GItLab - The build from another repository does not work

I have two projects:
devops/deploy/landing and frontend/landing.
The gitlab-ci.yml file is stored in devops/deploy/landing, and everything works fine there.
But when I add this file to the CI frontend/landing settings the external gitlab-ci.yml: .gitlab-ci.yml#devops/deploy/landing build process starts, but it writes that there are no available runners, although the same runner as for devops/deploy/landing is added to frontend/landing.
GitLab version - 13.4.3
gitlab-ci.yml
stages:
- build
- deploy
build_node:
stage: build
script:
- docker login $DOCKER_REGISTRY -u $DOCKER_USER -p $DOCKER_PASSWORD
- git clone https://$GIT_USER:$GIT_TOKEN#gitlab.domain.dev/frontend/landing.git
- docker build --network host -t $DOCKER_REGISTRY/landing:$VERSION . -f Dockerfile
- docker push $DOCKER_REGISTRY/landing:$VERSION
- docker image rm $DOCKER_REGISTRY/landing:$VERSION
- docker logout $DOCKER_REGISTRY
only:
- master
deploy_dev1:
image: ubuntu:latest
stage: deploy
script:
- apt update && apt install openssh-client -y
- eval $(ssh-agent -s) && ssh-add <(echo "$SSH_PRIVATE_KEY_DEV" | base64 --decode)
- ssh -o StrictHostKeyChecking=no root#$IP_DEV1 /home/deploy/deploy.sh
only:
- master
Runner Configuration
Added projects

Azure DevOps private Linux agent - YAML pipeline checkout failing - can fix with "git config --global --unset http.extraHeader" but not early enough

We have an issue regarding the following:
Azure DevOps Linux Private Agent
Possible issue with corrupt / stale bearer token
Can be fixed with this command by logging onto the box but this is not convenient: git config --global --unset http.extraHeader
Can be fixed with this command as part of script in YAML: git config --global --unset http.extraHeader but not early enough. (See next comment below).
I can't run this command early enough in the pipeline YAML to clear the header as checkout is not controlled by me.
It generally only happens if a previous run fails at some point on the same private agent
Syncing repository: test-project-azure-workspace (Git)
git version
git version 2.26.0
git lfs version
git-lfs/2.10.0 (GitHub; linux amd64; go 1.13.4)
git config --get remote.origin.url
git clean -ffdx
git reset --hard HEAD
HEAD is now at 5f9fd24 sql mi
git config gc.auto 0
git config --get-all http.https://xxxxxxx#dev.azure.com/xxxxxxx/xxxxxxx/_git/test-project-azure-workspace.extraheader
git config --get-all http.proxy
git config http.version HTTP/1.1
git -c http.extraheader="AUTHORIZATION: bearer ***" -c http.proxy="http://10.XXX.XXX.XX:80" fetch --force --tags --prune --progress --no-recurse-submodules --unshallow origin
* Couldn't find host dev.azure.com in the .netrc file; using defaults
Here is the code that can remedy the symptom:
- script: |
echo '======================================================================'
echo 'list all of git config values for your convenience:'
echo '======================================================================'
git config --list
echo '======================================================================'
existing_header=$(git config --get http.extraHeader)
if [ ${#existing_header} -gt 0 ]
then
echo 'We found the http.extraHeader'
echo 'un-setting extra header: http.extraHeader 🔥'
git config --global --unset http.extraHeader
else
echo 'no extra header: http.extraHeader was not found. Nothing to unset 👍'
fi
condition: always()
workingDirectory: '$(Agent.BuildDirectory)/s'
displayName: 'Remove Git Authentication'
Obviously, I am treating the symptom and not the cause so any pointers as to what is causing this would be a great help.
You can create a "cleaning" job at the start of your pipeline that uses the checkout task with none as the repo to checkout. This will allow you to then run the cleaning script.
- job: cleanAgent
steps:
- checkout: none
- script: |
echo "Put your git cleaner here"
A better option is really to clean the agent after you are done by running a cleaning task with a condition of always() at the end of your pipeline. This can be hard if other people also use the agents and don't clean-up though.
- job: cleanupAgents
condition: always()
steps:
- script: |
echo "Put your git cleaner here"

how to execute git commands in gitlab-ci scripts

I want to change a file and commit changes inside a gitlab-ci pipeline
I tried writing normal git commands in script
script:
- git clone git#gitlab.url.to.project.git
- cd project file
- touch test.txt
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git add .
- git commit -m "testing autocommit"
- git push
I get cannot find command git or something along those lines, I know it has something to do with tags, but if I try add a git tag it says no active runner. anyone has an idea how to run git commands on gitlab-ci ?
First you need to make sure you can actually use git, so either run your jobs on a shell executor located on a system that has git or use a docker executor and use an image that has git installed.
Next problem you will encounter is that you can't push to Git(lab) since you can't enter credentials.
So the solution is to create a ssh keypair and load the ssh private key into your CI environment through CI/CD variables, also add the corresponding public key to you your Git(lab) account.
Source: https://about.gitlab.com/2017/11/02/automating-boring-git-operations-gitlab-ci/
Your .gitlab-ci.yml will then look like this:
job-name:
stage: touch
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- ssh-add <(echo "$GIT_SSH_PRIV_KEY")
- git config --global user.name "${GITLAB_USER_NAME}"
- git config --global user.email "${GITLAB_USER_EMAIL}"
- mkdir -p ~/.ssh
- cat gitlab-known-hosts >> ~/.ssh/known_hosts
script:
- git clone git#gitlab.url.to.project.git
- cd project file
- touch test.txt
- git add .
- git commit -m "testing autocommit"
- git push
Gitlab CI/CD will clone the repository inside the running job automatically. What you need is the git command installed. You could use bitnami/git image to run the job in a container having the command installed.
This worked for me (trying to verify if a tag is available):
tag-available:
stage: .pre
image: bitnami/git:2.37.1
script:
# list all tags
- git tag -l
# check existence of tag "v1.0.0"
- >
if [ $(git tag -l "v1.0.0") ]; then
echo "yes"
else
echo "no."
If you need to authorize the job agains some gitlab registry (or api) please note that there are some predefined variables for user and password (tokens). For this kind of actions you might be most interested in these variables:
$CI_REGISTRY_PASSWORD
$CI_REGISTRY_USER
$CI_REGISTRY
$CI_REPOSITORY_URL
$CI_DEPLOY_PASSWORD
$CI_DEPLOY_USER
The CI_DEPLOY_USER must have a deploy user created and named "gitlab-deploy-token" to have his password loaded in the CI/CD. Read here more.

GitLab CI denies access to push using a deploy key with write access

I added a deploy key with write access to my GitLab repository. My .gitlab-ci.yml file contains:
- git clone git#gitlab.domain:user/repo.git
- git checkout master
- git add myfile.pdf
- git commit -m "Generated PDF file"
- git push origin master
The deploy key works when cloning the repository.
Pushing is not possible, even if the deploy key has write access.
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#domain/user/repo.git/': The requested URL returned error: 403
I just encountered the same problem and saw this question without answer, so there is my solution.
Problem
The problem is caused by the fact that the remote url used by git to push the code is in the form http(s)://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx#git.mydomain.com/group/project.git.
This url is using http(s) protocol so git doesn't use the ssh deploy key that you setup.
Solution
The solution is to change the push url of the remote origin so it matches ssh://git#git.mydomain.com/group/project.git.
The easiest way to do so is to use the predefined variable CI_REPOSITORY_URL.
Here is an example of code doing so by using sed:
# Change url from http(s) to ssh
url_host=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#|ssh://git#|g')
echo "${url_host}"
# ssh://git#git.mydomain.com/group/project.git
# Set the origin push url to the new one
git remote set-url --push origin "${url_host}"
Also, those using docker executor may want to verify the SSH host key as suggested by the gitlab documentation on deploy keys for docker executor.
So I give a more complete example for docker executor.
The code is mainly from gitlab documentation on ssh deploy keys.
In this example, the private deploy key is stored inside a variable named SSH_PRIVATE_KEY.
create:push:pdf:
before_script:
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
- eval $(ssh-agent -s)
- echo "${SSH_PRIVATE_KEY}" | tr -d '\r' | ssh-add - > /dev/null
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- git config --global user.email "email#example.com"
- git config --global user.name "User name"
- gitlab_hostname=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#||g' | sed -e 's|/.*||g')
- ssh-keyscan "${gitlab_hostname}" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- git checkout master
- git add myfile.pdf
- git commit -m "Generated PDF file"
- url_host=$(echo "${CI_REPOSITORY_URL}" | sed -e 's|https\?://gitlab-ci-token:.*#|ssh://git#|g')
- git remote set-url --push origin "${url_host}"
- git push origin master

Drone ssh plugin not triggered

My drone.yml file is straightforward:
build:
image: node
commands:
- echo $${BRANCH}
deploy:
ssh:
host: my-domain
user: admin
port: 22
commands:
- touch /home/admin/testdrone
But in the output it seems like the ssh plugin is never even pulled:
[info] Pulling image plugins/drone-git:latest
$ git init
Initialized empty Git repository in /drone/src/github.com/.../.git/
$ git remote add origin https://github.com/....git
$ git fetch --no-tags --depth=50 origin +refs/pull/782/merge:
From https://github.com/...
* branch refs/pull/782/merge -> FETCH_HEAD
$ git checkout -qf FETCH_HEAD
$ echo drone-deploy
drone-deploy
How can I investigate what's going wrong?
Turns out that deploy steps are only executed if the hook is not a Pull Request. That was what was going wrong with my setup