Is there any way to switch a job to manual mode (and vice-versa) dynamically?
So, instead of having when: manual in gitlab-ci.yml file, it would be switched to manual (or the opposite) according to any dynamic checking.
Example: Lacking of configuration from environment variables that prevents it to run, but the user could run it later if they set the variables at the launch time.
Just playing with the current syntax, it could be like:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
when: manual && [[ ! $my_var ]] # Using shell syntax just as an example of the condition
Instead of failing:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- [[ ! $my_var ]] && echo "my_var is undefined" && exit 1
- echo "Deploying $my_var..."
You can try using rules to set a condition based on a variable you set in the script of the job.
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- export my_var
rules:
- if: $my_var
when: manual
https://docs.gitlab.com/ee/ci/yaml/#rules-attributes
You can make use of rules if with below example
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
rules:
- if: $my_var
when: on_success
- if: !$my_var
when: manual
So if var is set then it will automatically run. if it is not set then it'll become a manual process
Ref: Gitlab Rules
Related
I want to run a Gitlab pipeline stage after pushing a commit on the 'master' branch and a directory named 'tests' exists in the project root. Here is the relevant snippet from .gitlab-ci.yaml that defines the stage and its rules:
stage: test
image: python:3.10.3-bullseye
rules:
- if: $CI_COMMIT_BRANCH == "master"
exists:
- tests
before_script: *setup-python
script:
- pip install -e .[dev]
- pytest tests/
However, after a commit on the master branch, it remains disabled. I'm a bit fuzzy on the syntax, despite reading the docs carefully. Does anyone know how to achieve what I'm after?
I think you should define the job like:
stage: test
image: python:3.10.3-bullseye
rules:
- if: $CI_COMMIT_BRANCH == "master"
- exists:
- tests/**/*.py
before_script: *setup-python
script:
- pip install -e .[dev]
- pytest tests/
reference:
https://docs.gitlab.com/14.10/ee/ci/yaml/#rulesexists
https://docs.gitlab.com/14.10/ee/ci/yaml/#onlyrefs--exceptrefs
I'm trying to force the pipeline to skip running job whenever the project's branch is equal to develop. I've created the custom variable name $CURRENT_BRANCH that gathering from predefined variable $CI_COMMIT_REF_NAME
- export CURRENT_BRANCH=`echo ${CI_COMMIT_REF_NAME} | awk -F '/' '{print $1}'`
The $CURRENT_BRANCH will pass along the first stage and it will save in the artifact
- >
if [ ${CURRENT_BRANCH} == 'feature' ]; then
export CURRENT_BRANCH="feature"
export FEATURE_NAME=`echo ${CI_COMMIT_REF_NAME} | awk -F '/' '{print $2}'`
echo "FEATURE_NAME=${FEATURE_NAME}" >> info.env
echo "PROJECT_NAME=${FEATURE_NAME}" >> info.env
else
export CURRENT_BRANCH=${CURRENT_BRANCH}
echo "PROJECT_NAME=${CURRENT_BRANCH}" >> info.env
fi
- echo "CURRENT_BRANCH=${CURRENT_BRANCH}" >> info.env
artifacts:
reports:
dotenv: info.env
Then in the next stage until the last stage, I've tried to skip running the pipeline with only/except and rules/when for example,
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
only:
variables:
- $CURRENT_BRANCH != "develop"
as well as using rules and when condition
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CURRENT_BRANCH == "develop"'
when: never
The pipeline still running. Are there anything that I miss?
Quoting from Gitlab docs https://docs.gitlab.com/ee/ci/yaml/#rules
Rules are evaluated when the pipeline is created, and evaluated in order until the first match. When a match is found, the job is either included or excluded from the pipeline, depending on the configuration.
You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run.
So you can not use variables created in one job to decide if a following job should triggered.
In your case why don't you use the predefined variable CI_COMMIT_REF_NAME directly?
CI_COMMIT_REF_NAME: The branch or tag name for which project is built
Change this
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CURRENT_BRANCH == "develop"'
when: never
To
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CI_COMMIT_REF_NAME == "develop"'
when: never
do you provide the full code of the job build_image?
The build_image job needs the info.env artifacts of your first job. Can you try to add this?
needs:
- job: '<first-job-name>'
artifacts: true
I realised that gitlab CI does not seem to allow multiple refs. It will only take the first.
E.g., for the instructions below, the merge_requests will be ignored, and will trigger whenever the develop branch is updated directly.
face-build:
stage: build
image: docker:19.03.8
services:
- docker:19.03.8-dind
script:
- sh some-scripts.sh
only:
refs:
- /^develop$/
- merge_requests
If I swap the merge_requests to be before /^develop$/ it will be triggered for all merge requests.
Is there anyway to set both to be valid?
If you are using GitLab 12.3 or later, try rules:if clause instead of only:
face-build:
stage: build
image: docker:19.03.8
services:
- docker:19.03.8-dind
script:
- sh some-scripts.sh
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME =~ /^develop$/'
when: always
Please check Rules attributes so you can choose the most appropriat value for when (on_success, always, delayed or never).
The following gitlab ci job will run if the variable $CI_COMMIT_TAG is set OR if the ./versions.txt file has changed.
some-job:
script:
- echo "Do some fancy stuff.";
rules:
- if: $CI_COMMIT_TAG
when: always
- changes:
- ./versions.txt
However, what I need is for this job to run when $CI_COMMIT_TAG is set AND ./versions.txt is changed. I don't want the job to run if only one of these evaluates to true. This was the behaviour in only/changes feature, but the only (and except)-feature is less powerful and deprecated.
Is what I want currently possible with gitlab ci?
From Docs:
In the following example:
We run the job manually if Dockerfile or any file in docker/scripts/ has changed AND $VAR == "string value". Otherwise, the job will not be included in the pipeline.
docker build:
script: docker build -t my-image:$CI_COMMIT_REF_SLUG .
rules:
- if: '$VAR == "string value"'
changes: # Will include the job and set to when:manual if any of the follow paths match a modified file.
- Dockerfile
- docker/scripts/*
when: manual
Your code will look something like this.
some-job:
script:
- echo "Do some fancy stuff.";
rules:
- if: $CI_COMMIT_TAG
changes:
- versions.txt
when: manual
Depending on branch the build comes from I need to use slightly different command line arguments. Particularly I would like to upload snapshot nexus artifacts when building from a branch, and release artifact when building off master.
Is there a way to conditionally alter variables?
I tried to use except/only keywords like this
stages:
- stage
variables:
TYPE: Release
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
upload:
extends:
- .upload_common
- .upload_snapshot
Unfortunately it skips whole upload step when building off master.
The reason I am using 'extends' pattern here is that I have win and mac platforms which use slightly different variables substitution syntax ($ vs %). I also have a few different build configuration - Debug/Release, 32bit/64bit.
The code below actually works, but I had to duplicate steps for release and snapshot, one is enabled at a time.
stages:
- stage
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
.upload_release:
variables:
TYPE: "Release"
only:
- master
upload_release:
extends:
- .upload_common
- .upload_release
upload_snapshot:
extends:
- .upload_common
- .upload_snapshot
The code gets much larger when snapshot/release configuration is multiplied by Debug/Release, Mac/Win, and 32/64bits. I would like to keep number of configurations at minimum.
Having ability to conditionally altering just a few variables would help me reducing this code a lot.
Another addition in GitLab 13.7 are the rules:variables. This allows some logic in setting vars:
job:
variables:
DEPLOY_VARIABLE: "default-deploy"
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
variables: # Override DEPLOY_VARIABLE defined
DEPLOY_VARIABLE: "deploy-production" # at the job level.
- if: $CI_COMMIT_REF_NAME =~ /feature/
variables:
IS_A_FEATURE: "true" # Define a new variable.
script:
- echo "Run script with $DEPLOY_VARIABLE as an argument"
- echo "Run another script if $IS_A_FEATURE exists"
Unfortunatelly YAML-anchors or GitLab-CI's extends don't seem to allow to combine things in script array of commands as of today.
I use built-in variable CI_COMMIT_REF_NAME in combination with global or job-only before_script to solve similar tasks without repeating myself.
Here is an example of my workaround on how to set environment variable to different values dynamically for PROD and DEV during delivery or deployment:
.provide ssh private deploy key: &provide_ssh_private_deploy_key
before_script:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- |
if [ "$CI_COMMIT_REF_NAME" == "master" ]; then
echo "$SSH_PRIVATE_DEPLOY_KEY_PROD" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are in master (PROD)"
else
echo "$SSH_PRIVATE_DEPLOY_KEY_DEV" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are NOT in master (DEV)"
fi
- chmod 600 ~/.ssh/id_rsa
deliver-via-ssh:
stage: deliver
<<: *provide_ssh_private_deploy_key
script:
- echo Stage is deliver
- echo $MY_DYNAMIC_VAR
- ssh ...
Also consider this workaround for concatenation of "script" commands: https://stackoverflow.com/a/57209078/470108
hopefully it helps.
A nice way to prepare variables for other jobs is the dotenv report artifact. Unfortunately, these variables cannot be used in many places, but if you only need to access them from other jobs scripts, this is the way:
# prepare environment variables for other jobs
env:
stage: .pre
script:
# Set application version from GIT tag or fake it
- echo "APPVERSION=${CI_COMMIT_TAG:-0.1-dev-$CI_COMMIT_REF_SLUG}+$CI_COMMIT_SHORT_SHA" | tee -a .env
artifacts:
reports:
dotenv: .env
In the script of this job you can conditionally prepare and write your environment values into a file, then make a dotenv artifact out of it. Subsequent - or better yet dependent - jobs will pick up the variables for their scripts from there.