Unable to skip running pipeline based on variable in Gitlab CI - gitlab-ci

I'm trying to force the pipeline to skip running job whenever the project's branch is equal to develop. I've created the custom variable name $CURRENT_BRANCH that gathering from predefined variable $CI_COMMIT_REF_NAME
- export CURRENT_BRANCH=`echo ${CI_COMMIT_REF_NAME} | awk -F '/' '{print $1}'`
The $CURRENT_BRANCH will pass along the first stage and it will save in the artifact
- >
if [ ${CURRENT_BRANCH} == 'feature' ]; then
export CURRENT_BRANCH="feature"
export FEATURE_NAME=`echo ${CI_COMMIT_REF_NAME} | awk -F '/' '{print $2}'`
echo "FEATURE_NAME=${FEATURE_NAME}" >> info.env
echo "PROJECT_NAME=${FEATURE_NAME}" >> info.env
else
export CURRENT_BRANCH=${CURRENT_BRANCH}
echo "PROJECT_NAME=${CURRENT_BRANCH}" >> info.env
fi
- echo "CURRENT_BRANCH=${CURRENT_BRANCH}" >> info.env
artifacts:
reports:
dotenv: info.env
Then in the next stage until the last stage, I've tried to skip running the pipeline with only/except and rules/when for example,
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
only:
variables:
- $CURRENT_BRANCH != "develop"
as well as using rules and when condition
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CURRENT_BRANCH == "develop"'
when: never
The pipeline still running. Are there anything that I miss?

Quoting from Gitlab docs https://docs.gitlab.com/ee/ci/yaml/#rules
Rules are evaluated when the pipeline is created, and evaluated in order until the first match. When a match is found, the job is either included or excluded from the pipeline, depending on the configuration.
You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run.
So you can not use variables created in one job to decide if a following job should triggered.
In your case why don't you use the predefined variable CI_COMMIT_REF_NAME directly?
CI_COMMIT_REF_NAME: The branch or tag name for which project is built
Change this
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CURRENT_BRANCH == "develop"'
when: never
To
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CI_COMMIT_REF_NAME == "develop"'
when: never

do you provide the full code of the job build_image?
The build_image job needs the info.env artifacts of your first job. Can you try to add this?
needs:
- job: '<first-job-name>'
artifacts: true

Related

Gitlab CI conditionally override a template variable based on which files were changes in a MR

I have template that looks like this:
/templates/.copy-echo.yml:
workflow:
rules:
- if: '$CI_COMMIT_REF_NAME == "master"'
variables:
FILE_PATH: /test_conf_1.txt
DESTINATION_HOST: somehost
stages:
- copy
- echo
copy step 1/2:
rules:
- changes:
- ${FILE_PATH}
stage: copy
script: |
echo "Add copy here!"
copy step 2/2:
rules:
- changes:
- ${FILE_PATH}
stage: echo
script: |
printenv
echo ${DESTINATION_HOST}
Now in my .gitlab-ci.yml:
include: '/templates/copy-echo.yml'
variables:
FILE_PATH: /test_conf_1.txt
DESTINATION_HOST: somehost2
Now what I want is conditionally assign a value to DESTINATION_HOST variable depending on which file got changed in merged request.
For e.g. if the merge request had updates to file test_conf_2.txt then the value for DESTINATION_HOST should be somehost2 and if the merge request had updates to file test_conf_3.txt then the value for DESTINATION_HOST should be somehost3.
Is it possible to achieve this?
You can use rules:variables: https://docs.gitlab.com/ee/ci/yaml/#rulesvariables
Example:
my job:
script: printenv
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- test_conf_2.txt
variables:
DESTINATION_HOST: "somehost2"
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- test_conf_3.txt
variables:
DESTINATION_HOST: "somehost3"
But it doesn't work if you launch pipeline manually:
You should use rules: changes only with branch pipelines or merge
request pipelines. You can use rules: changes with other pipeline
types, but rules: changes always evaluates to true when there is no
Git push event. Tag pipelines, scheduled pipelines, manual pipelines,
and so on do not have a Git push event associated with them. A rules:
changes job is always added to those pipelines if there is no if that
limits the job to branch or merge request pipelines.

Gitlab CI/CD switching job to manual mode dynamically

Is there any way to switch a job to manual mode (and vice-versa) dynamically?
So, instead of having when: manual in gitlab-ci.yml file, it would be switched to manual (or the opposite) according to any dynamic checking.
Example: Lacking of configuration from environment variables that prevents it to run, but the user could run it later if they set the variables at the launch time.
Just playing with the current syntax, it could be like:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
when: manual && [[ ! $my_var ]] # Using shell syntax just as an example of the condition
Instead of failing:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- [[ ! $my_var ]] && echo "my_var is undefined" && exit 1
- echo "Deploying $my_var..."
You can try using rules to set a condition based on a variable you set in the script of the job.
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- export my_var
rules:
- if: $my_var
when: manual
https://docs.gitlab.com/ee/ci/yaml/#rules-attributes
You can make use of rules if with below example
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
rules:
- if: $my_var
when: on_success
- if: !$my_var
when: manual
So if var is set then it will automatically run. if it is not set then it'll become a manual process
Ref: Gitlab Rules

How to create Gitlab CI rules that are evaluated as AND instead of OR

The following gitlab ci job will run if the variable $CI_COMMIT_TAG is set OR if the ./versions.txt file has changed.
some-job:
script:
- echo "Do some fancy stuff.";
rules:
- if: $CI_COMMIT_TAG
when: always
- changes:
- ./versions.txt
However, what I need is for this job to run when $CI_COMMIT_TAG is set AND ./versions.txt is changed. I don't want the job to run if only one of these evaluates to true. This was the behaviour in only/changes feature, but the only (and except)-feature is less powerful and deprecated.
Is what I want currently possible with gitlab ci?
From Docs:
In the following example:
We run the job manually if Dockerfile or any file in docker/scripts/ has changed AND $VAR == "string value". Otherwise, the job will not be included in the pipeline.
docker build:
script: docker build -t my-image:$CI_COMMIT_REF_SLUG .
rules:
- if: '$VAR == "string value"'
changes: # Will include the job and set to when:manual if any of the follow paths match a modified file.
- Dockerfile
- docker/scripts/*
when: manual
Your code will look something like this.
some-job:
script:
- echo "Do some fancy stuff.";
rules:
- if: $CI_COMMIT_TAG
changes:
- versions.txt
when: manual

Conditional variables in gitlab-ci.yml

Depending on branch the build comes from I need to use slightly different command line arguments. Particularly I would like to upload snapshot nexus artifacts when building from a branch, and release artifact when building off master.
Is there a way to conditionally alter variables?
I tried to use except/only keywords like this
stages:
- stage
variables:
TYPE: Release
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
upload:
extends:
- .upload_common
- .upload_snapshot
Unfortunately it skips whole upload step when building off master.
The reason I am using 'extends' pattern here is that I have win and mac platforms which use slightly different variables substitution syntax ($ vs %). I also have a few different build configuration - Debug/Release, 32bit/64bit.
The code below actually works, but I had to duplicate steps for release and snapshot, one is enabled at a time.
stages:
- stage
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
.upload_release:
variables:
TYPE: "Release"
only:
- master
upload_release:
extends:
- .upload_common
- .upload_release
upload_snapshot:
extends:
- .upload_common
- .upload_snapshot
The code gets much larger when snapshot/release configuration is multiplied by Debug/Release, Mac/Win, and 32/64bits. I would like to keep number of configurations at minimum.
Having ability to conditionally altering just a few variables would help me reducing this code a lot.
Another addition in GitLab 13.7 are the rules:variables. This allows some logic in setting vars:
job:
variables:
DEPLOY_VARIABLE: "default-deploy"
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
variables: # Override DEPLOY_VARIABLE defined
DEPLOY_VARIABLE: "deploy-production" # at the job level.
- if: $CI_COMMIT_REF_NAME =~ /feature/
variables:
IS_A_FEATURE: "true" # Define a new variable.
script:
- echo "Run script with $DEPLOY_VARIABLE as an argument"
- echo "Run another script if $IS_A_FEATURE exists"
Unfortunatelly YAML-anchors or GitLab-CI's extends don't seem to allow to combine things in script array of commands as of today.
I use built-in variable CI_COMMIT_REF_NAME in combination with global or job-only before_script to solve similar tasks without repeating myself.
Here is an example of my workaround on how to set environment variable to different values dynamically for PROD and DEV during delivery or deployment:
.provide ssh private deploy key: &provide_ssh_private_deploy_key
before_script:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- |
if [ "$CI_COMMIT_REF_NAME" == "master" ]; then
echo "$SSH_PRIVATE_DEPLOY_KEY_PROD" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are in master (PROD)"
else
echo "$SSH_PRIVATE_DEPLOY_KEY_DEV" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are NOT in master (DEV)"
fi
- chmod 600 ~/.ssh/id_rsa
deliver-via-ssh:
stage: deliver
<<: *provide_ssh_private_deploy_key
script:
- echo Stage is deliver
- echo $MY_DYNAMIC_VAR
- ssh ...
Also consider this workaround for concatenation of "script" commands: https://stackoverflow.com/a/57209078/470108
hopefully it helps.
A nice way to prepare variables for other jobs is the dotenv report artifact. Unfortunately, these variables cannot be used in many places, but if you only need to access them from other jobs scripts, this is the way:
# prepare environment variables for other jobs
env:
stage: .pre
script:
# Set application version from GIT tag or fake it
- echo "APPVERSION=${CI_COMMIT_TAG:-0.1-dev-$CI_COMMIT_REF_SLUG}+$CI_COMMIT_SHORT_SHA" | tee -a .env
artifacts:
reports:
dotenv: .env
In the script of this job you can conditionally prepare and write your environment values into a file, then make a dotenv artifact out of it. Subsequent - or better yet dependent - jobs will pick up the variables for their scripts from there.

How to have a gitlab CI job to be triggred only manual on branches and always automatically on master?

I have a pages job that I want to run manual on branches, but automatically triggered on master:
pages:
stage: deploy
cache:
paths:
- public
script:
- scripts/pages.sh
artifacts:
paths:
- public
expire_in: 2 days
So I want a combination of:
only:
- master
when: always
only:
- branches
except:
- master
when: manual
Is that possible?
This should be possible to do if you use GitLab CI rules. This is an example where the shell is powershell and it shows the current time and branch/tag name:
pages:
script:
- mkdir public
- date > public\index.html
- $CI_COMMIT_REF_NAME >> public\index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- if: '$CI_COMMIT_BRANCH == null'
when: never
- when: manual
GitLab matches each individual rule from top to bottom. If the branch is named 'master', the job gets marked with when: always. If the branch name is null, this is a tag, and the job is marked with never. If this is not a branch named master, nor a tag, this is a normal branch, and the job is marked with manual.
As Aleksey Tsalolikhin described, you can remove this rule:
- if: '$CI_COMMIT_BRANCH == null'
when: never
You will then get the option to run the pipeline for your tags as well, like this:
If this is what you want or not, that is up to you.
I've tweaked the answer by MrBerta -- the third command was missing the echo command.
I also changed the slashes from backslashes to regular forward slashes so I can use the Linux shell rather than Powershell.
It now works.
Here is the gitlab-ci.yml file -- with credit to MrBerta.
pages:
script:
- mkdir public
- date > public/index.html
- echo $CI_COMMIT_REF_NAME >> public/index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- if: '$CI_COMMIT_BRANCH == null'
when: never
- when: manual
I tried pushing to master, and my GitLab Pages content was updated as expected; and I tried pushing to a feature branch, and the manual "Play" button came up in the CI/CD pipeline UI.
When I pushed a tag (with detached head, i.e., not on any branch), I could not test it -- GitLab CI did not run a pipeline automatically, and when I tried to Run Pipeline, and picked my tag, GitLab threw an error: "The form contains the following error: No stages / jobs for this pipeline."
So, I would simplify this to:
pages:
script:
- mkdir public
- date > public/index.html
- echo $CI_COMMIT_REF_NAME >> public/index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- when: manual
This pages job runs manually on branches (and tags but I couldn't test it), but automatically triggered on master, as the original poster requested.
You will need to define two stages. you can either copy/paste or use anchors:
.deploy_stage: &deploy_stage
stage: deploy
cache:
paths:
- public
script:
- scripts/pages.sh
artifacts:
paths:
- public
expire_in: 2 days
deploy_manual:
<<: *deploy_stage
only:
- branches
when: manual
deploy_master:
<<: *deploy_stage
only:
- master