Why jobs running with set "only" keyword? - gitlab-ci

I tried to spice my gitlab gitflow with gitlab-cicd. My goal is to create a job which is running while a merge request and check if the source branch for this merge request is either a hotfix/* or release/* branch. A simple bash script is in charge for this:
#!/bin/bash
if [[ "$1" =~ ^hotfix/* ]]; then
echo "Begin Hotfix Merge"
exit 0
else
echo "ERROR: Directory does not exist."
exit 1
fi
where $1 is the predefined variable CI_MERGE_REQUEST_SOURCE_BRANCH_NAME as argument of the script.
My .gitlab-ci.yaml looks the following:
image: ubuntu:latest
stages:
- test
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
test1:
tags:
- docker
- linux
stage: test
script:
- echo "Do a test here"
- echo "For example run a test suite"
- echo $PWD
- ls -la .gitlab/
job1:
tags:
- docker
- linux
script:
- echo "This job runs in merge request pipelines"
- echo $PWD
- ls -la .gitlab/workflow-scripts/
- echo "$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME"
- bash ./.gitlab/workflow-scripts/check_sourcebranch.sh "$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME"
only:
- merge_requests
- master
I am not shure if the workflow:rules distrubs anything. If I now start a merge request the pipeline runs without any issues inside the merge request, but if I aprove the request and the pipeline gets triggered again, both jobs are running.
Any Idea, why job1 is triggered, even with the only rule?
EDIT: I removed the part with the unexpected error. This Error was a residual of a classic copy&pasta and was actually expected in this scenario.

Fixed job for testing branch name on MR:
stages:
- tests
image: ubuntu:latest
test-mr-branch-name:
stage: tests
only:
- merge_requests
tags:
- docker
- linux
script:
- |
if [[ ! "${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}" =~ ^(hotfix|release)\/.+$ ]]; then
echo "ERROR: Wrong Merge!"
exit 1
fi
p.s. Operator between values of only tag is or. Read me.
p.s. You may set multiline bash script in Gitlab CI.

Related

GITLAB CI/CD: How to know if a pipeline is a tag pipeline, an MR pipeline or a scheduled pipeline?

There's a part of the script which I want to run only if it's a tag pipeline. How do I represent that as a condition?
You can use rules or the only keyword to run only on tags or merge requests.
rules:
job1:
script:
- echo "This job runs in merge request pipelines"
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
only:
job1:
script:
- echo "This job runs in merge request pipelines"
only:
- merge_requests
To run only on tags you can use:
job1:
script:
- echo "This job runs only on tags"
only:
- tags
Docs: https://docs.gitlab.com/ee/ci/yaml/#onlyrefs--exceptrefs

How to add rule in the after script for gitlab-ci.yml. after_script config should be an array containing strings and arrays of strings error

I try to create yml file where the build calls another test(project) if the build succeeds.
So I try to add a rule when on_success into the after_script block:
before_script:
- echo "Before script section"
- echo "For example you might run an update here or install a build dependency"
- echo "Or perhaps you might print out some debugging details"
after_script:
when: on_success
script:
- echo "success"
test2:
stage: test
script:
- echo "Hello world"
deploy:
stage: deploy
script:
- echo "Do your build here"
test1:
stage: test
script:
- exit 1
In this result, I got the error GitLab CI configuration is invalid: after_script config should be an array containing strings and arrays of strings
How to implement this logic in the correct way?
When you put before_script: and after_script: on top of the level in the .gitlab-ci.yml file, you're basically saying that all jobs (that includes test2, deploy and test) will run before_script: and after_script:.
I try to create yml file where the build calls another test(project) if the build succeeds.
In this case, take a look at allow_failure: (see: https://docs.gitlab.com/ee/ci/yaml/#allow_failure). By default, allow_failure: is already false. I can't really figure out from the sample .gitlab-ci.yml file that you shared about what you actually wanted to do, but I can give you a quick example:
stages:
- build
- test
- deploy
build:
stage: build
script:
- echo build something
test:
stage: test
script:
- echo test something
deploy:
stage: deploy
script:
- echo test something

Unable to skip running pipeline based on variable in Gitlab CI

I'm trying to force the pipeline to skip running job whenever the project's branch is equal to develop. I've created the custom variable name $CURRENT_BRANCH that gathering from predefined variable $CI_COMMIT_REF_NAME
- export CURRENT_BRANCH=`echo ${CI_COMMIT_REF_NAME} | awk -F '/' '{print $1}'`
The $CURRENT_BRANCH will pass along the first stage and it will save in the artifact
- >
if [ ${CURRENT_BRANCH} == 'feature' ]; then
export CURRENT_BRANCH="feature"
export FEATURE_NAME=`echo ${CI_COMMIT_REF_NAME} | awk -F '/' '{print $2}'`
echo "FEATURE_NAME=${FEATURE_NAME}" >> info.env
echo "PROJECT_NAME=${FEATURE_NAME}" >> info.env
else
export CURRENT_BRANCH=${CURRENT_BRANCH}
echo "PROJECT_NAME=${CURRENT_BRANCH}" >> info.env
fi
- echo "CURRENT_BRANCH=${CURRENT_BRANCH}" >> info.env
artifacts:
reports:
dotenv: info.env
Then in the next stage until the last stage, I've tried to skip running the pipeline with only/except and rules/when for example,
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
only:
variables:
- $CURRENT_BRANCH != "develop"
as well as using rules and when condition
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CURRENT_BRANCH == "develop"'
when: never
The pipeline still running. Are there anything that I miss?
Quoting from Gitlab docs https://docs.gitlab.com/ee/ci/yaml/#rules
Rules are evaluated when the pipeline is created, and evaluated in order until the first match. When a match is found, the job is either included or excluded from the pipeline, depending on the configuration.
You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run.
So you can not use variables created in one job to decide if a following job should triggered.
In your case why don't you use the predefined variable CI_COMMIT_REF_NAME directly?
CI_COMMIT_REF_NAME: The branch or tag name for which project is built
Change this
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CURRENT_BRANCH == "develop"'
when: never
To
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CI_COMMIT_REF_NAME == "develop"'
when: never
do you provide the full code of the job build_image?
The build_image job needs the info.env artifacts of your first job. Can you try to add this?
needs:
- job: '<first-job-name>'
artifacts: true

Gitlab CI/CD switching job to manual mode dynamically

Is there any way to switch a job to manual mode (and vice-versa) dynamically?
So, instead of having when: manual in gitlab-ci.yml file, it would be switched to manual (or the opposite) according to any dynamic checking.
Example: Lacking of configuration from environment variables that prevents it to run, but the user could run it later if they set the variables at the launch time.
Just playing with the current syntax, it could be like:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
when: manual && [[ ! $my_var ]] # Using shell syntax just as an example of the condition
Instead of failing:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- [[ ! $my_var ]] && echo "my_var is undefined" && exit 1
- echo "Deploying $my_var..."
You can try using rules to set a condition based on a variable you set in the script of the job.
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- export my_var
rules:
- if: $my_var
when: manual
https://docs.gitlab.com/ee/ci/yaml/#rules-attributes
You can make use of rules if with below example
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
rules:
- if: $my_var
when: on_success
- if: !$my_var
when: manual
So if var is set then it will automatically run. if it is not set then it'll become a manual process
Ref: Gitlab Rules

Conditional variables in gitlab-ci.yml

Depending on branch the build comes from I need to use slightly different command line arguments. Particularly I would like to upload snapshot nexus artifacts when building from a branch, and release artifact when building off master.
Is there a way to conditionally alter variables?
I tried to use except/only keywords like this
stages:
- stage
variables:
TYPE: Release
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
upload:
extends:
- .upload_common
- .upload_snapshot
Unfortunately it skips whole upload step when building off master.
The reason I am using 'extends' pattern here is that I have win and mac platforms which use slightly different variables substitution syntax ($ vs %). I also have a few different build configuration - Debug/Release, 32bit/64bit.
The code below actually works, but I had to duplicate steps for release and snapshot, one is enabled at a time.
stages:
- stage
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
.upload_release:
variables:
TYPE: "Release"
only:
- master
upload_release:
extends:
- .upload_common
- .upload_release
upload_snapshot:
extends:
- .upload_common
- .upload_snapshot
The code gets much larger when snapshot/release configuration is multiplied by Debug/Release, Mac/Win, and 32/64bits. I would like to keep number of configurations at minimum.
Having ability to conditionally altering just a few variables would help me reducing this code a lot.
Another addition in GitLab 13.7 are the rules:variables. This allows some logic in setting vars:
job:
variables:
DEPLOY_VARIABLE: "default-deploy"
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
variables: # Override DEPLOY_VARIABLE defined
DEPLOY_VARIABLE: "deploy-production" # at the job level.
- if: $CI_COMMIT_REF_NAME =~ /feature/
variables:
IS_A_FEATURE: "true" # Define a new variable.
script:
- echo "Run script with $DEPLOY_VARIABLE as an argument"
- echo "Run another script if $IS_A_FEATURE exists"
Unfortunatelly YAML-anchors or GitLab-CI's extends don't seem to allow to combine things in script array of commands as of today.
I use built-in variable CI_COMMIT_REF_NAME in combination with global or job-only before_script to solve similar tasks without repeating myself.
Here is an example of my workaround on how to set environment variable to different values dynamically for PROD and DEV during delivery or deployment:
.provide ssh private deploy key: &provide_ssh_private_deploy_key
before_script:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- |
if [ "$CI_COMMIT_REF_NAME" == "master" ]; then
echo "$SSH_PRIVATE_DEPLOY_KEY_PROD" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are in master (PROD)"
else
echo "$SSH_PRIVATE_DEPLOY_KEY_DEV" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are NOT in master (DEV)"
fi
- chmod 600 ~/.ssh/id_rsa
deliver-via-ssh:
stage: deliver
<<: *provide_ssh_private_deploy_key
script:
- echo Stage is deliver
- echo $MY_DYNAMIC_VAR
- ssh ...
Also consider this workaround for concatenation of "script" commands: https://stackoverflow.com/a/57209078/470108
hopefully it helps.
A nice way to prepare variables for other jobs is the dotenv report artifact. Unfortunately, these variables cannot be used in many places, but if you only need to access them from other jobs scripts, this is the way:
# prepare environment variables for other jobs
env:
stage: .pre
script:
# Set application version from GIT tag or fake it
- echo "APPVERSION=${CI_COMMIT_TAG:-0.1-dev-$CI_COMMIT_REF_SLUG}+$CI_COMMIT_SHORT_SHA" | tee -a .env
artifacts:
reports:
dotenv: .env
In the script of this job you can conditionally prepare and write your environment values into a file, then make a dotenv artifact out of it. Subsequent - or better yet dependent - jobs will pick up the variables for their scripts from there.