Depending on branch the build comes from I need to use slightly different command line arguments. Particularly I would like to upload snapshot nexus artifacts when building from a branch, and release artifact when building off master.
Is there a way to conditionally alter variables?
I tried to use except/only keywords like this
stages:
- stage
variables:
TYPE: Release
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
upload:
extends:
- .upload_common
- .upload_snapshot
Unfortunately it skips whole upload step when building off master.
The reason I am using 'extends' pattern here is that I have win and mac platforms which use slightly different variables substitution syntax ($ vs %). I also have a few different build configuration - Debug/Release, 32bit/64bit.
The code below actually works, but I had to duplicate steps for release and snapshot, one is enabled at a time.
stages:
- stage
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
.upload_release:
variables:
TYPE: "Release"
only:
- master
upload_release:
extends:
- .upload_common
- .upload_release
upload_snapshot:
extends:
- .upload_common
- .upload_snapshot
The code gets much larger when snapshot/release configuration is multiplied by Debug/Release, Mac/Win, and 32/64bits. I would like to keep number of configurations at minimum.
Having ability to conditionally altering just a few variables would help me reducing this code a lot.
Another addition in GitLab 13.7 are the rules:variables. This allows some logic in setting vars:
job:
variables:
DEPLOY_VARIABLE: "default-deploy"
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
variables: # Override DEPLOY_VARIABLE defined
DEPLOY_VARIABLE: "deploy-production" # at the job level.
- if: $CI_COMMIT_REF_NAME =~ /feature/
variables:
IS_A_FEATURE: "true" # Define a new variable.
script:
- echo "Run script with $DEPLOY_VARIABLE as an argument"
- echo "Run another script if $IS_A_FEATURE exists"
Unfortunatelly YAML-anchors or GitLab-CI's extends don't seem to allow to combine things in script array of commands as of today.
I use built-in variable CI_COMMIT_REF_NAME in combination with global or job-only before_script to solve similar tasks without repeating myself.
Here is an example of my workaround on how to set environment variable to different values dynamically for PROD and DEV during delivery or deployment:
.provide ssh private deploy key: &provide_ssh_private_deploy_key
before_script:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- |
if [ "$CI_COMMIT_REF_NAME" == "master" ]; then
echo "$SSH_PRIVATE_DEPLOY_KEY_PROD" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are in master (PROD)"
else
echo "$SSH_PRIVATE_DEPLOY_KEY_DEV" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are NOT in master (DEV)"
fi
- chmod 600 ~/.ssh/id_rsa
deliver-via-ssh:
stage: deliver
<<: *provide_ssh_private_deploy_key
script:
- echo Stage is deliver
- echo $MY_DYNAMIC_VAR
- ssh ...
Also consider this workaround for concatenation of "script" commands: https://stackoverflow.com/a/57209078/470108
hopefully it helps.
A nice way to prepare variables for other jobs is the dotenv report artifact. Unfortunately, these variables cannot be used in many places, but if you only need to access them from other jobs scripts, this is the way:
# prepare environment variables for other jobs
env:
stage: .pre
script:
# Set application version from GIT tag or fake it
- echo "APPVERSION=${CI_COMMIT_TAG:-0.1-dev-$CI_COMMIT_REF_SLUG}+$CI_COMMIT_SHORT_SHA" | tee -a .env
artifacts:
reports:
dotenv: .env
In the script of this job you can conditionally prepare and write your environment values into a file, then make a dotenv artifact out of it. Subsequent - or better yet dependent - jobs will pick up the variables for their scripts from there.
Related
I am wanting to store the output from a script in a variable for use in subsequent commands from within Gitlab CI.
Here is the script:
image: ...
build c-ares:
variables:
CARES_ARTIFACTS_DIR: "-"
script:
- CARES_ARTIFACTS_DIR=$(./build-c-ares.sh)
after_script:
- echo $CARES_ARTIFACTS_DIR
artifacts:
name: CARES_ARTIFACTS
paths:
- $CARES_ARTIFACTS_DIR
My intention is to:
first declare the variable CARES_ARTIFACTS_DIR with global scope
Set the variable value using the output from the build-c-ares.sh script
Recover the output from the build-c-ares.sh script on a later command using the variable
My code does not behave as intended - on dereferencing the variable I find it contains the original value it was assigned at declaration:
$ CARES_ARTIFACTS_DIR=$(./build-c-ares.sh)
Cloning into 'c-ares'...
Running after_script
00:01
Running after script...
$ echo $CARES_ARTIFACTS_DIR
-
Uploading artifacts for successful job
00:00
Uploading artifacts...
WARNING: -: no matching files. Ensure that the artifact path is relative to the working directory
ERROR: No files to upload
It is probably easier to just redirect the script output to a file and define that as an artifact.
Something similar to:
image: ...
build c-ares:
script:
- ./build-c-ares.sh > script_output
- cat script_output
artifacts:
paths:
- script_output
In regards to the specific issue, the variables used in the "artefacts" step will again use the variable initialisation defined for the job. Both the "artefacts" and the "script" steps for the job will start with the custom CARES_ARTIFACTS_DIR variable set to the value "-":
build c-ares:
variables:
CARES_ARTIFACTS_DIR: "-"
script:
# $CARES_ARTIFACTS_DIR=="-"
- CARES_ARTIFACTS_DIR=$(./build-c-ares.sh)
# $CARES_ARTIFACTS_DIR=="hello from build-c-ares.sh"
- echo $CARES_ARTIFACTS_DIR # prints "hello from build-c-ares.sh"
after_script:
# $CARES_ARTIFACTS_DIR=="-"
- echo $CARES_ARTIFACTS_DIR # prints "-"
Fundamentally, Gitlab variables cannot feed information across job steps as intended in the original post. My subjective opinion is to keep steps independent where possible and restrict input to artefacts from upstream jobs or variables explicitly defined in the pipeline script or settings.
I don't get it. I have two repos. One for infrastructure, and the other for project code. Inside project code, I have .gitlab-ci.yml file that will include one job for creating env variables, another include file that will include other stages inside the job. Every other stage is triggered, but the first include is not triggered no matter what I do. What am I doing wrong?
Project gitlab-ci
# Stages
stages:
- pre-build test
- build
- post-build test
- deploy
- environment
- e2e
# Main Variables
variables:
GIT_SUBMODULE_STRATEGY: normal
IMAGE_VERSION: "latest"
CARGO_HOME: $CI_PROJECT_DIR/.cargo
FF_USE_FASTZIP: "true"
ARTIFACT_COMPRESSION_LEVEL: "fast"
CACHE_COMPRESSION_LEVEL: "fastest"
STAGING_BRANCH: "master"
VARIABLES_FILE: variables.txt
# Include main CI files from infrastructure repository
include:
- project: 'project/repo-one'
ref: master
file: '/gitlab-ci/env/ci-app-env.yml'
- project: 'project/repo-one'
ref: master
file: '/gitlab-ci/app/ci-merge-request.yml'
env ci file
env mr:
stage: .pre
before_script:
- TEST_VAR="TEST"
- IMAGE_PATH="/var/www"
script:
- echo "export TEST_VAR=$TEST_VAR" > $VARIABLES_FILE
- echo "export IMAGE_PATH=$IMAGE_PATH" > $VARIABLES_FILE
- cat $VARIABLES_FILE
artifacts:
paths:
- $VARIABLES_FILE
I recognize that within a pipeline step I can run a simple export, like:
commands:
- export MY_ENV_VAR=$(my-command)
...but if I want to use this env var throughout the whole pipeline, is it possible to do something like this:
environment:
MY_ENV_VAR: $(my-command)
When I do this, I get yaml: unmarshal errors: line 23: cannot unmarshal !!seq into map[string]*yaml.Variable which suggests this isn't possible. My end goal is to write a drone plugin that accepts the output of $(...) as one if it's settings. I'd prefer to have the drone plugin not run the command, but just use the output.
I've also attempted to use step dependencies to export an env var, however it's state doesn't carry over between steps:
- name: export
image: bash
commands:
- export MY_VAR=$(my-command)
- name: echo
image: bash
depends_on:
- export
commands:
- echo $MY_VAR // empty
Writing the command output to a script file might be a better way to do what you want, since filesystem changes are persisted between individual steps.
---
kind: pipeline
type: docker
steps:
- name: generate-script
image: bash
commands:
# - my-command > plugin-script.sh
- printf "echo Fetching Google;\n\ncurl -I https://google.com/" > plugin-script.sh
- name: test-script-1
image: curlimages/curl
commands:
- sh plugin-script.sh
- name: test-script-2
image: curlimages/curl
commands:
- sh plugin-script.sh
From Drone's Docker pipeline documentation:
Workspace
Drone automatically creates a temporary volume, known as your workspace, where it clones your repository. The workspace is the current working directory for each step in your pipeline.
Because the workspace is a volume, filesystem changes are persisted between pipeline steps. In other words, individual steps can communicate and share state using the filesystem.
⚠ Workspace volumes are ephemeral. They are created when the pipeline starts and destroyed after the pipeline completes.
if cant execute command in environment period.
maybe you can define a "command string" in "environment" block, like:
environment:
MY_ENV_VAR: 'echo "this is command to execute"' # note the single quote
then in commands block,
commands:
- eval $MY_ENV_VAR
worth a try
I'm trying to force the pipeline to skip running job whenever the project's branch is equal to develop. I've created the custom variable name $CURRENT_BRANCH that gathering from predefined variable $CI_COMMIT_REF_NAME
- export CURRENT_BRANCH=`echo ${CI_COMMIT_REF_NAME} | awk -F '/' '{print $1}'`
The $CURRENT_BRANCH will pass along the first stage and it will save in the artifact
- >
if [ ${CURRENT_BRANCH} == 'feature' ]; then
export CURRENT_BRANCH="feature"
export FEATURE_NAME=`echo ${CI_COMMIT_REF_NAME} | awk -F '/' '{print $2}'`
echo "FEATURE_NAME=${FEATURE_NAME}" >> info.env
echo "PROJECT_NAME=${FEATURE_NAME}" >> info.env
else
export CURRENT_BRANCH=${CURRENT_BRANCH}
echo "PROJECT_NAME=${CURRENT_BRANCH}" >> info.env
fi
- echo "CURRENT_BRANCH=${CURRENT_BRANCH}" >> info.env
artifacts:
reports:
dotenv: info.env
Then in the next stage until the last stage, I've tried to skip running the pipeline with only/except and rules/when for example,
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
only:
variables:
- $CURRENT_BRANCH != "develop"
as well as using rules and when condition
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CURRENT_BRANCH == "develop"'
when: never
The pipeline still running. Are there anything that I miss?
Quoting from Gitlab docs https://docs.gitlab.com/ee/ci/yaml/#rules
Rules are evaluated when the pipeline is created, and evaluated in order until the first match. When a match is found, the job is either included or excluded from the pipeline, depending on the configuration.
You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run.
So you can not use variables created in one job to decide if a following job should triggered.
In your case why don't you use the predefined variable CI_COMMIT_REF_NAME directly?
CI_COMMIT_REF_NAME: The branch or tag name for which project is built
Change this
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CURRENT_BRANCH == "develop"'
when: never
To
build_image:
image: docker:19.03
services:
- docker:dind
stage: release
rules:
- if: '$CI_COMMIT_REF_NAME == "develop"'
when: never
do you provide the full code of the job build_image?
The build_image job needs the info.env artifacts of your first job. Can you try to add this?
needs:
- job: '<first-job-name>'
artifacts: true
Is there any way to switch a job to manual mode (and vice-versa) dynamically?
So, instead of having when: manual in gitlab-ci.yml file, it would be switched to manual (or the opposite) according to any dynamic checking.
Example: Lacking of configuration from environment variables that prevents it to run, but the user could run it later if they set the variables at the launch time.
Just playing with the current syntax, it could be like:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
when: manual && [[ ! $my_var ]] # Using shell syntax just as an example of the condition
Instead of failing:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- [[ ! $my_var ]] && echo "my_var is undefined" && exit 1
- echo "Deploying $my_var..."
You can try using rules to set a condition based on a variable you set in the script of the job.
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- export my_var
rules:
- if: $my_var
when: manual
https://docs.gitlab.com/ee/ci/yaml/#rules-attributes
You can make use of rules if with below example
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
rules:
- if: $my_var
when: on_success
- if: !$my_var
when: manual
So if var is set then it will automatically run. if it is not set then it'll become a manual process
Ref: Gitlab Rules