Variable inside variable gitlab ci - variables

Is there a way to use predefined variable inside custom variable in gitlab ci like this:
before_script:
- cat "${$CI_COMMIT_REF_NAME}" >> .env
to extract the name of branch from $CI_COMMIT_REF_NAME and use it as a name of custom variable
Update:

Check out GitLab 14.3 (September 2021)
Use variables in other variables
CI/CD pipeline execution scenarios can depend on expanding variables declared in a pipeline or using GitLab predefined variables within another variable declaration.
In 14.3, we are enabling the “variables inside other variables” feature on GitLab SaaS.
Now you can define a variable and use it in another variable definition within the same pipeline.
You can also use GitLab predefined variables inside of another variable declaration.
This feature simplifies your pipeline definition and eliminates pipeline management issues caused by the duplicating of variable data.
Note - for GitLab self-managed users the feature is disabled by default.
To use this feature, your GitLab administrator will need to enable the feature flag.
(demo -- video)
See Documentation and Issue.
dba asks in the comments:
Does this include or exclude using globally defined variables?
dba's own answer:
Global variables can be reused, but they need the local_var: ${global_var} syntax with recursive expansion (independent of the shell).

Check if this matches gitlab-org/gitlab-runner issue 1809:
Description
In the .gitlab-ci.yml file, a user can define a variable and use it in another variable definition within the same .gitlab-ci.yml file.
A user can also use a GitLab pre-defined variable in a variable declaration.
Example
variables:
variable_1: "foo" # here, variable_1 is assigned the value foo
variable_2: "${variable_1}" # variable_2 is assigned the value variable_1.
# The expectation is that the value in variable_2 = value set for variable_1
If it is, it should be completed/implemented for GitLab 14.1 (July 2021)

Lots of options.
But you could just pass the predefined var into the .env
image: busybox:latest
variables:
MY_CUSTOM_VARIABLE: $CI_JOB_STAGE
ANIMAL_TESTING: "cats"
before_script:
- echo "Before script section"
- echo $CI_JOB_STAGE
- echo $MY_CUSTOM_VARIABLE
- echo $MY_CUSTOM_VARIABLE >> .env
- echo $CI_COMMIT_BRANCH >> .env
- cat .env
example pipeline output
$ echo "Before script section"
Before script section
$ echo $CI_JOB_STAGE
build
$ echo $MY_CUSTOM_VARIABLE
build
$ echo $MY_CUSTOM_VARIABLE >> .env
$ echo $CI_COMMIT_BRANCH >> .env
$ cat .env
build
exper/ci-var-into-env
$ echo "Do your build here"
Do your build here
or pass it in earlier.
image: busybox:latest
variables:
MY_CUSTOM_VARIABLE: "${CI_JOB_STAGE}"
ANIMAL_TESTING: "cats"
before_script:
- echo "Before script section"
- echo $CI_JOB_STAGE
- echo $MY_CUSTOM_VARIABLE
- echo $MY_CUSTOM_VARIABLE >> .env
- cat .env
example: https://gitlab.com/codeangler/make-ci-var-custom-var-in-script/-/blob/master/.gitlab-ci.yml

Related

transferring strings across gitlab ci tasks stages using variables

I am wanting to store the output from a script in a variable for use in subsequent commands from within Gitlab CI.
Here is the script:
image: ...
build c-ares:
variables:
CARES_ARTIFACTS_DIR: "-"
script:
- CARES_ARTIFACTS_DIR=$(./build-c-ares.sh)
after_script:
- echo $CARES_ARTIFACTS_DIR
artifacts:
name: CARES_ARTIFACTS
paths:
- $CARES_ARTIFACTS_DIR
My intention is to:
first declare the variable CARES_ARTIFACTS_DIR with global scope
Set the variable value using the output from the build-c-ares.sh script
Recover the output from the build-c-ares.sh script on a later command using the variable
My code does not behave as intended - on dereferencing the variable I find it contains the original value it was assigned at declaration:
$ CARES_ARTIFACTS_DIR=$(./build-c-ares.sh)
Cloning into 'c-ares'...
Running after_script
00:01
Running after script...
$ echo $CARES_ARTIFACTS_DIR
-
Uploading artifacts for successful job
00:00
Uploading artifacts...
WARNING: -: no matching files. Ensure that the artifact path is relative to the working directory
ERROR: No files to upload
It is probably easier to just redirect the script output to a file and define that as an artifact.
Something similar to:
image: ...
build c-ares:
script:
- ./build-c-ares.sh > script_output
- cat script_output
artifacts:
paths:
- script_output
In regards to the specific issue, the variables used in the "artefacts" step will again use the variable initialisation defined for the job. Both the "artefacts" and the "script" steps for the job will start with the custom CARES_ARTIFACTS_DIR variable set to the value "-":
build c-ares:
variables:
CARES_ARTIFACTS_DIR: "-"
script:
# $CARES_ARTIFACTS_DIR=="-"
- CARES_ARTIFACTS_DIR=$(./build-c-ares.sh)
# $CARES_ARTIFACTS_DIR=="hello from build-c-ares.sh"
- echo $CARES_ARTIFACTS_DIR # prints "hello from build-c-ares.sh"
after_script:
# $CARES_ARTIFACTS_DIR=="-"
- echo $CARES_ARTIFACTS_DIR # prints "-"
Fundamentally, Gitlab variables cannot feed information across job steps as intended in the original post. My subjective opinion is to keep steps independent where possible and restrict input to artefacts from upstream jobs or variables explicitly defined in the pipeline script or settings.

Variables in gitlab pipeline don't seem to be available to helm file being deployed

I'm trying to deploy a helm file using gitlab ci/cd process, and it seems like variables aren't being provided correctly.
I've added some echo statements to make sure the runner knew about it:
echo $CI_ENVIRONMENT_NAME
devplatform
$ echo $DOCKER_REGISTRY
1234abcd.dkr.ecr.us-west-2.amazonaws.com
$ echo $CI_PROJECT_NAME
gitlab-project
$ echo $CI_COMMIT_SHA
7d3f491cf........7c73f65daec4d
$ export IMAGE_NAME="$DOCKER_REGISTRY/$CI_PROJECT_NAME"
$ export IMAGE_TAG=$CI_COMMIT_SHA
$ echo $IMAGE_NAME
abcd1234.dkr.ecr.us-west-2.amazonaws.com/gitlabproject
$ echo $IMAGE_TAG
7d3f491.........3100e7c73f65daec4d
$ gl-helmfile --environment $CI_ENVIRONMENT_NAME --namespace $CI_ENVIRONMENT_NAME --state-values-set image_name="$DOCKER_REGISTRY/$CI_PROJECT_NAME",image_tag=$CI_COMMIT_SHA sync
however when gl-helmfile deployes the helmfile, which has some values being set using "env" (such as "env $APP_PORT"), the resulting pods have blank values for things using "env". The only way I've been able to get around this is by setting blank values in values file, like
image_tag=""
image_name=""
app_port=""
and then setting the helmfile to use .Values.image_name, and then using:
--state-values-set image_name="$DOCKER_REGISTRY/$CI_PROJECT_NAME",image_tag=$CI_COMMIT_SHA
Is there a way to get environmental variables passed to helm during deployment?

Setting Environment Variable in gitlab-ci.yml in script

Please help me in converting below written GitHub Action to Gitlab CI script. I am new to scripting in Gitlab.
From Github documentation I could understand that the below written line is for setting the value of environment variable. But I couldn't find any resource for setting environment variable in Gitlab.
run: >
DISPLAY=:0 xvfb-run -s '-screen 0 1024x768x24' julia --project=monorepo -e 'using Pkg; Pkg.test("GLMakie", coverage=true)'
&& echo "TESTS_SUCCESSFUL=true" >> $GITHUB_ENV
There are multiple ways to set environment variables, and it depends on what you want to achieve:
use it within the same job
use it in another job
Use it within the same job
In Bash or other Shells you can set an environment variable via export - in your case it would look like:
job:
script:
- DISPLAY=:0 xvfb-run -s '-screen 0 1024x768x24' julia --project=monorepo -e 'using Pkg; Pkg.test("GLMakie", coverage=true)' && export TESTS_SUCCESSFUL=true
- echo $TESTS_SUCCESSFUL #verification that it is set and can be used within the same job
Use it within another job
To handover variables to another job you need to define an artifact:report:dotenv. It is a file which can contain a list of key-value-pairs which will be injected as Environment variable in the follow up jobs.
The structure of the file looks like:
KEY1=VALUE1
KEY2=VALUE2
and the definition in the .gitlab-ci.yml looks like
job:
# ...
artifacts:
reports:
dotenv: <path to file>
and in your case this would look like
job:
script:
- DISPLAY=:0 xvfb-run -s '-screen 0 1024x768x24' julia --project=monorepo -e 'using Pkg; Pkg.test("GLMakie", coverage=true)' && echo "TESTS_SUCCESSFUL=true" >> build.env
artifacts:
reports:
dotenv: build.env
job2:
needs: ["job"]
script:
- echo $TESTS_SUCCESSFUL
see https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job for further information.

Conditional variables in gitlab-ci.yml

Depending on branch the build comes from I need to use slightly different command line arguments. Particularly I would like to upload snapshot nexus artifacts when building from a branch, and release artifact when building off master.
Is there a way to conditionally alter variables?
I tried to use except/only keywords like this
stages:
- stage
variables:
TYPE: Release
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
upload:
extends:
- .upload_common
- .upload_snapshot
Unfortunately it skips whole upload step when building off master.
The reason I am using 'extends' pattern here is that I have win and mac platforms which use slightly different variables substitution syntax ($ vs %). I also have a few different build configuration - Debug/Release, 32bit/64bit.
The code below actually works, but I had to duplicate steps for release and snapshot, one is enabled at a time.
stages:
- stage
.upload_common:
stage: stage
tags: ["Win"]
script:
- echo Uploading %TYPE%
.upload_snapshot:
variables:
TYPE: "Snapshot"
except:
- master
.upload_release:
variables:
TYPE: "Release"
only:
- master
upload_release:
extends:
- .upload_common
- .upload_release
upload_snapshot:
extends:
- .upload_common
- .upload_snapshot
The code gets much larger when snapshot/release configuration is multiplied by Debug/Release, Mac/Win, and 32/64bits. I would like to keep number of configurations at minimum.
Having ability to conditionally altering just a few variables would help me reducing this code a lot.
Another addition in GitLab 13.7 are the rules:variables. This allows some logic in setting vars:
job:
variables:
DEPLOY_VARIABLE: "default-deploy"
rules:
- if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH
variables: # Override DEPLOY_VARIABLE defined
DEPLOY_VARIABLE: "deploy-production" # at the job level.
- if: $CI_COMMIT_REF_NAME =~ /feature/
variables:
IS_A_FEATURE: "true" # Define a new variable.
script:
- echo "Run script with $DEPLOY_VARIABLE as an argument"
- echo "Run another script if $IS_A_FEATURE exists"
Unfortunatelly YAML-anchors or GitLab-CI's extends don't seem to allow to combine things in script array of commands as of today.
I use built-in variable CI_COMMIT_REF_NAME in combination with global or job-only before_script to solve similar tasks without repeating myself.
Here is an example of my workaround on how to set environment variable to different values dynamically for PROD and DEV during delivery or deployment:
.provide ssh private deploy key: &provide_ssh_private_deploy_key
before_script:
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
- |
if [ "$CI_COMMIT_REF_NAME" == "master" ]; then
echo "$SSH_PRIVATE_DEPLOY_KEY_PROD" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are in master (PROD)"
else
echo "$SSH_PRIVATE_DEPLOY_KEY_DEV" > ~/.ssh/id_rsa
MY_DYNAMIC_VAR="we are NOT in master (DEV)"
fi
- chmod 600 ~/.ssh/id_rsa
deliver-via-ssh:
stage: deliver
<<: *provide_ssh_private_deploy_key
script:
- echo Stage is deliver
- echo $MY_DYNAMIC_VAR
- ssh ...
Also consider this workaround for concatenation of "script" commands: https://stackoverflow.com/a/57209078/470108
hopefully it helps.
A nice way to prepare variables for other jobs is the dotenv report artifact. Unfortunately, these variables cannot be used in many places, but if you only need to access them from other jobs scripts, this is the way:
# prepare environment variables for other jobs
env:
stage: .pre
script:
# Set application version from GIT tag or fake it
- echo "APPVERSION=${CI_COMMIT_TAG:-0.1-dev-$CI_COMMIT_REF_SLUG}+$CI_COMMIT_SHORT_SHA" | tee -a .env
artifacts:
reports:
dotenv: .env
In the script of this job you can conditionally prepare and write your environment values into a file, then make a dotenv artifact out of it. Subsequent - or better yet dependent - jobs will pick up the variables for their scripts from there.

How do we use the 'variables' keyword in gitlab-ci.yml?

I am trying to make use of the variables: keyword documented in the Gitlab CI Documentation here:
FROM: https://docs.gitlab.com/ce/ci/yaml/README.html
variables
This feature requires gitlab-runner with version equal or greater than
0.5.0.
GitLab CI allows you to add to .gitlab-ci.yml variables that are set
in build environment. The variables are stored in repository and are
meant to store non-sensitive project configuration, ie. RAILS_ENV or
DATABASE_URL.
variables:
DATABASE_URL: "postgres://postgres#postgres/my_database"
These variables can be later used in all executed commands and
scripts.
The YAML-defined variables are also set to all created service
containers, thus allowing to fine tune them.
When I attempt to use it, my builds do not run any stages and are marked successful anyway, a good sign of bad YAML. I pasted my gitlab-ci.yml contents into the LINT tool in the settings area and the output error is:
Status: syntax is incorrect
Error: variables job: unknown parameter PACKAGE_NAME
I'm using my YAML syntax the same as the docs, however it will not work. I'm unable to find any open bugs related to this. Below are my current versions and a sanitized version of my gitlab-ci.yml.
Gitlab Version: 7.13.2 Omnibus
Gitlab Runner Version: 0.5.2
gitlab-ci.yml (Sanitized)
types:
- test
- build
variables:
PACKAGE_NAME: "awesome-django-app"
PACKAGE_SUMMARY: "Awesome webapp backend."
MAJOR_RELEASE: "1"
MINOR_RELEASE: "0"
PATCH_LEVEL: "0dev"
DEV_DB_URL: "db"
DEV_SERVER: "pydev.example.com"
PROD_SERVER: "pyprod.example.com"
TEST_SERVER: "pytest.example.com"
envtest:
type: test
script:
- ". ./testbuild.sh"
tags:
- python2.7
- postgres
- linux
except:
- tags
buildrpm:
type: build
script:
- mkdir -p ~/rpmbuild/SOURCES
- mkdir -p ~/rpmbuild/SPECS
- mkdir -p ~/tarbuild/$PACKAGE_NAME-$MAJOR_RELEASE.$MINOR_RELEASE.$PATCH_LEVEL
- cp $PACKAGE_NAME.spec ~/rpmbuild/SPECS/.
- cp -r * ~/tarbuild/$PACKAGE_NAME-$MAJOR_RELEASE.$MINOR_RELEASE.$PATCH_LEVEL/.
- cd ~/tarbuild
- tar -zcf ~/rpmbuild/SOURCES/$PACKAGE_NAME-$MAJOR_RELEASE.$MINOR_RELEASE.$PATCH_LEVEL.tar.gz *
- cd ~
- rm -Rf ~/tarbuild
- rpmlint -i ~/rpmbuild/SPECS/$PACKAGE_NAME.spec
- echo $CI_BUILD_ID
- 'rpmbuild -ba ~/rpmbuild/SPECS/$PACKAGE_NAME.spec \
--define="_build_number $CI_BUILD_ID" \
--define="_python_version_min 2.7" \
--define="_version $MAJOR_RELEASE.$MINOR_RELEASE.$PATCH_LEVEL" \
--define="_package_name $PACKAGE_NAME" \
--define="_summary $SUMMARY"'
- scp rpmbuild/RPMS/noarch/$PACKAGE_NAME-$MAJOR_RELEASE.$MINOR_RELEASE.$PATCH_LEVEL-$CI_BUILD_ID.noarch.rpm $DEV_SERVER:~/.
tags:
- python2.7
- postgres
- linux
- rpm
except:
- tags
Question:
How do I use this value properly?
Additional Info:
Removing this section from the YAML file causes everything to work so the rest of the file is in working order. (Of course undefined variables lead to script errors...)
Even just reducing the variables for testing down to just PACKAGE_NAME causes the same break.
The original answer is no longer correct.
The original documentation now stands, Now there are more ways as well. Variables can be created from the GUI, API, or by being defined in the .gitlab-ci.yml as well.
https://docs.gitlab.com/ce/ci/variables/README.html
While it is in the documentation, I do not believe that variables were included in the latest version of gitlab (7.13). The functionality to read variables out of the yaml files was brought in by a commit by ayufan 9 days ago.
Looking at the parser on the 7.13 stable branch, you can see that his contribution did not make it in. So assuming you're on 7.13 or earlier, I'm afraid we are out of luck. Since it is on master, I am fairly certain that we'll see it in the next release. Until then, we could either monkey patch, do a git pull if you're using the source directly, or just rely on the project variables until the next release.