Unable to pass variables through gitlab bridge to another pipeline - variables

Here is my simple yaml file
image: my/docker/image
stages:
- print
- testvarbridge
variables:
INCOMING_VAR: $ENV_VAR
print_these:
stage: print
script:
- echo $INCOMING_VAR
- export $INCOMING_VAR
testvarbridge:
stage: testvarbridge
variables:
TEST_VAR: $INCOMING_VAR
trigger:
project: my-project/pipeline-two
branch: ci-cd
the $ENV_VAR is a variable in the project for testing... it just says "this_is_the_variable"
When I trigger the pipleine..the print stage correctly prints:
echo $INCOMING_VAR
this_is_the_variable
But when the second pipeline is triggered, it is just set up to do a simple echo command of the variable that is passed in.. it echo's this:
echo TEST_VAR
$ENV_VAR
As you can see, the when the testvarbridge stage sets up the variable TEST_VAR, it is grabbing the $ENV_VAR variable up top as a literal string. It does not evaluate it and grab the value associated with that variable. Am I missing something?

This was reported 3 years ago in gitlab-org/gitlab-runner issue 1809: "Use variable inside other variable".
A workaround is to set vars in a before_script instead of variables.
So the example given in the issue would work if written as:
before_script:
- export VAR1="${CI_PIPELINE_ID}"
- export VAR2="test-${VAR1}"
Update February 2020: Philippe Charrière adds:
the issue is not closed - the milestone is 13.0 (for May 2020)
Update Sept. 2021,
One last validation is pending and if that's solved then we are shipping this thing in 14.3.
So, soon (14.3, Sept. 2021):
this-job-name-123:
except:
variables:
- $CI_COMMIT_MESSAGE =~ /Skip $CI_JOB_NAME/i
script:
- echo The job runs unless the commit message contains "Skip this-job-name-123"

Related

Azure Devops YAML: Looping Jobs With Different Variables Passed

I am struggling to figure out how to execute an API test using a pipeline where the command used can be modified using a loop. For example:
TEMPLATE.yaml
parameters:
JobName: ''
TestDirectory: '.\tests\smoke\'
PositiveTest: ''
NegativeTest: ''
- name: environments
type: object
values:
- dev01
- dev02
- test01
- test02
jobs:
- job: ${{ parameters.JobName }}
pool:
name: pool
demands:
- Cmd
variables:
PosTest: ${{ parameters.PositiveTest }}
NegTest: ${{ parameters.NegativeTest }}
Directory: ${{ parameters.TestDirectory }}
- script: |
call .\venv\Scripts\activate.bat
cd $(Directory)
python $(PosTest)
displayName: 'Executing Positive Test Scenarios'
condition: and(succeeded(), ne('${{ variables.PosTest }}', ''))
- script: |
call .\venv\Scripts\activate.bat
cd $(Directory)
python $(NegTest)
displayName: 'Executing Negative Test Scenarios'
condition: and(succeeded(), ne('${{ variables.NegTest }}', ''))
TEST_FILE.yaml
...
jobs:
# Get this file: templates\TEMPLATE.yml from the `build` repository (imported above)
- template: templates\api-test-build.yml#build
- ${{ each env in parameters.environments }}:
parameters:
TestDirectory: '.\tests\smoke\job_class'
PositiveTest: 'python_test.py http://apient${{ env }}/arbitrary/api/path/name'
NegativeTest: ''
This of course doesn't work (the each directive returns an error like "the first property must be template". If I move it up a line it then says "the first property must be a job" and this cycle of errors just continues...).
The idea is just that I have a loop that iterates through environment strings (top of the TEMPLATE.yaml example). The yaml file that references the template passes the command python_test.py http://apient<whatever env string the current iteration is on>/arbitrary/api/path/name for each iterated string (bottom of TEST_FILE.yaml) and the template just executes each of those api tests. At the end of a run there should be 4 environments that have been tested on.
This is just an idea I have and I am still learned all the in's and out's of Azure Devops YAML. If anyone knows how I can get this to work, any improvements I can make to the idea itself or any other workarounds/solutions that would be highly appreciated. Thank you!
You can try using Multi-job configuration (matrix) in your pipeline.
When you want to run the same job with multi-configuration in a pipeline, the matrix strategy is a good choose.
For example, in your pipeline, you want to run the jobs that have the same steps and input parameters but different values of the input parameters. You can just set up one job with the matrix strategy in the pipeline.

How to change variable in gitlab cicd pipeline depending on branch

I need to call a bash script from my gitlab cicd pipeline. When I call it, the input parameter needs to change depending on whether or not this is a merge into master. Basically what I want is this:
if master:
test:
stage: test
script:
- INPUT="foo"
- $(myscript.sh $INPUT)
if NOT master:
test:
stage: test
script:
- INPUT=""
- $(myscript.sh $INPUT)
I'm trying to figure out a way to set INPUT depending on which branch the pipeline is running on. I know there are rules as well as only/except, but they don't seem to allow you to set variable, only test them. I know the brute force way is to just write this twice, one with "only master" and another with "except master", but I would really like to avoid that.
Thanks
I implement this kind of thing using yml anchors to extend tasks.
I find it easier to read and customization can include other things, not only variables.
For example:
.test_my_software:
stage: test
script:
- echo ${MESSAGE}
- bash test.sh
test stable:
<<: *test_my_software
variables:
MESSAGE: testing stable code!
only:
- /stable.*/
test:
<<: *test_my_software
variables:
MESSAGE: testing our code!
except:
- /stable.*/
you get the idea...
Why not have two jobs to run the script and use rules to control when they are ran against master:
test:
stage: test
script:
- if [$CI_COMMIT_REF_NAME == 'master']; then export INPUT="foo"; else INPUT=""; fi
- $(myscript.sh $INPUT)

How can I use variables passed between jobs in a condition?

I am building a YAML pipeline in Azure DevOps. I'm generating couple of variables in first job, which I'm passing to other job and want to use them to conditionally run some steps.
Here's how the code looks like:
stages:
- stage: ConditionalVars
jobs:
- job: outputVars
steps:
- pwsh: |
Write-Host "##vso[task.setvariable variable=varExample;isOutput=true]testVar"
name: setVars
- job: useVars
dependsOn: outputVars
variables:
varFromJob: $[ dependencies.outputVars.outputs['setVars.varExample']]
steps:
- ${{ if eq($(varFromJob), 'testVar')}}:
- pwsh: |
Write-Output "Conditionally run"
I am able to print $(varFromJob) variable in step in job useVars. But I'm not able to use it in this if (condition) example.
I was trying different approaches like variables.varFromJob and other taken from https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml.
Have you guys been struggling with similar issue? How could I define it to make it work?
Thanks a lot!
Rafal
Try to use a condition for the step:
steps:
- pwsh: |
Write-Output "Conditionally run"
condition: eq(variables['varFromJob'], 'testVar')
${{ if eq... }} is a compile-time expression that is evaluated when the YAML file is compiled.

GitLab-CI: run job only when a branch is made

In gitlab runner, I want to run job by separating branch creation and branch update.
(.yml example)
creat job:
stage: create
script:
- echo 'branch checkout'
only:
- ????
update job:
stage: deply
script:
- echo 'branch update'
only:
- branches
You might be able to check CI_COMMIT_BEFORE_SHA. If it is all zeros, then there is no previous commit for that ref. (It is also all zeros for a merge request.)
See predefined variables.

Way to use anchors/references in job.<spec>.script for DRYness

I'm fairly new to using gitlab-ci and as such, I've run into a problem where the following fails ci-lint because of my use of anchors/references:
image: docker:latest
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay2
DOCKER_HOST: tcp://localhost:2375
.install_thing1: &install_thing1
- do things
- to install
- thing1
.install_thing2: &install_thing2
- do things to
- install thing2
.setup_thing1: &setup_things1
variables:
VAR: var
FOO: bar
script:
- all
- the
- things
before_script:
...
stages:
- deploy-test
- deploy-stage
- deploy-prod
test:
stage: deploy-test
variables:
RUN_ENV: "test"
...
only:
- tags
- branches
script:
- *install_thing1
- *install_thing2
- *setup_thing1
- other stuff
...
test:
stage: deploy-stage
variables:
RUN_ENV: "stage"
...
only:
- master
script:
- *install_thing1
- *install_thing2
- *setup_thing1
- other stuff
When I attempt to lint the gitlab-ci.yml, I get the following error:
Status: syntax is incorrect
Error: jobs:test:script config should be a string or an array of strings
The error eludes to just needing an array for the script piece, which I believe I have. Use of the <<: *anchor pragma causes an error as well.
So, how can one accomplish what I'm trying to do here where I don't have to repeat the code in every -block?
You can fix it and even make it more DRY, take a look at the Auto DevOps template Gitlab created.
It can fix your issue and even more improve your CI file, just have a template job like their auto_devops job, include it in a before_script and then you can combine and call multiple functions in a script block.
The anchors only give you limited flexibility.
(This concept made it possible for me to have one CI file for 20+ projects and a centralized functions file I wget and load in my before_script.)