I have a gitlab-ci.yml that creates and trigger a child .yml
stages:
- child-pipeline-generator
- child-pipeline-trigger
generate-child-pipeline:
stage: child-pipeline-generator
tags:
- GroupRunner
script:
- $(./generate-build.ps1) *>&1 > child-pipeline-gitlab-ci.yml
- (Get-Content child-pipeline-gitlab-ci.yml) | Set-Content child-pipeline-gitlab-ci.yml -Encoding UTF8
artifacts:
paths:
- child-pipeline-gitlab-ci.yml
trigger-child-pipeline:
stage: child-pipeline-trigger
trigger:
include:
- artifact: child-pipeline-gitlab-ci.yml
job: generate-child-pipeline
strategy: depend
The resulting yml looks like
build_1:
tags:
- GroupRunner
script:
- echo 'build_1'
build_2:
tags:
- GroupRunner
script:
- echo 'build_2'
But when executed only job 1 (build_1) shows up in the Downstream list
Turned out the problem was the encoding of the powershell-output. Default encoding from powershell 5 is UFT16BOM and my reencoding to UTF8 resulted in UFT8BOM wich gitlab can´t handle properly. My sollution was to encode in ASCII.
What I can´t explain is why it was able to interpret the first job correctly, I thought that encodeing would result in an all or nothing outcome. Maybe the CRLF-CRLF after the first job caused the errror
Related
(How) Can I use variable value as a name for another variable?
I have a job with matrix and dotenv artifacts as follows:
build-names:
stage: build
...
script:
...
<lines omitted>
...
- echo "${NAME}_DEB_PACKAGE_VERSION=${DEB_PACKAGE_VERSION}" >> build.env
artifacts:
reports:
dotenv: build.env
parallel:
matrix:
- NAME:
- name1
- name2
type:
- 0
- 1
environment: $NAME/$TYPE
Then I have a downstream trigger job again using matrix and I want to pass the appropriate package version based on the ${NAME}
build-images:
stage: .post
needs:
- job: build-names
artifacts: true
variables:
PACKAGE_VERSION_VARIABLE_NAME: ${NAME}_DEB_PACKAGE_VERSION
PACKAGE_VERSION: ${${BACKEND_VERSION_VARIABLE_NAME}}
// OR
PACKAGE_VERSION_VARIABLE_NAME: ${NAME}_DEB_PACKAGE_VERSION
PACKAGE_VERSION: ${!BACKEND_VERSION_VARIABLE_NAME}
trigger:
project: group/project-${NAME}
parallel:
matrix:
- NAME:
- name1
- name2
Neither of the two approaches above (double ${${}} or !) works in the variables section.
I could 'generate' the variable within script section, but AFAIK you cannot have both trigger and script within the same job.
Is there a workaround for similar use cases?
(Using self-hosted gitlab 15.4)
I need to call a bash script from my gitlab cicd pipeline. When I call it, the input parameter needs to change depending on whether or not this is a merge into master. Basically what I want is this:
if master:
test:
stage: test
script:
- INPUT="foo"
- $(myscript.sh $INPUT)
if NOT master:
test:
stage: test
script:
- INPUT=""
- $(myscript.sh $INPUT)
I'm trying to figure out a way to set INPUT depending on which branch the pipeline is running on. I know there are rules as well as only/except, but they don't seem to allow you to set variable, only test them. I know the brute force way is to just write this twice, one with "only master" and another with "except master", but I would really like to avoid that.
Thanks
I implement this kind of thing using yml anchors to extend tasks.
I find it easier to read and customization can include other things, not only variables.
For example:
.test_my_software:
stage: test
script:
- echo ${MESSAGE}
- bash test.sh
test stable:
<<: *test_my_software
variables:
MESSAGE: testing stable code!
only:
- /stable.*/
test:
<<: *test_my_software
variables:
MESSAGE: testing our code!
except:
- /stable.*/
you get the idea...
Why not have two jobs to run the script and use rules to control when they are ran against master:
test:
stage: test
script:
- if [$CI_COMMIT_REF_NAME == 'master']; then export INPUT="foo"; else INPUT=""; fi
- $(myscript.sh $INPUT)
I am building a YAML pipeline in Azure DevOps. I'm generating couple of variables in first job, which I'm passing to other job and want to use them to conditionally run some steps.
Here's how the code looks like:
stages:
- stage: ConditionalVars
jobs:
- job: outputVars
steps:
- pwsh: |
Write-Host "##vso[task.setvariable variable=varExample;isOutput=true]testVar"
name: setVars
- job: useVars
dependsOn: outputVars
variables:
varFromJob: $[ dependencies.outputVars.outputs['setVars.varExample']]
steps:
- ${{ if eq($(varFromJob), 'testVar')}}:
- pwsh: |
Write-Output "Conditionally run"
I am able to print $(varFromJob) variable in step in job useVars. But I'm not able to use it in this if (condition) example.
I was trying different approaches like variables.varFromJob and other taken from https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml.
Have you guys been struggling with similar issue? How could I define it to make it work?
Thanks a lot!
Rafal
Try to use a condition for the step:
steps:
- pwsh: |
Write-Output "Conditionally run"
condition: eq(variables['varFromJob'], 'testVar')
${{ if eq... }} is a compile-time expression that is evaluated when the YAML file is compiled.
I want to have a CI to deploy two commands ("bash X" and "bash Y") on different production servers (server 1, server 2, server 3, etc.).
I looked for multiple stages but it don't seems to answer my question.
I don't really care if it runs in parallel or B after A. (the manual section is for debugging)
I don't know how to do it : I tried with multiple extends but it only takes the last one (bashB) in my pipeline.
stages:
- get_password
- bashA
- bashB
get_password:
stage: get_password
# Steps
.bashA:
stage: bashA
script:
- lorem ipsum
when: manual
only:
changes:
- script/bashA.sh
.bashB:
stage: bashB
script:
- ipsum loreem
when: manual
only:
changes:
- script/bashB.sh
# SRV1
deploy-srv1:
extends:
- .bashA
- .bashB
variables:
SRV_1: urlsrv1
# SRV2
deploy-srv2:
extends:
- .bashA
- .bashB
variables:
SRV_1: urlsrv2
I just want to be able to deploy bashA and bash B on X servers (I just took 2 servers for example).
When using multiple extend in GitLab, some of the values will not be merged, but overwritten. If you check the documentation here:
https://docs.gitlab.com/ee/ci/yaml/#extends
They write:
The algorithm used for merge is “closest scope wins”, so keys from the last member will always shadow anything defined on other levels
You are not alone in wanting a feature to be able to merge scripts instead of overwriting them. Here's an open issue on GitLab to do what you described:
https://gitlab.com/gitlab-org/gitlab/issues/16376
In the meantime, and only looking at the example you provided, you can get something like what you want by manually merging bashA and bashB into one job:
stages:
- get_password
- bash
get_password:
stage: get_password
# Steps
.bash_both:
stage: bash
script:
- lorem ipsum
- ipsum loreem
when: manual
only:
changes:
- script/bashA.sh
- script/bashB.sh
# SRV1
deploy-srv1:
extends:
- .bash_both
variables:
SRV_1: urlsrv1
# SRV2
deploy-srv2:
extends:
- .bash_both
variables:
SRV_1: urlsrv2
I'm fairly new to using gitlab-ci and as such, I've run into a problem where the following fails ci-lint because of my use of anchors/references:
image: docker:latest
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay2
DOCKER_HOST: tcp://localhost:2375
.install_thing1: &install_thing1
- do things
- to install
- thing1
.install_thing2: &install_thing2
- do things to
- install thing2
.setup_thing1: &setup_things1
variables:
VAR: var
FOO: bar
script:
- all
- the
- things
before_script:
...
stages:
- deploy-test
- deploy-stage
- deploy-prod
test:
stage: deploy-test
variables:
RUN_ENV: "test"
...
only:
- tags
- branches
script:
- *install_thing1
- *install_thing2
- *setup_thing1
- other stuff
...
test:
stage: deploy-stage
variables:
RUN_ENV: "stage"
...
only:
- master
script:
- *install_thing1
- *install_thing2
- *setup_thing1
- other stuff
When I attempt to lint the gitlab-ci.yml, I get the following error:
Status: syntax is incorrect
Error: jobs:test:script config should be a string or an array of strings
The error eludes to just needing an array for the script piece, which I believe I have. Use of the <<: *anchor pragma causes an error as well.
So, how can one accomplish what I'm trying to do here where I don't have to repeat the code in every -block?
You can fix it and even make it more DRY, take a look at the Auto DevOps template Gitlab created.
It can fix your issue and even more improve your CI file, just have a template job like their auto_devops job, include it in a before_script and then you can combine and call multiple functions in a script block.
The anchors only give you limited flexibility.
(This concept made it possible for me to have one CI file for 20+ projects and a centralized functions file I wget and load in my before_script.)