I have 3 job groups. In the current setup, all jobs for a given stage are started concurrently.
I would like them run in the following order based on a $jobs variable.
1. prep (always first to run)
2. build1 (if build1 one is present in $jobs string variable)
3. deploy1 (ran after build1)
4. build2 (if build2 present in $jobs)
5. deploy2 (ran after build2)
6. build3 (if build3 present)
7. deploy3 (ran after build3)
If build1 is not present then skip build1 and start build2. If build2 is not present then start with build1, skipping build2 (missing job) and then start build3.
stages:
- prep
- build
- deploy
getjobs
stage: prep
script: <script assigns string value with jobs to be ran to $jobs global variable>
rules:
-if: $CI_COMMIT_MESSAGE =~ /See merge request when: on_success
build1
stage: build
needs: [getjobs]
script: ...
rules:
-if: $CI_COMMIT_MESSAGE =~ /See merge request when: on_success
deploy1
stage: deploy
needs: [build1]
script: ...
rules:
-if: $CI_COMMIT_MESSAGE =~ /See merge request && when: on_success
build2
stage: build
needs: [getjobs]
script: ...
rules:
-if: $CI_COMMIT_MESSAGE =~ /See merge request when: on_success
deploy2
stage: deploy
needs: [build2]
script: ...
rules:
-if: $CI_COMMIT_MESSAGE =~ /See merge request && when: on_success
build3
stage: build
needs: [getjobs]
script: ...
rules:
-if: $CI_COMMIT_MESSAGE =~ /See merge request when: on_success
deploy3
stage: deploy
needs: [build3]
script: ...
rules:
-if: $CI_COMMIT_MESSAGE =~ /See merge request && when: on_success
I'm wondering if it's actually possible to set that kind of job order dependency keeping stages as they currently are.
You can use only to control when to add jobs to pipelines
build1:
script: ...
only:
variables:
- $jobs =~ "build1"
Related
I am trying to build a monorepo with two independent build pipelines. Ideally, they should only run IF files in the projects have changed. Unfortunately, the both get triggered always and I do not understand why.
My idea is basically:
run a pipeline: if we are in a merge request AND files relevant to that pipeline have changed
run a pipeline if we are on the develop or release branch because we want the artifacts
my .gitlab-ci.yml:
projectA:
trigger:
include: gitlab/ci-projectA.yml
strategy: depend # fail if the projectAbuild fails
rules:
# we want to run this pipeline IF it's a merge request event AND something changed in the relevant filer OR when its on develop OR release branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
changes:
paths:
- projectA.sln
- projectA/**/*
- gitlab/ci-projectA.yml
- .gitlab-ci.yml
when: on_success
- if: $CI_COMMIT_REF_NAME == "release"
when: on_success
- if: $CI_COMMIT_REF_NAME == "develop"
when: on_success
- when: never
project:
trigger:
include: gitlab/ci-projectB.yml
strategy: depend # fail if the projectBbuild fails
rules:
# we want to run this pipeline IF it's a merge request event AND something changed in the relevant filer OR when its on develop OR release branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
changes:
paths:
- projectB.sln
- projectB/**/*
- gitlab/ci-projectB.yml
- .gitlab-ci.yml
when: on_success
- if: $CI_COMMIT_REF_NAME == "release"
when: on_success
- if: $CI_COMMIT_REF_NAME == "develop"
when: on_success
- when: never
I have template that looks like this:
/templates/.copy-echo.yml:
workflow:
rules:
- if: '$CI_COMMIT_REF_NAME == "master"'
variables:
FILE_PATH: /test_conf_1.txt
DESTINATION_HOST: somehost
stages:
- copy
- echo
copy step 1/2:
rules:
- changes:
- ${FILE_PATH}
stage: copy
script: |
echo "Add copy here!"
copy step 2/2:
rules:
- changes:
- ${FILE_PATH}
stage: echo
script: |
printenv
echo ${DESTINATION_HOST}
Now in my .gitlab-ci.yml:
include: '/templates/copy-echo.yml'
variables:
FILE_PATH: /test_conf_1.txt
DESTINATION_HOST: somehost2
Now what I want is conditionally assign a value to DESTINATION_HOST variable depending on which file got changed in merged request.
For e.g. if the merge request had updates to file test_conf_2.txt then the value for DESTINATION_HOST should be somehost2 and if the merge request had updates to file test_conf_3.txt then the value for DESTINATION_HOST should be somehost3.
Is it possible to achieve this?
You can use rules:variables: https://docs.gitlab.com/ee/ci/yaml/#rulesvariables
Example:
my job:
script: printenv
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- test_conf_2.txt
variables:
DESTINATION_HOST: "somehost2"
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- test_conf_3.txt
variables:
DESTINATION_HOST: "somehost3"
But it doesn't work if you launch pipeline manually:
You should use rules: changes only with branch pipelines or merge
request pipelines. You can use rules: changes with other pipeline
types, but rules: changes always evaluates to true when there is no
Git push event. Tag pipelines, scheduled pipelines, manual pipelines,
and so on do not have a Git push event associated with them. A rules:
changes job is always added to those pipelines if there is no if that
limits the job to branch or merge request pipelines.
my Gitlab pipeline consists of several stages. The second to last stage tests and the last stage does cleanup.
The last stage must always be executed even if the tests fail. The pipeline starts when a merge request is created and the target is the main branch. The pipeline has to be successful otherwise the developer is not able to merge.
Unfortunately, the pipeline aborts and does not run the last stage, if the tests fail. "allow_failure" is not an option. If the test stage fails, the pipeline is succcessful, so that the developer is able to merge the branch. Can anyone give me a tip on this?
#other stages
...
myTests:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
script:
- ...# Do some tests
tags:
- bla
myCleaner:
stage: clean
script:
- ... //delete everything
tags:
- bla
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
- when: always
I'd merge the test and cleanup jobs this way (untested):
# other stages
...
myTests:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
script:
- ./run_tests || retcode=$? # Do some tests
- ... # delete everything
- [ -n "$retcode" ] && false # "false" or "exit 1"
tags:
- bla
When I run following Gitlab CI, it invokes duplicate for both jobs(i.e 4 pipelines). What I need here is to invoke only one job if the condition qualifies
default:
image: 'napp/docker-aws-cli'
variables:
AWS_BUCKET: ******-docker
PM_S3_FOLDER: ********_manager
SNAP_S3_FOLDER: ********_GDAL3_SNAP
********_manager:
inherit:
default: [image]
variables: [PM_S3_FOLDER]
script:
- zip -jrm Dockerfile.zip docker_containers/********_manager/redis/Dockerfile docker_containers/********_manager/redis/buildspec.yaml
- aws s3 cp Dockerfile.zip s3://******-docker/$PM_S3_FOLDER/
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_manager/redis/Dockerfile
- ********/docker_containers/********_manager/redis/buildspec.yaml
allow_failure: true
when: never
- when: on_success
snap:
inherit:
default: [image]
variables: [SNAP_S3_FOLDER]
script:
- zip -jrm Dockerfile.zip docker_containers/********_GDAL3_SNAP/Dockerfile docker_containers/********_GDAL3_SNAP/buildspec.yaml
- aws s3 cp Dockerfile.zip s3://signaleyes-docker/$SNAP_S3_FOLDER/
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_GDAL3_SNAP/Dockerfile
- ********/docker_containers/********_GDAL3_SNAP/buildspec.yaml
allow_failure: true
The issue seems to be with the rules of the inherit job
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_manager/redis/Dockerfile
- ********/docker_containers/********_manager/redis/buildspec.yaml
allow_failure: true
when: never
- when: on_success
Quoting from https://docs.gitlab.com/ee/ci/jobs/job_control.html#rules-examples
If you use a when clause as the final rule (not including when: never), two simultaneous pipelines may start. Both push pipelines and merge request pipelines can be triggered by the same event (a push to the source branch for an open merge request).
In order to avoid this rewrite the rules to run the job only in very specific cases, and avoid a final when rule. In your case remove
- when: on_success
Or use workflow to specify which types of pipelines can run
Is there any way to switch a job to manual mode (and vice-versa) dynamically?
So, instead of having when: manual in gitlab-ci.yml file, it would be switched to manual (or the opposite) according to any dynamic checking.
Example: Lacking of configuration from environment variables that prevents it to run, but the user could run it later if they set the variables at the launch time.
Just playing with the current syntax, it could be like:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
when: manual && [[ ! $my_var ]] # Using shell syntax just as an example of the condition
Instead of failing:
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- [[ ! $my_var ]] && echo "my_var is undefined" && exit 1
- echo "Deploying $my_var..."
You can try using rules to set a condition based on a variable you set in the script of the job.
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- export my_var
rules:
- if: $my_var
when: manual
https://docs.gitlab.com/ee/ci/yaml/#rules-attributes
You can make use of rules if with below example
myjob:
stage: deploy
environment: proj-shared-env-qa
script:
- echo "Deploying $my_var..."
rules:
- if: $my_var
when: on_success
- if: !$my_var
when: manual
So if var is set then it will automatically run. if it is not set then it'll become a manual process
Ref: Gitlab Rules