When I run following Gitlab CI, it invokes duplicate for both jobs(i.e 4 pipelines). What I need here is to invoke only one job if the condition qualifies
default:
image: 'napp/docker-aws-cli'
variables:
AWS_BUCKET: ******-docker
PM_S3_FOLDER: ********_manager
SNAP_S3_FOLDER: ********_GDAL3_SNAP
********_manager:
inherit:
default: [image]
variables: [PM_S3_FOLDER]
script:
- zip -jrm Dockerfile.zip docker_containers/********_manager/redis/Dockerfile docker_containers/********_manager/redis/buildspec.yaml
- aws s3 cp Dockerfile.zip s3://******-docker/$PM_S3_FOLDER/
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_manager/redis/Dockerfile
- ********/docker_containers/********_manager/redis/buildspec.yaml
allow_failure: true
when: never
- when: on_success
snap:
inherit:
default: [image]
variables: [SNAP_S3_FOLDER]
script:
- zip -jrm Dockerfile.zip docker_containers/********_GDAL3_SNAP/Dockerfile docker_containers/********_GDAL3_SNAP/buildspec.yaml
- aws s3 cp Dockerfile.zip s3://signaleyes-docker/$SNAP_S3_FOLDER/
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_GDAL3_SNAP/Dockerfile
- ********/docker_containers/********_GDAL3_SNAP/buildspec.yaml
allow_failure: true
The issue seems to be with the rules of the inherit job
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_manager/redis/Dockerfile
- ********/docker_containers/********_manager/redis/buildspec.yaml
allow_failure: true
when: never
- when: on_success
Quoting from https://docs.gitlab.com/ee/ci/jobs/job_control.html#rules-examples
If you use a when clause as the final rule (not including when: never), two simultaneous pipelines may start. Both push pipelines and merge request pipelines can be triggered by the same event (a push to the source branch for an open merge request).
In order to avoid this rewrite the rules to run the job only in very specific cases, and avoid a final when rule. In your case remove
- when: on_success
Or use workflow to specify which types of pipelines can run
Related
I am trying to build a monorepo with two independent build pipelines. Ideally, they should only run IF files in the projects have changed. Unfortunately, the both get triggered always and I do not understand why.
My idea is basically:
run a pipeline: if we are in a merge request AND files relevant to that pipeline have changed
run a pipeline if we are on the develop or release branch because we want the artifacts
my .gitlab-ci.yml:
projectA:
trigger:
include: gitlab/ci-projectA.yml
strategy: depend # fail if the projectAbuild fails
rules:
# we want to run this pipeline IF it's a merge request event AND something changed in the relevant filer OR when its on develop OR release branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
changes:
paths:
- projectA.sln
- projectA/**/*
- gitlab/ci-projectA.yml
- .gitlab-ci.yml
when: on_success
- if: $CI_COMMIT_REF_NAME == "release"
when: on_success
- if: $CI_COMMIT_REF_NAME == "develop"
when: on_success
- when: never
project:
trigger:
include: gitlab/ci-projectB.yml
strategy: depend # fail if the projectBbuild fails
rules:
# we want to run this pipeline IF it's a merge request event AND something changed in the relevant filer OR when its on develop OR release branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
changes:
paths:
- projectB.sln
- projectB/**/*
- gitlab/ci-projectB.yml
- .gitlab-ci.yml
when: on_success
- if: $CI_COMMIT_REF_NAME == "release"
when: on_success
- if: $CI_COMMIT_REF_NAME == "develop"
when: on_success
- when: never
I have template that looks like this:
/templates/.copy-echo.yml:
workflow:
rules:
- if: '$CI_COMMIT_REF_NAME == "master"'
variables:
FILE_PATH: /test_conf_1.txt
DESTINATION_HOST: somehost
stages:
- copy
- echo
copy step 1/2:
rules:
- changes:
- ${FILE_PATH}
stage: copy
script: |
echo "Add copy here!"
copy step 2/2:
rules:
- changes:
- ${FILE_PATH}
stage: echo
script: |
printenv
echo ${DESTINATION_HOST}
Now in my .gitlab-ci.yml:
include: '/templates/copy-echo.yml'
variables:
FILE_PATH: /test_conf_1.txt
DESTINATION_HOST: somehost2
Now what I want is conditionally assign a value to DESTINATION_HOST variable depending on which file got changed in merged request.
For e.g. if the merge request had updates to file test_conf_2.txt then the value for DESTINATION_HOST should be somehost2 and if the merge request had updates to file test_conf_3.txt then the value for DESTINATION_HOST should be somehost3.
Is it possible to achieve this?
You can use rules:variables: https://docs.gitlab.com/ee/ci/yaml/#rulesvariables
Example:
my job:
script: printenv
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- test_conf_2.txt
variables:
DESTINATION_HOST: "somehost2"
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- test_conf_3.txt
variables:
DESTINATION_HOST: "somehost3"
But it doesn't work if you launch pipeline manually:
You should use rules: changes only with branch pipelines or merge
request pipelines. You can use rules: changes with other pipeline
types, but rules: changes always evaluates to true when there is no
Git push event. Tag pipelines, scheduled pipelines, manual pipelines,
and so on do not have a Git push event associated with them. A rules:
changes job is always added to those pipelines if there is no if that
limits the job to branch or merge request pipelines.
my Gitlab pipeline consists of several stages. The second to last stage tests and the last stage does cleanup.
The last stage must always be executed even if the tests fail. The pipeline starts when a merge request is created and the target is the main branch. The pipeline has to be successful otherwise the developer is not able to merge.
Unfortunately, the pipeline aborts and does not run the last stage, if the tests fail. "allow_failure" is not an option. If the test stage fails, the pipeline is succcessful, so that the developer is able to merge the branch. Can anyone give me a tip on this?
#other stages
...
myTests:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
script:
- ...# Do some tests
tags:
- bla
myCleaner:
stage: clean
script:
- ... //delete everything
tags:
- bla
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
- when: always
I'd merge the test and cleanup jobs this way (untested):
# other stages
...
myTests:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
script:
- ./run_tests || retcode=$? # Do some tests
- ... # delete everything
- [ -n "$retcode" ] && false # "false" or "exit 1"
tags:
- bla
I have many rules in my CI config, and also many anchors.
It's not an offence to the linter to mention rules: multiple times per job, but the same linter does not help with testing whether multiple rules: clauses add up and in which order.
So the trouble is, now I have to repeat the slightly changing set of rules in every job.
.build-rules: &build-rules
rules:
- if: '$DEPLOY_TAG'
when: never
- if: '$CI_COMMIT_REF_NAME == "master"'
- if: '$CI_PIPELINE_SOURCE == "web"'
- if: '$CI_COMMIT_REF_NAME =~ "/^v[0-9]+\.[0-9]+.*$/"'
job_with_changed_rule:
<<: *build-rules
rules:
- if: '$DEPLOY_TAG'
script:
- do something
job_with_another_rule:
<<: *build-rules
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule"'
script:
- do something else
Before Gitlab 14.3 the only thing you could do was placing extends block
But now you can re-use rules from other jobs with !reference tag.
For more details see this docs
I have a pages job that I want to run manual on branches, but automatically triggered on master:
pages:
stage: deploy
cache:
paths:
- public
script:
- scripts/pages.sh
artifacts:
paths:
- public
expire_in: 2 days
So I want a combination of:
only:
- master
when: always
only:
- branches
except:
- master
when: manual
Is that possible?
This should be possible to do if you use GitLab CI rules. This is an example where the shell is powershell and it shows the current time and branch/tag name:
pages:
script:
- mkdir public
- date > public\index.html
- $CI_COMMIT_REF_NAME >> public\index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- if: '$CI_COMMIT_BRANCH == null'
when: never
- when: manual
GitLab matches each individual rule from top to bottom. If the branch is named 'master', the job gets marked with when: always. If the branch name is null, this is a tag, and the job is marked with never. If this is not a branch named master, nor a tag, this is a normal branch, and the job is marked with manual.
As Aleksey Tsalolikhin described, you can remove this rule:
- if: '$CI_COMMIT_BRANCH == null'
when: never
You will then get the option to run the pipeline for your tags as well, like this:
If this is what you want or not, that is up to you.
I've tweaked the answer by MrBerta -- the third command was missing the echo command.
I also changed the slashes from backslashes to regular forward slashes so I can use the Linux shell rather than Powershell.
It now works.
Here is the gitlab-ci.yml file -- with credit to MrBerta.
pages:
script:
- mkdir public
- date > public/index.html
- echo $CI_COMMIT_REF_NAME >> public/index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- if: '$CI_COMMIT_BRANCH == null'
when: never
- when: manual
I tried pushing to master, and my GitLab Pages content was updated as expected; and I tried pushing to a feature branch, and the manual "Play" button came up in the CI/CD pipeline UI.
When I pushed a tag (with detached head, i.e., not on any branch), I could not test it -- GitLab CI did not run a pipeline automatically, and when I tried to Run Pipeline, and picked my tag, GitLab threw an error: "The form contains the following error: No stages / jobs for this pipeline."
So, I would simplify this to:
pages:
script:
- mkdir public
- date > public/index.html
- echo $CI_COMMIT_REF_NAME >> public/index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- when: manual
This pages job runs manually on branches (and tags but I couldn't test it), but automatically triggered on master, as the original poster requested.
You will need to define two stages. you can either copy/paste or use anchors:
.deploy_stage: &deploy_stage
stage: deploy
cache:
paths:
- public
script:
- scripts/pages.sh
artifacts:
paths:
- public
expire_in: 2 days
deploy_manual:
<<: *deploy_stage
only:
- branches
when: manual
deploy_master:
<<: *deploy_stage
only:
- master