I am trying to build a monorepo with two independent build pipelines. Ideally, they should only run IF files in the projects have changed. Unfortunately, the both get triggered always and I do not understand why.
My idea is basically:
run a pipeline: if we are in a merge request AND files relevant to that pipeline have changed
run a pipeline if we are on the develop or release branch because we want the artifacts
my .gitlab-ci.yml:
projectA:
trigger:
include: gitlab/ci-projectA.yml
strategy: depend # fail if the projectAbuild fails
rules:
# we want to run this pipeline IF it's a merge request event AND something changed in the relevant filer OR when its on develop OR release branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
changes:
paths:
- projectA.sln
- projectA/**/*
- gitlab/ci-projectA.yml
- .gitlab-ci.yml
when: on_success
- if: $CI_COMMIT_REF_NAME == "release"
when: on_success
- if: $CI_COMMIT_REF_NAME == "develop"
when: on_success
- when: never
project:
trigger:
include: gitlab/ci-projectB.yml
strategy: depend # fail if the projectBbuild fails
rules:
# we want to run this pipeline IF it's a merge request event AND something changed in the relevant filer OR when its on develop OR release branch
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
changes:
paths:
- projectB.sln
- projectB/**/*
- gitlab/ci-projectB.yml
- .gitlab-ci.yml
when: on_success
- if: $CI_COMMIT_REF_NAME == "release"
when: on_success
- if: $CI_COMMIT_REF_NAME == "develop"
when: on_success
- when: never
Related
my Gitlab pipeline consists of several stages. The second to last stage tests and the last stage does cleanup.
The last stage must always be executed even if the tests fail. The pipeline starts when a merge request is created and the target is the main branch. The pipeline has to be successful otherwise the developer is not able to merge.
Unfortunately, the pipeline aborts and does not run the last stage, if the tests fail. "allow_failure" is not an option. If the test stage fails, the pipeline is succcessful, so that the developer is able to merge the branch. Can anyone give me a tip on this?
#other stages
...
myTests:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
script:
- ...# Do some tests
tags:
- bla
myCleaner:
stage: clean
script:
- ... //delete everything
tags:
- bla
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
- when: always
I'd merge the test and cleanup jobs this way (untested):
# other stages
...
myTests:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
script:
- ./run_tests || retcode=$? # Do some tests
- ... # delete everything
- [ -n "$retcode" ] && false # "false" or "exit 1"
tags:
- bla
When I run following Gitlab CI, it invokes duplicate for both jobs(i.e 4 pipelines). What I need here is to invoke only one job if the condition qualifies
default:
image: 'napp/docker-aws-cli'
variables:
AWS_BUCKET: ******-docker
PM_S3_FOLDER: ********_manager
SNAP_S3_FOLDER: ********_GDAL3_SNAP
********_manager:
inherit:
default: [image]
variables: [PM_S3_FOLDER]
script:
- zip -jrm Dockerfile.zip docker_containers/********_manager/redis/Dockerfile docker_containers/********_manager/redis/buildspec.yaml
- aws s3 cp Dockerfile.zip s3://******-docker/$PM_S3_FOLDER/
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_manager/redis/Dockerfile
- ********/docker_containers/********_manager/redis/buildspec.yaml
allow_failure: true
when: never
- when: on_success
snap:
inherit:
default: [image]
variables: [SNAP_S3_FOLDER]
script:
- zip -jrm Dockerfile.zip docker_containers/********_GDAL3_SNAP/Dockerfile docker_containers/********_GDAL3_SNAP/buildspec.yaml
- aws s3 cp Dockerfile.zip s3://signaleyes-docker/$SNAP_S3_FOLDER/
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_GDAL3_SNAP/Dockerfile
- ********/docker_containers/********_GDAL3_SNAP/buildspec.yaml
allow_failure: true
The issue seems to be with the rules of the inherit job
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_COMMIT_BRANCH == "master"'
changes: # Include the job and set to when:manual if any of the follow paths match a modified file.
- ********/docker_containers/********_manager/redis/Dockerfile
- ********/docker_containers/********_manager/redis/buildspec.yaml
allow_failure: true
when: never
- when: on_success
Quoting from https://docs.gitlab.com/ee/ci/jobs/job_control.html#rules-examples
If you use a when clause as the final rule (not including when: never), two simultaneous pipelines may start. Both push pipelines and merge request pipelines can be triggered by the same event (a push to the source branch for an open merge request).
In order to avoid this rewrite the rules to run the job only in very specific cases, and avoid a final when rule. In your case remove
- when: on_success
Or use workflow to specify which types of pipelines can run
I have a job in a pipeline as such:
msi_build:
stage: Build
script:
- cd project_name
- ls -a
- wine python setup.py bdist_msi
rules:
- if: '$CI_COMMIT_REF_NAME == $BUILD_BRANCH'
- changes:
- /*.{py, html, css, yml, json}
tags:
- pywine
artifacts:
when: on_success
paths:
- /builds/$CI_PROJECT_PATH/project_name/dist/*.msi
expire_in: 1 yrs
But this also runs when I tag a commit and push it in another branch.
BUILD_BRANCH is master. This won't run when only pushing normal commits to other branches. And it'll run when pushing to master. But for some reason unknown to me, it also runs on tagged commits. What is the cause?
If it matters I used: git push --atomic origin <branch name> <tag> when pushing to another branch.
rules:
- if: '$CI_COMMIT_REF_NAME == $BUILD_BRANCH'
- changes:
- /*.{py, html, css, yml, json}
is not
rules:
- if: '$CI_COMMIT_REF_NAME == $BUILD_BRANCH'
changes:
- /*.{py, html, css, yml, json}
The first one runs when $CI_COMMIT_REF_NAME == $BUILD_BRANCH OR when the files were changed. The second one runs the job when that and that is equal AND the files with extensions were changed. Most probably in the commit the tag is run for the files with those extensions were changed, so the command is run, ignoring the first condition, because it's "or".
It's a good idea to add when: on_success to rules: explicitly, so it's nicely visible.
I'm a bit confused with the Gitlab I syntax.
I have some jobs that I don't want to trigger when I simply push tags like
git tag -a 1.0.0 -m "Added version 1.0.0"
git push --tags
The following works correctly, as this job is not being triggered for above scenario
build:
stage: build
script:
- dotnet restore --no-cache --force
- dotnet build --configuration Release --no-restore
artifacts:
paths:
- test
expire_in: 8 hour
rules:
- if: $CI_COMMIT_TAG
when: never
- when: always
Now I also have jobs with a rule to run conditionally depending on whether the commit branch is stable (i.e: main) or not.
Like this one
package_beta:
stage: publish
variables:
PACKAGE_UNSTABLE_SUFFIX: beta
before_script:
- mkdir $PACKAGE_OUTPUT_DIR
script:
- dotnet pack *.sln --configuration Release --output $PACKAGE_OUTPUT_DIR --version-suffix $PACKAGE_UNSTABLE_SUFFIX --include-source --include-symbols
rules:
- if: $CI_COMMIT_BRANCH != "main"
artifacts:
paths:
- $PACKAGE_OUTPUT_DIR/
expire_in: 8 hour
The above also works well. The problem is when I want to have several conditions in rules, like: I want my job to run only when branch is not main AND ALSO when the push is not a tag commit. How to do that with rules?
Here is the problem.
This does not work. It will actually run the job even when the branch is main.
rules:
- if: $CI_COMMIT_BRANCH != "main"
- if: $CI_COMMIT_TAG
when: never
This does not work either
rules:
- if: $CI_COMMIT_BRANCH != "main"
- if: $CI_COMMIT_TAG
when: never
- when: always
I have a pages job that I want to run manual on branches, but automatically triggered on master:
pages:
stage: deploy
cache:
paths:
- public
script:
- scripts/pages.sh
artifacts:
paths:
- public
expire_in: 2 days
So I want a combination of:
only:
- master
when: always
only:
- branches
except:
- master
when: manual
Is that possible?
This should be possible to do if you use GitLab CI rules. This is an example where the shell is powershell and it shows the current time and branch/tag name:
pages:
script:
- mkdir public
- date > public\index.html
- $CI_COMMIT_REF_NAME >> public\index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- if: '$CI_COMMIT_BRANCH == null'
when: never
- when: manual
GitLab matches each individual rule from top to bottom. If the branch is named 'master', the job gets marked with when: always. If the branch name is null, this is a tag, and the job is marked with never. If this is not a branch named master, nor a tag, this is a normal branch, and the job is marked with manual.
As Aleksey Tsalolikhin described, you can remove this rule:
- if: '$CI_COMMIT_BRANCH == null'
when: never
You will then get the option to run the pipeline for your tags as well, like this:
If this is what you want or not, that is up to you.
I've tweaked the answer by MrBerta -- the third command was missing the echo command.
I also changed the slashes from backslashes to regular forward slashes so I can use the Linux shell rather than Powershell.
It now works.
Here is the gitlab-ci.yml file -- with credit to MrBerta.
pages:
script:
- mkdir public
- date > public/index.html
- echo $CI_COMMIT_REF_NAME >> public/index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- if: '$CI_COMMIT_BRANCH == null'
when: never
- when: manual
I tried pushing to master, and my GitLab Pages content was updated as expected; and I tried pushing to a feature branch, and the manual "Play" button came up in the CI/CD pipeline UI.
When I pushed a tag (with detached head, i.e., not on any branch), I could not test it -- GitLab CI did not run a pipeline automatically, and when I tried to Run Pipeline, and picked my tag, GitLab threw an error: "The form contains the following error: No stages / jobs for this pipeline."
So, I would simplify this to:
pages:
script:
- mkdir public
- date > public/index.html
- echo $CI_COMMIT_REF_NAME >> public/index.html
artifacts:
paths:
- public
rules:
- if: '$CI_COMMIT_BRANCH == "master"'
when: always
- when: manual
This pages job runs manually on branches (and tags but I couldn't test it), but automatically triggered on master, as the original poster requested.
You will need to define two stages. you can either copy/paste or use anchors:
.deploy_stage: &deploy_stage
stage: deploy
cache:
paths:
- public
script:
- scripts/pages.sh
artifacts:
paths:
- public
expire_in: 2 days
deploy_manual:
<<: *deploy_stage
only:
- branches
when: manual
deploy_master:
<<: *deploy_stage
only:
- master