I have a CI configuration for merge requests like this:
# build artifacts and run tests
build-and-test:
stage: build
script:
- mvn $MAVEN_CLI_OPTS clean verify
only:
- master
- merge_requests
When I push to a MR, gitlab checks out a specific commit, instead of the branch being used in the MR. This makes my gitver configuration ignore the branch name.
Can I make Gitlab fetch the branch instead of the commit? Every MR has a specific branch, right?
MRs are actually built in a detached commit, it seems like it's an implementation detail of gitlab. Maybe it's made to protect from accidental pushes?
If you have git on you machine or build container, you can manually checkout to $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME variable or to use it somehow in your gitver.
$CI_COMMIT_REF_NAME also seems to be set to source branch name in my tests, it might be a bit more convenient in case of having single job for both master and MR branches.
Edit: example pipeline for merge checks only:
build-and-test:
stage: build
script:
- git checkout $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
- mvn $MAVEN_CLI_OPTS clean verify
only:
- merge_requests
Related
I have a CICD configuration that looks something like this:
.rule_template: &rule_configuration
rules:
- changes:
- file/dev/script1.txt
variables:
DESTINATION_HOST: somehost1
RUNNER_TAG: somerunner1
- changes:
- file/test/script1.txt
variables:
DESTINATION_HOST: somehost2
RUNNER_TAG: somerunner2
default:
tags:
- scripts
stages:
- lint
deploy scripts 1/6:
<<: *rule_configuration
tags:
- $RUNNER_TAG
stage: lint
script: |
echo "Add linting here!"
....
In short, which runner to choose depends on which file was changed, hence the runner tag has to be conditionally decided. However, these jobs never execute and the value of never gets assigned as I always get:
This job is stuck because you don't have any active runners online or available with any of these tags assigned to them: $RUNNER_TAG
I believe it is because the rules blocks isn't executed and hence the $RUNNER_TAG variable not resolved to its actual value at the point when job/workflow is being initialized and runner being searched.
If my doubt is correct, then probably it's a circular dependency that job initialization requires $RUNNER_TAG but the resolution of $RUNNER_TAG requires job initialization.
If the above is correct, what is the right way to handle it and what stage can I conditionally decide and assign $RUNNER_TAG its value so it doesn’t hinder job/workflow initialization?
gitlab-runner --version
Version: 14.7.0
Git revision: 98daeee0
Git branch: 14-7-stable
GO version: go1.17.5
Built: 2022-01-19T17:11:48+0000
OS/Arch: linux/amd64
I think what you are doing is over complicating what you need to do.
Instead of trying to abstract the tag and dynamically create some variable, simply make each job responsible for registering itself within a pipeline run based on if a particular file path changed.
It might feel like code duplication but it actually keeps your CI a lot simpler and easier to understand.
Job1:
Run when file changes
Tag : some tag
Job2:
Run when some other file changes
Tag: sometag2
Job 3:
Run when a third different file changes
Tag: sometag3
I have a CICD configuration that looks something like this:
.rule_template: &rule_configuration
rules:
- changes:
- file/dev/script1.txt
variables:
DESTINATION_HOST: somehost1
RUNNER_TAG: somerunner1
- changes:
- file/test/script1.txt
variables:
DESTINATION_HOST: somehost2
RUNNER_TAG: somerunner2
default:
tags:
- scripts
stages:
- lint
deploy scripts 1/6:
<<: *rule_configuration
tags:
- ${RUNNER_TAG}
stage: lint
script: |
echo "Add linting here!"
....
In short, which runner to choose depends on which file was changed, hence the runner tag has to be conditionally decided. However, these jobs never execute and the value of never gets assigned as i always get:
This job is stuck because you don't have any active runners online or available with any of these tags assigned to them: ${RUNNER_TAG}
Any idea, what is it this way and what can I do to resolve this?
gitlab-runner --version
Version: 14.7.0
Git revision: 98daeee0
Git branch: 14-7-stable
GO version: go1.17.5
Built: 2022-01-19T17:11:48+0000
OS/Arch: linux/amd64
Tags map jobs to runners. I tag my runners with the type of executor they use, e.g. - shell, docker.
Based on the error message, you do not have any runners with the tag ${RUNNER_TAG}, which means that it is not resolving the variable the way you want it to.
Instead of combining rules like this, make separate jobs for each, and a rule for each to say when to trigger it.
I have faced this issue, and similar issues many times while trying to do some dynamic pipelines for a multi-client environment.
The config you have above should work for your purposes to the best of my knowledge, but since it is not there is another way to accomplish this with trigger jobs.
Create a trigger job for each possible runner tag. You can use extends to reduce the total code required for this.
gitlab-ci.yml
stages:
- trigger
- lint
.trigger:
stage: trigger
trigger:
include:
- local: ./lint-job.yml
strategy: depend
trigger-lint-script1:
extends:
- .trigger
variables:
RUNNER_TAG: somerunner1
rules:
- changes:
- file/dev/script1.txt
trigger-lint-script2:
extends:
- .trigger
variables:
RUNNER_TAG: somerunner2
rules:
- changes:
- file/dev/script2.txt
Create a trigger job with associated rules for each possible tag. This way you can change more than one of the specified files in a single commit with no issues. Define the triggered job in lint-job.yml
lint-job.yml
deploy scripts 1/6:
tags: [$RUNNER_TAG]
stage: lint
script: |
echo "Add linting here!"
There are other ways to accomplish this, but this method is by far the simplest and cleanest for this particular use.
I have a dependency problem. My pipeline looks like it gets the dependencies required for jobs first, and finally runs a stage cleanup that cleans them all. The problem is that I have one stage with manual launch which also needs these dependencies but they are cleared.
Question can I somehow run a stage which has dependencies by running a manual stage? is there any other way i can solve this problem?
The normal behaviour of GitLab-CI is to clone the git repository at each job because the jobs can be run on different runners and thus need to be independent.
The automatic clone can be disabled by adding:
job-with-no-git-clone:
variables:
GIT_STRATEGY: none
If you need to use in a job some files/directories created in a previous stage, you must add them as GitLab artifacts
stages:
- one
- two
job-with-git-clone:
stage: one
script:
# this script creates something in the folder data
# (which means $CI_PROJECT_DIR/data)
do_something()
artifacts:
paths:
- data/
job2-with-git-clone:
stage: two
script:
# here you can use the files created in data
job2-with-no-git-clone:
stage: two
variables:
GIT_STRATEGY: none
script:
# here you can use the files created in data
How can I run different CI deployment scripts on merge to master depending on the labels attached to the merge request?
I have a repository from which I build different versions of my software. I keep it in one repository as the systems share 90% of the code but there are differences that defitively need code modifications. On merge requests all versions are buildt and a suite of tests is run. Usually I want to deploy on accepting the merge request.
As not always the changes are relevant for all systems I would like to attach labels to the merge request that decide which deployments scripts are run on accepting the merge request. I already tried to automatically decide on the changed code parts but this is not possible as often I expand a shared library but this is only relevant for one of the systems.
I am aware of variables but I don't know how to apply them on merge accept in YML like this
deploy:
stage: deploy
script:
...
only:
- master
Update on strategy:
As CI_MERGE_REQUEST_LABELS is not available with only:master I will try to do a beta deployment depending on merge request labels in only:merge-request. In only:master I will deploy the betas that have changed. This most likely will fit my needs. I will add it as a solution once it works.
I finally solved it this way:
My YML script has three stages:
stages:
- buildtest
- createbeta
- deploy
buildtest:
stage: buildtest
script:
- ... run unit tests
- ... build all systems
- ... run scripted tests on all systems
only:
refs:
- merge_requests
createbeta:
stage: createbeta
script:
- ... run setup and update package creation with parameter $CI_MERGE_REQUEST_LABELS
- ... run update package tests with parameter $CI_MERGE_REQUEST_LABELS
- ... run beta deployment scripts with parameter $CI_MERGE_REQUEST_LABELS (see text)
only:
refs:
- merge_requests
deploy:
stage: deploy
script:
- ... run production deployment scripts (see text)
only:
refs:
- master
The first stages are run on merge request creation.
As changes to shared libraries might affect all systems all builds and tests are run in stage "buildtest".
The scripts in stage "createbeta" check for existance of the merge request label for the corresponding system and are skipped if the system is not involved by the labels.
The script for beta deployment creates a signal file "deploy_me" in the beta folder (important) if it runs
When the request is merged the deployment script runs in stage "deploy". It checks for the existance of the "deploy_me" file and only deploys and informs via mail if the file exists.
This way I can easily decide which system I want to deploy by applying a labes to the merge request. I can thorowly test the new feature with the beta version and make sure that changes do not break the other systems as unittests and system tests are run for all systems.
As the GitLab runner runs in a Windows environment (yes, this makes sense as I work with Delphi) here is the way I find the system label in a Windows cmd file for those who are interested. I use %* as the labels are separated by spaces and treated as individual command line parameters.
echo %* | findstr /i /c:"MyCoolSystem" > nul
if %ERRORLEVEL% EQU 0 goto runit
rem If the label is not supplied with the merge request, do nothing
goto ok
:runit
... content
:ok
Perhaps this helps someone with a similar environment and similar workflow.
It seems rules replaces only/except functionality in the latests GitLab versions.
Before, specifying that a job had to be executed only for master branch, for example, was very straightforward.
How would that be done with rules?
I'm guessing GitLab provides some variable that specifies the current branch's name, but I cannot find that. The only examples I see are regarding merge requests.
In other words, if I have the following job, how to restrict it to run only in potato branch?
unit_tests:
stage: test
script: dotnet vstest test/*UnitTests/bin/Release/**/*UnitTests.dll --Blame
rules:
- exists:
- test/*UnitTests/*UnitTests.csproj
I guess this would be it:
unit_tests:
stage: test
script: dotnet vstest test/*UnitTests/bin/Release/**/*UnitTests.dll --Blame
rules:
- if: $CI_COMMIT_BRANCH == "potato"
Here are the variable references:
https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
Here is an example from gitlab-runner project source code itself
https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/.gitlab/ci/test.gitlab-ci.yml
job-name:
script:
- echo "i am potato"
rules:
- if: '$CI_COMMIT_BRANCH == "potato"'