How can I create a CI job that spans more than one stage, to improve parallelism?
As in the following diagram:
The idea is that slow_build should start as early as build, but test doesn't depend on it, so test should be able to start as soon as build is done.
(Note that this is a simplification: each stage has multiple processes running in parallel, otherwise I could just bundle build and test together.)
This is now possible as of Gitlab version 12.2. By adding the keyword needs to jobs that depend on other jobs, stages can now run concurrently. The full documentation for the needs keyword is here, but an example from the docs follows: https://docs.gitlab.com/ee/ci/yaml/#needs
linux:build:
stage: build
mac:build:
stage: build
lint:
stage: test
needs: []
linux:rspec:
stage: test
needs: ["linux:build"]
linux:rubocop:
stage: test
needs: ["linux:build"]
mac:rspec:
stage: test
needs: ["mac:build"]
mac:rubocop:
stage: test
needs: ["mac:build"]
production:
stage: deploy
Since the lint job doesn't need anything, it runs instantly, as does linux:build and mac:build. However, if linux:build finishes before mac:build then both linux:rspec and linux:rubocop can start, even before mac:build and the build stage complete.
As usual, without the needs keyword, the production job requires all previous jobs to complete before it starts.
When using needs in your pipelines, you can also view a Directed Acyclic Graph of your jobs in the pipeline view. More on that can be found here: https://docs.gitlab.com/ee/ci/directed_acyclic_graph/index.html
Related
I have many different GitLab CI jobs in my repository and dependent on variables that are set by an user in a config file I want to execute different sequences of jobs. My approach is to create a scheduler job that analyzes the config file and executes the jobs accordingly. However, I cannot figure out how to execute another job from within a job.
Any help is appreciated!
This would be a good use case for dynamic child pipelines. This is pretty much the only way to customize a pipeline based on the outcome of another job.
From the docs:
generate-config:
stage: build
script: generate-ci-config > generated-config.yml
artifacts:
paths:
- generated-config.yml
child-pipeline:
stage: test
trigger:
include:
- artifact: generated-config.yml
job: generate-config
In your case, the script generate-ci-config would be the analysis of your config files and creates a job configuration conditionally based on the config contents.
I have a dependency problem. My pipeline looks like it gets the dependencies required for jobs first, and finally runs a stage cleanup that cleans them all. The problem is that I have one stage with manual launch which also needs these dependencies but they are cleared.
Question can I somehow run a stage which has dependencies by running a manual stage? is there any other way i can solve this problem?
The normal behaviour of GitLab-CI is to clone the git repository at each job because the jobs can be run on different runners and thus need to be independent.
The automatic clone can be disabled by adding:
job-with-no-git-clone:
variables:
GIT_STRATEGY: none
If you need to use in a job some files/directories created in a previous stage, you must add them as GitLab artifacts
stages:
- one
- two
job-with-git-clone:
stage: one
script:
# this script creates something in the folder data
# (which means $CI_PROJECT_DIR/data)
do_something()
artifacts:
paths:
- data/
job2-with-git-clone:
stage: two
script:
# here you can use the files created in data
job2-with-no-git-clone:
stage: two
variables:
GIT_STRATEGY: none
script:
# here you can use the files created in data
Given the following .gitlab-ci.yml:
---
stages:
- .pre
- test
- build
compile-build-pipeline:
stage: .pre
script: [...]
artifacts:
paths: [".artifacts/build.yaml"]
lint-source:
stage: .pre
script: [...]
run-tests:
stage: test
rules:
- if: '$CI_COMMIT_BRANCH == "$CI_DEFAULT_BRANCH"'
trigger:
strategy: depend
include:
artifact: .artifacts/tests.yaml
job: "compile-test-pipeline"
needs: ["compile-test-pipeline"]
build-things:
stage: test
rules:
- if: '$CI_COMMIT_BRANCH == "$CI_DEFAULT_BRANCH"'
trigger:
strategy: depend
include:
artifact: .artifacts/build.yaml
job: "compile-build-pipeline"
needs: ["compile-build-pipeline"]
...
The configuration should always run (any branch, any source). Tests and build jobs should be run only on the default branch.
However, no jobs are run for merge requests, and manually triggering the pipeline on branches other than the default one give the error No Jobs/Stages for this Pipeline.
I've tried explicitly setting an always run rule using rules: [{if: '$CI_PIPELINE_SOURCE'}] to the jobs in the .pre stage, but no dice.
What am I doing wrong?
As per the docs:
You must have a job in at least one stage other than .pre or .post.
In the above configuration, no jobs other than the ones in .pre are added on merge request events, hence no jobs are added at all.
I'm glad you added this info into the question: However, no jobs are run for merge requests
CI_COMMIT_BRANCH is not available in MR-based events. Here is an fragment from official docs:
The commit branch name. Available in branch pipelines, including pipelines for the default branch.
**Not available in merge request pipelines or tag pipelines.**
When working with MR and willing to check against branch name, you might want to use:
CI_MERGE_REQUEST_SOURCE_BRANCH_NAME or CI_MERGE_REQUEST_TARGET_BRANCH_NAME
List of envs here
I'm looking for a way to run parallels jobs in different runners. I have several powerful runners set up for GitLab CI. In general, it's ok to run jobs on the same runner because they're executed in Docker container.
However, now I have a Pipeline that jobs are executed in parallel and each job consumes lots of CPU and Mem.(it's by design, not an issue). If it's unlucky that GitLab CI schedules those jobs to the same runner, job fails.
And, I want this limitation applies to this project ONLY, as my runners have 30+ CPU and 120GB+ Memory.
Thanks in advance.
It is possible, if you have set up say two runners (either specific, shared or group runners) with tags.
Say, runner1 has tags runner1-ci, my-runner1
Similarly, runner2 has tags runner2-ci, my-runner2
Now, in your .gitlab-ci.yml file, you can use the tags like below, so a job will pick up that particular runner and execute the job.
image: maven:latest
stages:
- build
- test
- deploy
install_dependencies:
stage: build
tags:
- runner1-ci
script:
- pwd
- echo "Build"
test:
stage: test
tags:
- runner2-ci
script:
- echo "Testing"
deploy:
stage: deploy
tags:
- runner1-ci
script:
- echo "Deploy to nexus"
Note: This is just an example .gitlab-ci.yml to demonstrate the use of tags in pipeline.
How can I run different CI deployment scripts on merge to master depending on the labels attached to the merge request?
I have a repository from which I build different versions of my software. I keep it in one repository as the systems share 90% of the code but there are differences that defitively need code modifications. On merge requests all versions are buildt and a suite of tests is run. Usually I want to deploy on accepting the merge request.
As not always the changes are relevant for all systems I would like to attach labels to the merge request that decide which deployments scripts are run on accepting the merge request. I already tried to automatically decide on the changed code parts but this is not possible as often I expand a shared library but this is only relevant for one of the systems.
I am aware of variables but I don't know how to apply them on merge accept in YML like this
deploy:
stage: deploy
script:
...
only:
- master
Update on strategy:
As CI_MERGE_REQUEST_LABELS is not available with only:master I will try to do a beta deployment depending on merge request labels in only:merge-request. In only:master I will deploy the betas that have changed. This most likely will fit my needs. I will add it as a solution once it works.
I finally solved it this way:
My YML script has three stages:
stages:
- buildtest
- createbeta
- deploy
buildtest:
stage: buildtest
script:
- ... run unit tests
- ... build all systems
- ... run scripted tests on all systems
only:
refs:
- merge_requests
createbeta:
stage: createbeta
script:
- ... run setup and update package creation with parameter $CI_MERGE_REQUEST_LABELS
- ... run update package tests with parameter $CI_MERGE_REQUEST_LABELS
- ... run beta deployment scripts with parameter $CI_MERGE_REQUEST_LABELS (see text)
only:
refs:
- merge_requests
deploy:
stage: deploy
script:
- ... run production deployment scripts (see text)
only:
refs:
- master
The first stages are run on merge request creation.
As changes to shared libraries might affect all systems all builds and tests are run in stage "buildtest".
The scripts in stage "createbeta" check for existance of the merge request label for the corresponding system and are skipped if the system is not involved by the labels.
The script for beta deployment creates a signal file "deploy_me" in the beta folder (important) if it runs
When the request is merged the deployment script runs in stage "deploy". It checks for the existance of the "deploy_me" file and only deploys and informs via mail if the file exists.
This way I can easily decide which system I want to deploy by applying a labes to the merge request. I can thorowly test the new feature with the beta version and make sure that changes do not break the other systems as unittests and system tests are run for all systems.
As the GitLab runner runs in a Windows environment (yes, this makes sense as I work with Delphi) here is the way I find the system label in a Windows cmd file for those who are interested. I use %* as the labels are separated by spaces and treated as individual command line parameters.
echo %* | findstr /i /c:"MyCoolSystem" > nul
if %ERRORLEVEL% EQU 0 goto runit
rem If the label is not supplied with the merge request, do nothing
goto ok
:runit
... content
:ok
Perhaps this helps someone with a similar environment and similar workflow.