I have a gitlab-ci pipeline in the first step of which npm dependencies are downloaded. These npm packages are also required in all subsequent stages of the pipeline. That's why I wanted to take a closer look at the topic of caching in gitlab-ci.
Is there a key that makes the cache globally available for all stages belonging to the pipeline (of a specific branch) but generates it again on the next pipeline run?
cache:
key: $CI_COMMIT_REF_SLUG
The cache key should be unique for each pipeline run in the same branch.
You can use the $CI_PIPELINE_ID variable as the cache key to make the cache available for all stages belonging to the pipeline of a specific branch. Also it's a unique identifier for each pipeline run, which makes it ideal for use as a cache key.
Related
I've written several pipeline templates that our development teams use in their projects and what I've found is that if I make a change
in a template to fix a bug, when we "retry" a job it doesn't pick up the latest changes from the template. The only way the job will receive the latest
reference to the template is if we push another commit to the developer's project.
Is there a flag or parameter we can set so that it always picks up the latest copy?
Here is what our gitlab yml looks like in the developer projects.
include:
- project: 'ops/awesome-pipelines'
file: 'pipelines/fresh_apps.yml'
Is there a flag or parameter we can set so that it always picks up the
latest copy?
I think the functionality that you are trying is not possible.
Quoting the gitlab docs https://docs.gitlab.com/ee/ci/yaml/index.html#includefile
When the pipeline starts, the .gitlab-ci.yml file configuration
included by all methods is evaluated. The configuration is a snapshot
in time and persists in the database. GitLab does not reflect any
changes to the referenced .gitlab-ci.yml file configuration until the
next pipeline starts.
Template changes are only reflected in new pipelines. The benefit of this is that you can reliably retry jobs and assure that template changes won't change the behavior of a retry.
You don't necessarily need to push another commit, but you will have to start another pipeline (for example, manually running a new pipeline for a branch or MR).
We use GitLab CI/CD on a private GitLab server and maintain a quite complex gitlab-ci.yaml with includes and rules:
Some pipeline jobs only run on specific branches
Some pipeline jobs only run when specific variables are set
Some pipeline jobs are taken from the include without changes
Some pipeline jobs are taken from the include, but overwritten in the current file
etc.
Currently, our quality assurance steps are:
Using the GitLab API to lint the file
Building, testing and deploying a test project to test the pipeline roughly (smoke tests)
What we would like to add are unit tests where we can test the behaviour of the pipeline under specific conditions (specific branches, variables, other conditions) so we can ensure that specific settings lead to a specific pipeline configuration. Unfortunately, I was not able to find any information on how to properly test GitLab CI pipelines. How can we do that?
The Gitlab documentation says the following about GIT_STRATEGY: none:
none also re-uses the project workspace, but skips all Git operations (including GitLab Runner's pre-clone script, if present). It is mostly useful for jobs that operate exclusively on artifacts (e.g., deploy). Git repository data may be present, but it is certain to be out of date, so you should only rely on files brought into the project workspace from cache or artifacts.
I'm still a bit confused about how this is supposed to work. If the source code is not guaranteed to exist, then there might be no source in the project workspace and thus the .gitlab-ci.yml file would also be missing. Without a build script the job must fail. If the source is missing only part of the time depending on external factors, the job will fail randomly, which is even worse than failing every time. However, if it fails every single time then what's the point of the feature?
Another possibility I see is that .gitlab-ci.yml might be injected at runtime, so that even without a fresh copy of the repository there would be a build script. If so, could I define further files from my repository to inject into the build process? What are the restrictions on these particular jobs?
Yes, the .gitlab-ci.yml file is not copied onto the system just like all the other files. But that doesn't matter as the job is not run from the file. The job is run as a script on your target (and even before that as it defines the target it will run on). It is not possible to copy only selected files without a git clone although you may want to copy the files from some other server.
A good example of when you want to run GIT_STRATEGY: none are things like slackchat notifications as last stage of a build when you really don't want to clone gigabytes of repository data just to push a notification.
I am planning to create 4 stages
Source code checkout stages
Build for dev env stages
Build for uat env stages
Build for Prod env stages
Is it possible to use the same source code check out for all the stages? How?
This is actually straight forward:
Define your repository in the Repository tab of the plan configuration
Add a Sourcecode Checkout task for each build job in the plan.
By virtue of the repository definition for the plan, a consistent snapshot at the time the plan was started, will be used for the checkout tasks i.e. they will each fetch the same code.
This is not clearly documented in the Bamboo docs but is discussed here https://answers.atlassian.com/questions/33651/stages-and-artifact-passing
Above answer works, but I think you should not build same branch with all environments. It might be better to use better branching workflow,so that you can easily deploy correct change to required environment.
I have a simple Gitlab pipeline setup with two stages: build & test. Both stages are supposed to share cached files but they don't appear to, resulting in the test stage failing. As best I can, the problem is that each stage uses a different runner and the cached files use the runner ID as part of the path.
.gitlab-ci.ym
...
cache:
key: "build"
untracked: true
...
The build stage outputs the following
Creating cache build...
untracked: found 787 files
Uploading cache.zip to https://runners-cache-1.gitlab.com:443/runner/runner/30dcea4b/project/1704442/build
The test stage outputs the following
Checking cache for build...
$ mvn test
I believe this means the cache was NOT found because there is no download information; but it's not clear.
I can also see that each stage uses a different runner and since the runner ID is part of the cache path, I suspect that is the problem.
I need to either use the same runner for each stage or share the cache across runners. I don't understand how to do either.
Any help would be appreciated.
It appears the cache feature is appropriately named, it's only for improving build performance and is not guaranteed to have the data, much like a real cache.
The correct approach is to use artifacts with dependencies.