I have a simple Gitlab pipeline setup with two stages: build & test. Both stages are supposed to share cached files but they don't appear to, resulting in the test stage failing. As best I can, the problem is that each stage uses a different runner and the cached files use the runner ID as part of the path.
.gitlab-ci.ym
...
cache:
key: "build"
untracked: true
...
The build stage outputs the following
Creating cache build...
untracked: found 787 files
Uploading cache.zip to https://runners-cache-1.gitlab.com:443/runner/runner/30dcea4b/project/1704442/build
The test stage outputs the following
Checking cache for build...
$ mvn test
I believe this means the cache was NOT found because there is no download information; but it's not clear.
I can also see that each stage uses a different runner and since the runner ID is part of the cache path, I suspect that is the problem.
I need to either use the same runner for each stage or share the cache across runners. I don't understand how to do either.
Any help would be appreciated.
It appears the cache feature is appropriately named, it's only for improving build performance and is not guaranteed to have the data, much like a real cache.
The correct approach is to use artifacts with dependencies.
Related
I just started using Bazel for our team's GitLab CI pipeline, so I wonder if we could use the same GCS bucket as remote cache for master and feature branches.
My thinking is that it could lead to a race condition if the cache is updated by 2 build job simultaneously. One more scenario is creating docker image right after a build job: it could put the wrong binaries into the Docker image.
Could anyone correct me on that? Any help would be much appreciated.
Bazel has reproducible builds, so you can be sure that when branches share some code it's identical.
It is possible (although unlikely) that you build the same part twice, but since both builds should give the same result it doesn't matter.
The Gitlab documentation says the following about GIT_STRATEGY: none:
none also re-uses the project workspace, but skips all Git operations (including GitLab Runner's pre-clone script, if present). It is mostly useful for jobs that operate exclusively on artifacts (e.g., deploy). Git repository data may be present, but it is certain to be out of date, so you should only rely on files brought into the project workspace from cache or artifacts.
I'm still a bit confused about how this is supposed to work. If the source code is not guaranteed to exist, then there might be no source in the project workspace and thus the .gitlab-ci.yml file would also be missing. Without a build script the job must fail. If the source is missing only part of the time depending on external factors, the job will fail randomly, which is even worse than failing every time. However, if it fails every single time then what's the point of the feature?
Another possibility I see is that .gitlab-ci.yml might be injected at runtime, so that even without a fresh copy of the repository there would be a build script. If so, could I define further files from my repository to inject into the build process? What are the restrictions on these particular jobs?
Yes, the .gitlab-ci.yml file is not copied onto the system just like all the other files. But that doesn't matter as the job is not run from the file. The job is run as a script on your target (and even before that as it defines the target it will run on). It is not possible to copy only selected files without a git clone although you may want to copy the files from some other server.
A good example of when you want to run GIT_STRATEGY: none are things like slackchat notifications as last stage of a build when you really don't want to clone gigabytes of repository data just to push a notification.
I am planning to create 4 stages
Source code checkout stages
Build for dev env stages
Build for uat env stages
Build for Prod env stages
Is it possible to use the same source code check out for all the stages? How?
This is actually straight forward:
Define your repository in the Repository tab of the plan configuration
Add a Sourcecode Checkout task for each build job in the plan.
By virtue of the repository definition for the plan, a consistent snapshot at the time the plan was started, will be used for the checkout tasks i.e. they will each fetch the same code.
This is not clearly documented in the Bamboo docs but is discussed here https://answers.atlassian.com/questions/33651/stages-and-artifact-passing
Above answer works, but I think you should not build same branch with all environments. It might be better to use better branching workflow,so that you can easily deploy correct change to required environment.
I am preparing for gitlab-ci setup but I've reached 2 questions that I can not find answer to:
1) If there is no support for artifacts on Windows (using shell executor), what will be uploaded after build script finishes?
2) Are artifacts the right way to tell the runner what should be uploaded? Or is it only something extra to tell the runner that some generated garbage might be interesting so it would be nice to upload it beside the true result?
If you don't specify artifacts nothing will get uploaded automatically. You will only see the build log.
E.g. if you use the docker runner the entire container will be removed after the build finishes. The runner will upload your artifacts to gitlab and that is it.
Obviously you are free to copy/upload any asset during the build as you see fit.
When we build a Maven project without doing mvn clean, we sometimes get "voodoo errors" such as NoSuchMethodError. I believe these are caused by moving/renaming files.
I don't want to use the clean option in the CI, because it makes the build process take much longer. Is there another alternative?
You should always use clean in a CI build. CI builds must be reproducible and that requires starting from scratch!
And about the process taking longer: the whole point of using CI (one of the many) is that you can keep working while it's running, so that should not be a problem.
But what I like to do is use multiple layers of CI per project:
A first job compiles and executes some basic tests*, this build should take less than 5 minutes
if that succeeds, a second job executes all tests*, code metrics, javadocs etc
if that succeeds a third job deploys the build to a test server
(Or you can let the first job trigger both the second and the third job at once)
* You can implement the some tests / all tests functionality by configuring the maven surefire plugin differently per profile)
We have three build targets:
Continuous Integration: Builds without doing a Clean, and only run the tests identified by Clover. This runs after each commit. On success it deploys to the test server.
Nightly: Does a clean build and runs every single test. This runs every night. On success it deploys to the test server.
Release: Same as Nightly plus creates a source control label. Run manually.
The nightly build is more trustworthy in that a clean build is conducted. However, the CI build is quicker meaning feedback is faster on those occasions.
There is an underlying problem here with the build time, but this is at least a work around while you look at more permanent ways to address that.