I'm running a maven build in Jenkins and my builds are failing because Jenkins isn't smart enough to clean up after my build - it just tries to delete the entire workspace and the builds fails immediatly. Is there a way to NOT have jenkins fail because of this (or even try to do it), but instead let my "mvn clean" script run first? If I can do this then I can do the specialized cleanup that is required.
I suspect that you are using Workspace Cleanup Plugin. First thing to do is to check what happens if you do it as a pre-build step, not post-build.
Alternatively there is a custom step in the build that tries to wipe out the workspace. See what happens if you invoke it as the first build step, not the last. Or try using the aforementioned plugin.
If it does not help, here are other general clenup strategies you may employ:
Start another job as a post-build step that will cleanup for the upstream job.
Have a regularly scheduled (say, nightly) job that will do the cleanup. In this case you'll probably need to 'partition' your workspace by builds - i.e. create a separate subdirectory in the workspace for every build instance (keyed by BUILD_ID) so that builds do not accidentally use leftover files from previous builds.
Have a cleanup build step in your job. In this case you do not need to create extra jobs. However, as it runs before the build is over it may not be able to cleanup completely (say, it won't be able to delete the artifacts that need to be archived in a post-build step).
To summarize: Yes, you can run you script first, just make sure it does not delete things prematurely. Also examine your job configuration for where that cleanup of yours is called and set it up as a pre-build step or remove it.
Related
The Gitlab documentation says the following about GIT_STRATEGY: none:
none also re-uses the project workspace, but skips all Git operations (including GitLab Runner's pre-clone script, if present). It is mostly useful for jobs that operate exclusively on artifacts (e.g., deploy). Git repository data may be present, but it is certain to be out of date, so you should only rely on files brought into the project workspace from cache or artifacts.
I'm still a bit confused about how this is supposed to work. If the source code is not guaranteed to exist, then there might be no source in the project workspace and thus the .gitlab-ci.yml file would also be missing. Without a build script the job must fail. If the source is missing only part of the time depending on external factors, the job will fail randomly, which is even worse than failing every time. However, if it fails every single time then what's the point of the feature?
Another possibility I see is that .gitlab-ci.yml might be injected at runtime, so that even without a fresh copy of the repository there would be a build script. If so, could I define further files from my repository to inject into the build process? What are the restrictions on these particular jobs?
Yes, the .gitlab-ci.yml file is not copied onto the system just like all the other files. But that doesn't matter as the job is not run from the file. The job is run as a script on your target (and even before that as it defines the target it will run on). It is not possible to copy only selected files without a git clone although you may want to copy the files from some other server.
A good example of when you want to run GIT_STRATEGY: none are things like slackchat notifications as last stage of a build when you really don't want to clone gigabytes of repository data just to push a notification.
TLDR: How can I arrange it so that a snapshot dependency does not trigger new builds?
For my test processes to run, they need to run on a "Test" environment. Creating such an environment is simple, but lengthy; it can take as much as 45 minutes to an hour to finish building a test environment. Further, the name of the environment, and other such variables, is not fixed until the environment has finished building.
In my TeamCity build definition, I could put "build environment if missing" as a build step. However, that means that the first test of the day will take 45 minutes to run.
Instead, we created a separate build, that is scheduled to run every morning, that builds the test environment for the day. Our test build then has a snapshot dependency to that build in order to use the parameters of that build to determine the environment information, and everything works as expected, except for one issue:
When a new test is run, it frequently seems to trigger a rebuild of the test environment creation.
We don't want this to ever happen; the test environment creation is 'done' for the day and should not need to run again until tomorrow. How can we achieve this?
You already have time based trigger => environment will be prepared every morning
Create Snapshot Dependency in your product TC configuration (not in that one which is preparing test environment) and tick 'Do not run new build if there is a suitable one'
Your configuration used to setup test environment should not have any VCS root (or point to some calm place of source control where submits will not happen). To physical setup your environment you should not need any source code mapping etc. - you may consume everything needed through your own NuGet packages for example.
Note: In this workflow every build of your real project will enlist build of configuration which sets test environment (so it's physically in build queue) but
when it's turn come up it will compare changes since last build => no submits on VCS found (it's pointing to calm place in SourceControl due step 3)) and so build will be skipped in <1s
I have a plan in Bamboo that starts whenever changes are made to the attached repositories (via polling).
Now, on each build, if successful, a CHANGELOG file is updated in the repo, which in turn, triggers another build. How can I omit certain files from Bamboo's polling, so that a build isn't started if changes are found for those files? Because otherwise, I enter in infinite loop, with a change to CHANGELOG triggering another build which in turn updated CHANGELOG and so on.
If this is not possible, what other viable solutions are there? Is it possible to attach a shell script somewhere before the build starts to check whether it's desired to start a new build?
It turned out that this was simpler than I've thought. In Plan Configuration, in the Repositories tab, on each repository, under Advanced, there is an Include / exclude files input where you can Customise what files Bamboo uses to detect changes. By adding a regular expression there, I got everything solved and working as expected.
Bamboo pattern matching reference: https://confluence.atlassian.com/display/BAMBOO/Pattern+matching+reference
The Bamboo Documentation says:
Bamboo will ignore build triggers if the local working copy and the
repository copy have the same revision numbers.
This might not be the best solution, but you might add an additional task at the end of the job/build which updates the repository again to avoid triggering a new build.
I'm not sure if this would then skip builds from repository updates which occur during the current build.
I'm wondering if there is a way to force TeamCity to run a given build step even if preceding build steps fail. I want to do this to clean up after the build and terminate any started processes etc.
In Msbuild you can set the property ContinueOnError for many tasks. I usually add this using a property so it can be switched on and off:
<MSBuild Projects="my.csproj" ContinueOnError="$(ContinueOnError)"/>
Then you can set it on and off via teamcity.
So if you do this for a build step it should continue thru to the others.
I don't know if you can do it with build steps, but I've done it with two builds. The second one uses a finished build trigger pointing at the first. Finished build triggers can be configured to fire even if the first build failed.
We do our builds by constructing NAnt scripts, and then using TeamCity's NAnt runner. NAnt (well, actually NAntContrib, but still ...) has a try ... catch ... finally mechanism that does what any programmer would expect - run the catch code if the try code fails, and run the finally code no matter what happens. We use it for the same purposes as you want, plus to collect artifacts that we want the build to publish even in case of failure (like test logs).
Newer versions of TeamCity has this feature built in.
When we build a Maven project without doing mvn clean, we sometimes get "voodoo errors" such as NoSuchMethodError. I believe these are caused by moving/renaming files.
I don't want to use the clean option in the CI, because it makes the build process take much longer. Is there another alternative?
You should always use clean in a CI build. CI builds must be reproducible and that requires starting from scratch!
And about the process taking longer: the whole point of using CI (one of the many) is that you can keep working while it's running, so that should not be a problem.
But what I like to do is use multiple layers of CI per project:
A first job compiles and executes some basic tests*, this build should take less than 5 minutes
if that succeeds, a second job executes all tests*, code metrics, javadocs etc
if that succeeds a third job deploys the build to a test server
(Or you can let the first job trigger both the second and the third job at once)
* You can implement the some tests / all tests functionality by configuring the maven surefire plugin differently per profile)
We have three build targets:
Continuous Integration: Builds without doing a Clean, and only run the tests identified by Clover. This runs after each commit. On success it deploys to the test server.
Nightly: Does a clean build and runs every single test. This runs every night. On success it deploys to the test server.
Release: Same as Nightly plus creates a source control label. Run manually.
The nightly build is more trustworthy in that a clean build is conducted. However, the CI build is quicker meaning feedback is faster on those occasions.
There is an underlying problem here with the build time, but this is at least a work around while you look at more permanent ways to address that.