I've written several pipeline templates that our development teams use in their projects and what I've found is that if I make a change
in a template to fix a bug, when we "retry" a job it doesn't pick up the latest changes from the template. The only way the job will receive the latest
reference to the template is if we push another commit to the developer's project.
Is there a flag or parameter we can set so that it always picks up the latest copy?
Here is what our gitlab yml looks like in the developer projects.
include:
- project: 'ops/awesome-pipelines'
file: 'pipelines/fresh_apps.yml'
Is there a flag or parameter we can set so that it always picks up the
latest copy?
I think the functionality that you are trying is not possible.
Quoting the gitlab docs https://docs.gitlab.com/ee/ci/yaml/index.html#includefile
When the pipeline starts, the .gitlab-ci.yml file configuration
included by all methods is evaluated. The configuration is a snapshot
in time and persists in the database. GitLab does not reflect any
changes to the referenced .gitlab-ci.yml file configuration until the
next pipeline starts.
Template changes are only reflected in new pipelines. The benefit of this is that you can reliably retry jobs and assure that template changes won't change the behavior of a retry.
You don't necessarily need to push another commit, but you will have to start another pipeline (for example, manually running a new pipeline for a branch or MR).
Related
The pipeline is triggered by new commits to a master branch and publishes the package
Currently, the version is set manually and I'de be happy to have it set automatically.
What I thought at first was adding the following tasks to the pipeline:
checkout $Build.SourceBranch
run version patch --force
git push
This works and the version is really incremented properly, the thing is, it triggers another run of the pipeline, which will increment again, which will.. you get the idea, endless loop.
Is there a better way for doing that?
I've added the same tasks as mentioned in the question with a small change.
Apparently there is a way to skip the pipeline triggering, see here
So the npm version task looks like this:
version patch -m "Bump version to %s [skip ci]" --force
which prevents the next build from being triggered.
TIP: remember to give the 'author' (Azure DevOps user) permissions to Bypass policies when pushing if any.
Don't use the files in the source repo to keep track of the current/next version. I don't think you can break the loop easily, if you do.
You might be able to get away with running npm --no-git-tag-version version to increment the package.json version inside the build agent without the commit, so that you don't have a change you would have to push back to the origin. It should just change package.json and leave it dirty.
Wait until after the build has succeeded. Use a custom script task to extract the version from package.json, then git reset --hard (there is no reason to keep anything that has changed on the build server). While this will undo the change in package.json, you can now create a tag on the head that contains that version, then do a git push origin {tag-name} which should not introduce a new commit on the origin which would then re-trigger your pipeline.
Actually, I don't think a pipeline will be triggered just because a tag was added, but I honestly haven't tested it. I'm pretty sure it won't.
Sequence of tasks in your pipeline:
[Given a source repo where you have made sure to set the major and minor versions in package.json to reflect the current state of the code according to the rules of Semantic Versioning, and leave the patch value always set to 0, and, if desired, using pre-* values to describe the quality of the major/minor value:]
(This is not really a task, but just describing the start of the execution:) The job starts. Automatically, the code is pulled from the source repo into the build agent.
Using a utility command-line task and code or script that you write, run git describe --tags to find the most recent tag with a tagname matching the pattern you use (see below). (I prefer this sequence over calling npm version from-git because npm will just use the latest tag, which might not be a version number, depending on how much control you have over the branch.) Use string or regular expression operations to extract the major, minor, patch, and whatever pre-* value you might have. This is the previous version that we'll use to compare with what's in the package.json file. Note that you may have to follow these instructions to run a git command. Save it to a pipeline variable.
Using a utility command-line task and code or script that you write, run npm version to get the current major/minor/pre version out of the package.json file and save it to a different pipeline variable. I'm using PowerShell Core, so my command would look something like this to create a "currentPackageVersion" pipeline variable:
& npm.cmd version | ConvertFrom-Json | Select-Object -ExpandProperty {name-of-your-package} | Set-Variable -Name 'packageVersion' | Write-Output "#vso[task.setvariable variable=currentPackageVersion]$packageVersion"
Using a utility command-line task and code or script that you write, compare the previous version's major, minor, and pre-* values to determine whether any of them have changed. Set a new pipeline variable to reflect whether it has changed or not. I'll use the name "restartVersionPatchNumber", which is true if the current major, minor, or pre-* values are different from the previous version's major, minor, or pre-* values.
The npm Task conditionally runs a custom command: npm --no-git-tag-version version patch, which updates package.json in the build agent but does not commit the change, leaving your Working Area modified (dirty) (which might cause issues on subsequent builds if you are using your own build agents instead of hosted agents). The condition expression of the Task uses a custom condition that evaluates to the variable that I just set in the previous step ("restartVersionPatchNumber"). Note that if this task does not run, it should just use the value of the version that is in the package.json file (the current version that is now in the "currentPackageVersion" pipeline variable).
Using a utility command-line task and code or script that you write, run npm version to extract the new version that the npm version command set. Save it to a new pipeline variable; I'll call it "newVersionNumber".
The regular build tasks run, producing artifacts and possibly publishing them
Using a utility command-line task, run git reset --hard. You'll need to do this even if you're using a hosted build agent, because of the next step.
Using a utility command-line task, create a variable from the saved version number ("newVersionNumber") that contains the value of the tag's tagname that you want to use. Use a distinctive patter, like "AzPipelineBuild-PipelineName-PackageVersion-1.0.0" but using your version instead of 1.0.0.
Using a utility command-line task, run git tag {*tagname*}. For PowerShell, the syntax would be & git.exe tag $env:newVersionNumber
Using a utility command-line task, run git push origin {*tagname*}
Profit
You could (and maybe should) combine the command-line steps. I'm really partial to the PowerShell task running PowerShellCore (pwsh) as you may have guessed.
The package that got created as an artifact of the job will have the updated version as it was created in the build, and it will match the tag that is now in your original source code repo.
Alternately, use an external source (an Azure Function, etc.) or a different (second) git repo or even another branch that isn't tied to your CI trigger in your project that you use just for tracking build numbers, then use token replacement tasks to set the version just before you start the build. I don't like this idea much, but it would prevent re-triggering a new build.
Also, you're hopefully building just one package with that repo. The next thing to do is to publish that package to your Azure Artifacts feed, or npmjs.org or wherever. Don't rely on having the version baked into the original source code; anything that depends on that package should be pulling it out of that feed you published it to, not relying on it being built earlier in the build steps with a new version number.
Here's my implementation in Powershell which is a complete step based on the accepted answer. Note that this also handles some of the git issues such as the detached head that comes with this type of approach.
steps:
- checkout: self
persistCredentials: true
.
.
.
- task: PowerShell#2
displayName: 'Bump the version'
inputs:
targetType: 'inline'
script: |
$BranchName = "$(Build.SourceBranch)" -replace "refs/heads/"
git checkout $BranchName
git config --global user.email "anymail#anycompany.com"
git config --global user.name "Your name"
npm version prerelease -m "Auto increment pre-release version to %s [skip ci]" --force
git push
The Gitlab documentation says the following about GIT_STRATEGY: none:
none also re-uses the project workspace, but skips all Git operations (including GitLab Runner's pre-clone script, if present). It is mostly useful for jobs that operate exclusively on artifacts (e.g., deploy). Git repository data may be present, but it is certain to be out of date, so you should only rely on files brought into the project workspace from cache or artifacts.
I'm still a bit confused about how this is supposed to work. If the source code is not guaranteed to exist, then there might be no source in the project workspace and thus the .gitlab-ci.yml file would also be missing. Without a build script the job must fail. If the source is missing only part of the time depending on external factors, the job will fail randomly, which is even worse than failing every time. However, if it fails every single time then what's the point of the feature?
Another possibility I see is that .gitlab-ci.yml might be injected at runtime, so that even without a fresh copy of the repository there would be a build script. If so, could I define further files from my repository to inject into the build process? What are the restrictions on these particular jobs?
Yes, the .gitlab-ci.yml file is not copied onto the system just like all the other files. But that doesn't matter as the job is not run from the file. The job is run as a script on your target (and even before that as it defines the target it will run on). It is not possible to copy only selected files without a git clone although you may want to copy the files from some other server.
A good example of when you want to run GIT_STRATEGY: none are things like slackchat notifications as last stage of a build when you really don't want to clone gigabytes of repository data just to push a notification.
I am trying to integrate Atlassian Bamboo with a CloudFormation template that creates and deploys environments in AWS. One of the tags we add to the VPCs and EC2 instances is the name of the engineer that triggered the build.
I can't find a bamboo variable that contains the name of the user that triggered a build. I found one that has the username of the person who created a release in Jira, but this isn't what I want.
https://confluence.atlassian.com/display/BAMBOO/Bamboo+variables
Is there a way to get this variable?
You are looking for ManualBuildTriggerReason.userName, that will be the name of the person who manually triggered a build but if it was off a trigger this will not be present.
For the name of the person who didn't trigger the build manually but did it through a source code commit then you could probably be:
git log <commitFrom>..<commitTo> --format="%aN <%aE>" --reverse
And then you could use repository.revision.number for the current revision and then repository.previous.revision.number for the previous revision.
I am looking for a solution to automatically increment a package version (not to be confused with an assembly version) after each build on CI server (particularly Atlassian Bamboo). Every appx package has a version defined in its manifest file (appxmanifest). Thus in order to increase the version a manifest must be edited before commit. I am considering different approaches to implement this. The first one makes changes in manifest and pushes it back to the repo.
Starts building a plan (in order to lock a build number)
Modifies manifest so that a revision is set to the current build number
Pushes changes to SCM (particularly Atlassian Stash). This step shouldn't trigger the next build.
Continues building the package (invoke MSBuild, UT and other tasks)
Cons
Leads to incorrect workflow on Bamboo: checkout -> push -> build
Each build makes a new commit
Another approach is to setup post receive Stash hook which would modify appxmanifest.
Cons Hard to keep a build number in sync with Bamboo.
Is there any other (cleaner and proper) way to achieve this?
ex-Stash developer here (not that it matters),
I would highly recommend not checking in derived/version information or files. It's going to cause you no end of problems (some of which you have pointed out in your question).
My advice - generate what information you need on the build. I don't know anything about appx packaging, but can you use a placeholder/property (like this) which can be resolved on the Bamboo build? For our builds we use the git hash and timestamp as the version, and in the past I've also used the job/build number (timestamp is better though).
As more food for thought - if that appx version is important for developers to see locally, and it becomes hard to match up with the Git version then you can also attach a Git tag/note to the commit in Bamboo as well. The nice thing about that is that anyone fetching from Git can easily see that extra metadata, but it doesn't result in extra commits for every build. If the appx version need to be based off the previous version then this makes it possible for the build scripts to inspect the previous commit and bump the version appropriately.
I hope that helps.
I have a plan in Bamboo that starts whenever changes are made to the attached repositories (via polling).
Now, on each build, if successful, a CHANGELOG file is updated in the repo, which in turn, triggers another build. How can I omit certain files from Bamboo's polling, so that a build isn't started if changes are found for those files? Because otherwise, I enter in infinite loop, with a change to CHANGELOG triggering another build which in turn updated CHANGELOG and so on.
If this is not possible, what other viable solutions are there? Is it possible to attach a shell script somewhere before the build starts to check whether it's desired to start a new build?
It turned out that this was simpler than I've thought. In Plan Configuration, in the Repositories tab, on each repository, under Advanced, there is an Include / exclude files input where you can Customise what files Bamboo uses to detect changes. By adding a regular expression there, I got everything solved and working as expected.
Bamboo pattern matching reference: https://confluence.atlassian.com/display/BAMBOO/Pattern+matching+reference
The Bamboo Documentation says:
Bamboo will ignore build triggers if the local working copy and the
repository copy have the same revision numbers.
This might not be the best solution, but you might add an additional task at the end of the job/build which updates the repository again to avoid triggering a new build.
I'm not sure if this would then skip builds from repository updates which occur during the current build.