How to increase a version of an npm package using Azure Devops pipeline - npm

The pipeline is triggered by new commits to a master branch and publishes the package
Currently, the version is set manually and I'de be happy to have it set automatically.
What I thought at first was adding the following tasks to the pipeline:
checkout $Build.SourceBranch
run version patch --force
git push
This works and the version is really incremented properly, the thing is, it triggers another run of the pipeline, which will increment again, which will.. you get the idea, endless loop.
Is there a better way for doing that?

I've added the same tasks as mentioned in the question with a small change.
Apparently there is a way to skip the pipeline triggering, see here
So the npm version task looks like this:
version patch -m "Bump version to %s [skip ci]" --force
which prevents the next build from being triggered.
TIP: remember to give the 'author' (Azure DevOps user) permissions to Bypass policies when pushing if any.

Don't use the files in the source repo to keep track of the current/next version. I don't think you can break the loop easily, if you do.
You might be able to get away with running npm --no-git-tag-version version to increment the package.json version inside the build agent without the commit, so that you don't have a change you would have to push back to the origin. It should just change package.json and leave it dirty.
Wait until after the build has succeeded. Use a custom script task to extract the version from package.json, then git reset --hard (there is no reason to keep anything that has changed on the build server). While this will undo the change in package.json, you can now create a tag on the head that contains that version, then do a git push origin {tag-name} which should not introduce a new commit on the origin which would then re-trigger your pipeline.
Actually, I don't think a pipeline will be triggered just because a tag was added, but I honestly haven't tested it. I'm pretty sure it won't.
Sequence of tasks in your pipeline:
[Given a source repo where you have made sure to set the major and minor versions in package.json to reflect the current state of the code according to the rules of Semantic Versioning, and leave the patch value always set to 0, and, if desired, using pre-* values to describe the quality of the major/minor value:]
(This is not really a task, but just describing the start of the execution:) The job starts. Automatically, the code is pulled from the source repo into the build agent.
Using a utility command-line task and code or script that you write, run git describe --tags to find the most recent tag with a tagname matching the pattern you use (see below). (I prefer this sequence over calling npm version from-git because npm will just use the latest tag, which might not be a version number, depending on how much control you have over the branch.) Use string or regular expression operations to extract the major, minor, patch, and whatever pre-* value you might have. This is the previous version that we'll use to compare with what's in the package.json file. Note that you may have to follow these instructions to run a git command. Save it to a pipeline variable.
Using a utility command-line task and code or script that you write, run npm version to get the current major/minor/pre version out of the package.json file and save it to a different pipeline variable. I'm using PowerShell Core, so my command would look something like this to create a "currentPackageVersion" pipeline variable:
& npm.cmd version | ConvertFrom-Json | Select-Object -ExpandProperty {name-of-your-package} | Set-Variable -Name 'packageVersion' | Write-Output "#vso[task.setvariable variable=currentPackageVersion]$packageVersion"
Using a utility command-line task and code or script that you write, compare the previous version's major, minor, and pre-* values to determine whether any of them have changed. Set a new pipeline variable to reflect whether it has changed or not. I'll use the name "restartVersionPatchNumber", which is true if the current major, minor, or pre-* values are different from the previous version's major, minor, or pre-* values.
The npm Task conditionally runs a custom command: npm --no-git-tag-version version patch, which updates package.json in the build agent but does not commit the change, leaving your Working Area modified (dirty) (which might cause issues on subsequent builds if you are using your own build agents instead of hosted agents). The condition expression of the Task uses a custom condition that evaluates to the variable that I just set in the previous step ("restartVersionPatchNumber"). Note that if this task does not run, it should just use the value of the version that is in the package.json file (the current version that is now in the "currentPackageVersion" pipeline variable).
Using a utility command-line task and code or script that you write, run npm version to extract the new version that the npm version command set. Save it to a new pipeline variable; I'll call it "newVersionNumber".
The regular build tasks run, producing artifacts and possibly publishing them
Using a utility command-line task, run git reset --hard. You'll need to do this even if you're using a hosted build agent, because of the next step.
Using a utility command-line task, create a variable from the saved version number ("newVersionNumber") that contains the value of the tag's tagname that you want to use. Use a distinctive patter, like "AzPipelineBuild-PipelineName-PackageVersion-1.0.0" but using your version instead of 1.0.0.
Using a utility command-line task, run git tag {*tagname*}. For PowerShell, the syntax would be & git.exe tag $env:newVersionNumber
Using a utility command-line task, run git push origin {*tagname*}
Profit
You could (and maybe should) combine the command-line steps. I'm really partial to the PowerShell task running PowerShellCore (pwsh) as you may have guessed.
The package that got created as an artifact of the job will have the updated version as it was created in the build, and it will match the tag that is now in your original source code repo.
Alternately, use an external source (an Azure Function, etc.) or a different (second) git repo or even another branch that isn't tied to your CI trigger in your project that you use just for tracking build numbers, then use token replacement tasks to set the version just before you start the build. I don't like this idea much, but it would prevent re-triggering a new build.
Also, you're hopefully building just one package with that repo. The next thing to do is to publish that package to your Azure Artifacts feed, or npmjs.org or wherever. Don't rely on having the version baked into the original source code; anything that depends on that package should be pulling it out of that feed you published it to, not relying on it being built earlier in the build steps with a new version number.

Here's my implementation in Powershell which is a complete step based on the accepted answer. Note that this also handles some of the git issues such as the detached head that comes with this type of approach.
steps:
- checkout: self
persistCredentials: true
.
.
.
- task: PowerShell#2
displayName: 'Bump the version'
inputs:
targetType: 'inline'
script: |
$BranchName = "$(Build.SourceBranch)" -replace "refs/heads/"
git checkout $BranchName
git config --global user.email "anymail#anycompany.com"
git config --global user.name "Your name"
npm version prerelease -m "Auto increment pre-release version to %s [skip ci]" --force
git push

Related

Gitlab CI Jobs Not Using Latest Template Changes

I've written several pipeline templates that our development teams use in their projects and what I've found is that if I make a change
in a template to fix a bug, when we "retry" a job it doesn't pick up the latest changes from the template. The only way the job will receive the latest
reference to the template is if we push another commit to the developer's project.
Is there a flag or parameter we can set so that it always picks up the latest copy?
Here is what our gitlab yml looks like in the developer projects.
include:
- project: 'ops/awesome-pipelines'
file: 'pipelines/fresh_apps.yml'
Is there a flag or parameter we can set so that it always picks up the
latest copy?
I think the functionality that you are trying is not possible.
Quoting the gitlab docs https://docs.gitlab.com/ee/ci/yaml/index.html#includefile
When the pipeline starts, the .gitlab-ci.yml file configuration
included by all methods is evaluated. The configuration is a snapshot
in time and persists in the database. GitLab does not reflect any
changes to the referenced .gitlab-ci.yml file configuration until the
next pipeline starts.
Template changes are only reflected in new pipelines. The benefit of this is that you can reliably retry jobs and assure that template changes won't change the behavior of a retry.
You don't necessarily need to push another commit, but you will have to start another pipeline (for example, manually running a new pipeline for a branch or MR).

Is there anyway to check that an npm package has been published from the correct branch?

So, we have Project A that has a develop and release branch. We reference a package from another one of our projects, Project B, which also has develop and release branches. What I'd like to be able to do is check that if I'm building / deploying Project A from it's release branch through Azure Devops that it is referencing the package that was created from the release branch for Project B.
My initial thoughts were to look into npm dist tags: https://docs.npmjs.com/cli/dist-tag but if I understand this correctly this doesn't guarantee that the package has ACTUALLY been published from that branch i.e. I could be on my own branch and simply publish the package with a tag of 'release'. Is there a way to automatically add a dist tag if you publish from a specified branch?
My next issue, which may well need to be another question, is whether there is a way in Azure devops to check against this package to ensure it has been published from the correct branch and if not then fail the build. So, if dist tags were used is there a way to check against a package dist tag as a part of the build to ensure it had the right tag e.g. 'release'?
For first question. you can automatically publish and add dist-tag to your package with azure devops build pipeline, which means you need to create a build pipeline to publish your package. And you can add a powershell task to guarantee that the package is published from a certain branch.
To add a powershell task run an inline script before npm publish task.
$sourceBranch = "$(Build.SourceBranchName)"
if($sourceBranch -ne "release")
{
exit 1
}
Above script checks if the sourceBranch is release branch. If the source Branch is not from release branch it will fail the task.
To publish your npm task with tag information. You can add a npm task to run custom command as shown in below pic. You can dynamic defined your tag name using the variables.(eg. publish --tag $(build.Build.SourceBranchName))
Make sure you enable CI build for this build pipeline. And the trigger is release branch
With above steps, your package will be automatically published with your self-defined tag.
2:
For your second question, to check against a package dist-tag in you build pipeline. You can also add a powershell task to check the tag information of your package and fail the task if the condition fails.
set-location -Path "$(Build.SourcesDirectory)"
$tags = npm view packageName dist-tags
if( $tags.Contains("release") -and ($tags -ne $null))
{
exit 1
}
Above script get the tag information using command npm view packageName dist-tags. and check if the tag contains "release". And the task will fail if the condition falses.

Is the .gitlab-ci.yml available for jobs with GIT_STRATEGY=none in Gitlab CI?

The Gitlab documentation says the following about GIT_STRATEGY: none:
none also re-uses the project workspace, but skips all Git operations (including GitLab Runner's pre-clone script, if present). It is mostly useful for jobs that operate exclusively on artifacts (e.g., deploy). Git repository data may be present, but it is certain to be out of date, so you should only rely on files brought into the project workspace from cache or artifacts.
I'm still a bit confused about how this is supposed to work. If the source code is not guaranteed to exist, then there might be no source in the project workspace and thus the .gitlab-ci.yml file would also be missing. Without a build script the job must fail. If the source is missing only part of the time depending on external factors, the job will fail randomly, which is even worse than failing every time. However, if it fails every single time then what's the point of the feature?
Another possibility I see is that .gitlab-ci.yml might be injected at runtime, so that even without a fresh copy of the repository there would be a build script. If so, could I define further files from my repository to inject into the build process? What are the restrictions on these particular jobs?
Yes, the .gitlab-ci.yml file is not copied onto the system just like all the other files. But that doesn't matter as the job is not run from the file. The job is run as a script on your target (and even before that as it defines the target it will run on). It is not possible to copy only selected files without a git clone although you may want to copy the files from some other server.
A good example of when you want to run GIT_STRATEGY: none are things like slackchat notifications as last stage of a build when you really don't want to clone gigabytes of repository data just to push a notification.

Auto increment appx package version after each build

I am looking for a solution to automatically increment a package version (not to be confused with an assembly version) after each build on CI server (particularly Atlassian Bamboo). Every appx package has a version defined in its manifest file (appxmanifest). Thus in order to increase the version a manifest must be edited before commit. I am considering different approaches to implement this. The first one makes changes in manifest and pushes it back to the repo.
Starts building a plan (in order to lock a build number)
Modifies manifest so that a revision is set to the current build number
Pushes changes to SCM (particularly Atlassian Stash). This step shouldn't trigger the next build.
Continues building the package (invoke MSBuild, UT and other tasks)
Cons
Leads to incorrect workflow on Bamboo: checkout -> push -> build
Each build makes a new commit
Another approach is to setup post receive Stash hook which would modify appxmanifest.
Cons Hard to keep a build number in sync with Bamboo.
Is there any other (cleaner and proper) way to achieve this?
ex-Stash developer here (not that it matters),
I would highly recommend not checking in derived/version information or files. It's going to cause you no end of problems (some of which you have pointed out in your question).
My advice - generate what information you need on the build. I don't know anything about appx packaging, but can you use a placeholder/property (like this) which can be resolved on the Bamboo build? For our builds we use the git hash and timestamp as the version, and in the past I've also used the job/build number (timestamp is better though).
As more food for thought - if that appx version is important for developers to see locally, and it becomes hard to match up with the Git version then you can also attach a Git tag/note to the commit in Bamboo as well. The nice thing about that is that anyone fetching from Git can easily see that extra metadata, but it doesn't result in extra commits for every build. If the appx version need to be based off the previous version then this makes it possible for the build scripts to inspect the previous commit and bump the version appropriately.
I hope that helps.

QuickBuild: How can I create a builder to open a tarball package (tar.gz) whose name will change with each version?

I'm using PMEase QuickBuild to perform automated builds of our Maven2 projects and a nightly sanity test to ensure nothing is broken.
The test needs to untar packages which are created by the automated Maven2 projects. The problem is that the package names change frequently due to project versions being incremented all the time.
Does anyone know how I can configure QuickBuild to pick up the version (ideally from the POM file of the individual components), if this is possible at all?
I don't know if this is an option for you but it looks like you can do it the other way around. Quoting Build with Maven:
Control build version
If you want to control the build
version from QuickBuild side, please
follow below steps:
Change the POM file and define the project version as
${buildVersion}. Do not forget to
commit the file into your SCM after
change.
Define a build property like below when define the Maven build
step:
buildVersion=${build.version}
There are maybe other options but I must admit that my knowledge (zero) of QuickBuild is very limited
I created a work around to this issue by having QuickBuild execute a shell script which did the untarring by using wildcards, similar to the following (to avoid computing the exact version):
tar xzf filename-*.tar.gz
I couldn't figure out how to do this in QuickBuild, so I offloaded the work to the shell script.