So, we have Project A that has a develop and release branch. We reference a package from another one of our projects, Project B, which also has develop and release branches. What I'd like to be able to do is check that if I'm building / deploying Project A from it's release branch through Azure Devops that it is referencing the package that was created from the release branch for Project B.
My initial thoughts were to look into npm dist tags: https://docs.npmjs.com/cli/dist-tag but if I understand this correctly this doesn't guarantee that the package has ACTUALLY been published from that branch i.e. I could be on my own branch and simply publish the package with a tag of 'release'. Is there a way to automatically add a dist tag if you publish from a specified branch?
My next issue, which may well need to be another question, is whether there is a way in Azure devops to check against this package to ensure it has been published from the correct branch and if not then fail the build. So, if dist tags were used is there a way to check against a package dist tag as a part of the build to ensure it had the right tag e.g. 'release'?
For first question. you can automatically publish and add dist-tag to your package with azure devops build pipeline, which means you need to create a build pipeline to publish your package. And you can add a powershell task to guarantee that the package is published from a certain branch.
To add a powershell task run an inline script before npm publish task.
$sourceBranch = "$(Build.SourceBranchName)"
if($sourceBranch -ne "release")
{
exit 1
}
Above script checks if the sourceBranch is release branch. If the source Branch is not from release branch it will fail the task.
To publish your npm task with tag information. You can add a npm task to run custom command as shown in below pic. You can dynamic defined your tag name using the variables.(eg. publish --tag $(build.Build.SourceBranchName))
Make sure you enable CI build for this build pipeline. And the trigger is release branch
With above steps, your package will be automatically published with your self-defined tag.
2:
For your second question, to check against a package dist-tag in you build pipeline. You can also add a powershell task to check the tag information of your package and fail the task if the condition fails.
set-location -Path "$(Build.SourcesDirectory)"
$tags = npm view packageName dist-tags
if( $tags.Contains("release") -and ($tags -ne $null))
{
exit 1
}
Above script get the tag information using command npm view packageName dist-tags. and check if the tag contains "release". And the task will fail if the condition falses.
The pipeline is triggered by new commits to a master branch and publishes the package
Currently, the version is set manually and I'de be happy to have it set automatically.
What I thought at first was adding the following tasks to the pipeline:
checkout $Build.SourceBranch
run version patch --force
git push
This works and the version is really incremented properly, the thing is, it triggers another run of the pipeline, which will increment again, which will.. you get the idea, endless loop.
Is there a better way for doing that?
I've added the same tasks as mentioned in the question with a small change.
Apparently there is a way to skip the pipeline triggering, see here
So the npm version task looks like this:
version patch -m "Bump version to %s [skip ci]" --force
which prevents the next build from being triggered.
TIP: remember to give the 'author' (Azure DevOps user) permissions to Bypass policies when pushing if any.
Don't use the files in the source repo to keep track of the current/next version. I don't think you can break the loop easily, if you do.
You might be able to get away with running npm --no-git-tag-version version to increment the package.json version inside the build agent without the commit, so that you don't have a change you would have to push back to the origin. It should just change package.json and leave it dirty.
Wait until after the build has succeeded. Use a custom script task to extract the version from package.json, then git reset --hard (there is no reason to keep anything that has changed on the build server). While this will undo the change in package.json, you can now create a tag on the head that contains that version, then do a git push origin {tag-name} which should not introduce a new commit on the origin which would then re-trigger your pipeline.
Actually, I don't think a pipeline will be triggered just because a tag was added, but I honestly haven't tested it. I'm pretty sure it won't.
Sequence of tasks in your pipeline:
[Given a source repo where you have made sure to set the major and minor versions in package.json to reflect the current state of the code according to the rules of Semantic Versioning, and leave the patch value always set to 0, and, if desired, using pre-* values to describe the quality of the major/minor value:]
(This is not really a task, but just describing the start of the execution:) The job starts. Automatically, the code is pulled from the source repo into the build agent.
Using a utility command-line task and code or script that you write, run git describe --tags to find the most recent tag with a tagname matching the pattern you use (see below). (I prefer this sequence over calling npm version from-git because npm will just use the latest tag, which might not be a version number, depending on how much control you have over the branch.) Use string or regular expression operations to extract the major, minor, patch, and whatever pre-* value you might have. This is the previous version that we'll use to compare with what's in the package.json file. Note that you may have to follow these instructions to run a git command. Save it to a pipeline variable.
Using a utility command-line task and code or script that you write, run npm version to get the current major/minor/pre version out of the package.json file and save it to a different pipeline variable. I'm using PowerShell Core, so my command would look something like this to create a "currentPackageVersion" pipeline variable:
& npm.cmd version | ConvertFrom-Json | Select-Object -ExpandProperty {name-of-your-package} | Set-Variable -Name 'packageVersion' | Write-Output "#vso[task.setvariable variable=currentPackageVersion]$packageVersion"
Using a utility command-line task and code or script that you write, compare the previous version's major, minor, and pre-* values to determine whether any of them have changed. Set a new pipeline variable to reflect whether it has changed or not. I'll use the name "restartVersionPatchNumber", which is true if the current major, minor, or pre-* values are different from the previous version's major, minor, or pre-* values.
The npm Task conditionally runs a custom command: npm --no-git-tag-version version patch, which updates package.json in the build agent but does not commit the change, leaving your Working Area modified (dirty) (which might cause issues on subsequent builds if you are using your own build agents instead of hosted agents). The condition expression of the Task uses a custom condition that evaluates to the variable that I just set in the previous step ("restartVersionPatchNumber"). Note that if this task does not run, it should just use the value of the version that is in the package.json file (the current version that is now in the "currentPackageVersion" pipeline variable).
Using a utility command-line task and code or script that you write, run npm version to extract the new version that the npm version command set. Save it to a new pipeline variable; I'll call it "newVersionNumber".
The regular build tasks run, producing artifacts and possibly publishing them
Using a utility command-line task, run git reset --hard. You'll need to do this even if you're using a hosted build agent, because of the next step.
Using a utility command-line task, create a variable from the saved version number ("newVersionNumber") that contains the value of the tag's tagname that you want to use. Use a distinctive patter, like "AzPipelineBuild-PipelineName-PackageVersion-1.0.0" but using your version instead of 1.0.0.
Using a utility command-line task, run git tag {*tagname*}. For PowerShell, the syntax would be & git.exe tag $env:newVersionNumber
Using a utility command-line task, run git push origin {*tagname*}
Profit
You could (and maybe should) combine the command-line steps. I'm really partial to the PowerShell task running PowerShellCore (pwsh) as you may have guessed.
The package that got created as an artifact of the job will have the updated version as it was created in the build, and it will match the tag that is now in your original source code repo.
Alternately, use an external source (an Azure Function, etc.) or a different (second) git repo or even another branch that isn't tied to your CI trigger in your project that you use just for tracking build numbers, then use token replacement tasks to set the version just before you start the build. I don't like this idea much, but it would prevent re-triggering a new build.
Also, you're hopefully building just one package with that repo. The next thing to do is to publish that package to your Azure Artifacts feed, or npmjs.org or wherever. Don't rely on having the version baked into the original source code; anything that depends on that package should be pulling it out of that feed you published it to, not relying on it being built earlier in the build steps with a new version number.
Here's my implementation in Powershell which is a complete step based on the accepted answer. Note that this also handles some of the git issues such as the detached head that comes with this type of approach.
steps:
- checkout: self
persistCredentials: true
.
.
.
- task: PowerShell#2
displayName: 'Bump the version'
inputs:
targetType: 'inline'
script: |
$BranchName = "$(Build.SourceBranch)" -replace "refs/heads/"
git checkout $BranchName
git config --global user.email "anymail#anycompany.com"
git config --global user.name "Your name"
npm version prerelease -m "Auto increment pre-release version to %s [skip ci]" --force
git push
I have a question about workflow with docker and gitlab-ci or automated builds in general.
This is how I am imagine how a build should look like↓.
How to do it with gitlab-ci ?
I know how to do one of this tasks, but I don't know how to.
In my imagination i would need more than one base image.
Maybe I am missunderstanding the hole thing.
How should this process be done in general ?
Thx four your help 😀
Since your question is very general, I will answer it with an example.
Consider a imaginary C++ project, which contains the code, a Makefile which creates the executable "app" and this Dockerfile:
FROM ubuntu:16.04
ADD ./app /app
CMD ["/app"]
To build the application and the docker image as you said, you could use a GitLab CI config like this:
stages:
- test
- build
- docker
test:
stage: test
script:
- make test
build:
stage: build
script:
- make
artifacts:
paths:
- ./app
docker:
stage: docker
dependencies:
- build
script:
- docker build -t your-repo/image-name:latest .
- docker push your-repo/image-name:latest
Explanation
This CI file creates three jobs: "test", "build" and "docker". "test" runs "make test" to execute any imaginary tests our codebase might have. If they suceed, the GitLab runner will execute the next job, "build".
"build" builds the application by calling "make". We expect make to create a file "app" in the current directory, which is our compiled application that will run in the container. The section "artifacts" states that we want to keep this resulting file, since we need it for the next job.
The next job "docker" has a section "dependencies"; in this section we state that this job depends on the output of the job called "build", which created our file "app" before. Then we first build the docker image using docker build and push it as usual.
As said before, these are just examples, and especially the script sections will greatly differ based on your projects and your runner config. See the official CI documentation for all possibilities.
I have a simple Gitlab pipeline setup with two stages: build & test. Both stages are supposed to share cached files but they don't appear to, resulting in the test stage failing. As best I can, the problem is that each stage uses a different runner and the cached files use the runner ID as part of the path.
.gitlab-ci.ym
...
cache:
key: "build"
untracked: true
...
The build stage outputs the following
Creating cache build...
untracked: found 787 files
Uploading cache.zip to https://runners-cache-1.gitlab.com:443/runner/runner/30dcea4b/project/1704442/build
The test stage outputs the following
Checking cache for build...
$ mvn test
I believe this means the cache was NOT found because there is no download information; but it's not clear.
I can also see that each stage uses a different runner and since the runner ID is part of the cache path, I suspect that is the problem.
I need to either use the same runner for each stage or share the cache across runners. I don't understand how to do either.
Any help would be appreciated.
It appears the cache feature is appropriately named, it's only for improving build performance and is not guaranteed to have the data, much like a real cache.
The correct approach is to use artifacts with dependencies.
I have a build configuration containing two build steps "Build" and "Deploy".
This build configuration is using a Git VCS and is configured to run all branches.
Let's assume that I currently have two branches "master" and "Feature in Development".
I want to divide the "Deploy" step into two. "Deploy Production" and "Deploy Beta".
The "Deploy Production" step should only be executed when the master branch was changed and is currently building.
Is there a built in possibility to do that or do I have to check the %teamcity.build.branch.is_default% variable manually within the script?
It seems that JetBrains added this feature in TeamCity v9.1.
For v9.0 I used a Powershell script and checked '%teamcity.build.branch.is_default%' -eq 'true' build property.
You can add a condition to a build step in TC using the dropdown:
Why not create a two build configuration for that, one checking code from master, second from branch? IMHO it should not be steps in the same build as they are not chained.