I'm trying to configure my pipeline to cache npm between builds.
My pipeline.yml looks like this
variables:
npm_config_cache: $(Pipeline.Workspace)/.npm
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- task: Cache#2
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
path: '$(npm_config_cache)'
cacheHitVar: 'CACHE_RESTORED'
restoreKeys: 'npm | "$(Agent.OS)"'
displayName: 'Cache npm'
- task: Npm#1
condition: ne(variables.CACHE_RESTORED, 'true')
inputs:
command: 'install'
- task: Npm#1
inputs:
command: 'custom'
customCommand: 'run lint'
displayName: 'Lint checking'
This works fine up until the npm run lint and then it fails.
However the cache key is found and the cache is restored.
The condition line evaluates to false which is also correct.
If I force the npm install then the lint line works.
Any ideas what the difference between restoring the cache and forcing an npm install would be?
Or any ideas of how else to get this to work?
Thanks for the tip - I should have been way more specific.
Basically this doesn't work for anyone else who is trying this.
This isn't the way to cache npm install :) There just doesn't seem to be a good way to do this between builds.
You can cache the .npm folder but that doesn't really speed builds up much if at all - probably due to Azure caching npm registries and high network speeds internal to Azure
Your question is a bit vague, so I got a suggestion. Look at the variables of your pipeline. See the system.debug variable ? Set it to true. Now, Azure will show usually much more info about what happens in your pipeline. If the linting fails, maybe you find more information. Now adjust your question with the new found diagnostics at hand here.
Related
I want to update the package.json version through azure devops pipeline.
This is the task I used. But it throws error.
- task: Npm#1
displayName: "npm version"
command: "custom"
workingDir: src
verbose: false
customCommand: "version $(Build.BuildNumber)"
How to correctly pass the pipeline build number to npm task.
Thanks in advance.
We can use command npm version <newversion> to update the package.json version. for example, current version is v1.0.1, using command npm version 2.0.0 will update its version to v2.0.0. Please note that the version format must be the same, like major.minor.patch.
In addition, yaml pipeline will set the build numer to be formatted like 20210426.5 by default. If your package.json is not formatted like this, this command npm version $(Build.BuildNumber) will cause issues. You could use UpdateBuildNumber to override the automatically generated build number.
BTW, the following yaml pipeline is for your reference supposed that your package.json is formatted like major.minor.patch. Also you could use custom variables to specify the version number.
pool:
vmImage: ubuntu-latest
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Hello World"
git config --global user.email "you#example.com"
git config --global user.name "Your Name"
echo "##vso[build.updatebuildnumber]2.1.0"
- task: Npm#1
inputs:
command: 'custom'
workingDir: ''
customCommand: 'version $(Build.BuildNumber)'
I want to build npm packages via Azure DevOps. My build pipeline fails because peer dependencies are not installed. Is there a way to install the peer dependencies from the package.json?
Below is my sample azure-pipelines.yml file for building and publishing my npm package.
pool:
name: Azure Pipelines
demands: npm
vmImage: 'ubuntu-latest'
steps:
- task: Npm#1
displayName: 'npm install'
inputs:
verbose: false
- task: Npm#1
displayName: 'npm install project'
inputs:
workingDir: 'projects/my-project'
verbose: false
- task: Npm#1
displayName: 'ng build'
inputs:
command: custom
verbose: false
customCommand: 'run ng build -- --prod'
- task: Npm#1
displayName: 'npm publish'
inputs:
command: publish
workingDir: 'dist/my-project'
verbose: false
publishEndpoint: NPM
condition: contains(variables['Build.SourceBranch'], 'master')
Bonus question: How do I apply a tag when publishing?
For this issue ,first, you need to make sure that the Working folder specified in the npm install task contains target package.json.
Secondly, do all the dependent packages you need to use exist in the public feed? If not, you need to use packages from your Azure Artifacts feed.
In addition, you need to make sure that the dependent packages are specified in your package.json. If everything is ok, can you run successfully locally?
Update:
The automatic install of peer dependencies was explicitly removed with npm 3.
Here is the case you can refer to .
I'm using Azure Pipelines with hosted builds to build a web project. Our build times were hitting 10-15 minutes, with most (5-10 minutes) of the time spent doing npm install. To speed this up, I'm trying to use the Cache task (https://learn.microsoft.com/en-us/azure/devops/pipelines/caching/?view=azure-devops).
However, when the auto-added task Post-job: Cache runs, it always errors out with:
##[error]The system cannot find the file specified
The host server is Windows Server 2017.
Here is my entire build YAML
# Node.js with Vue
# Build a Node.js project that uses Vue.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- develop
pool:
name: Default
variables:
FONTAWESOME_NPM_AUTH_TOKEN: $(FONTAWESOME_NPM_AUTH_TOKEN_VARIABLE)
npm_config_cache: $(Pipeline.Workspace)/.npm
steps:
- task: DutchWorkzToolsAllVariables#1
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- task: Cache#2
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
path: $(npm_config_cache)
cacheHitVar: NPM_CACHE_RESTORED
- task: Npm#1
displayName: 'npm install'
inputs:
command: 'install'
condition: ne(variables.NPM_CACHE_RESTORED, 'true')
- task: Npm#1
displayName: 'npm run build'
inputs:
command: 'custom'
customCommand: 'run build'
- task: CopyFiles#2
inputs:
SourceFolder: '$(Build.Repository.LocalPath)\dist'
Contents: '**'
TargetFolder: '$(Build.StagingDirectory)'
CleanTargetFolder: true
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
Cache task output:
Starting: Cache
==============================================================================
Task : Cache
Description : Cache files between runs
Version : 2.0.0
Author : Microsoft Corporation
Help : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
- npm [string]
- "Windows_NT" [string]
- package-lock.json [file] --> F93EFA0B87737CC825F422E1116A9E72DFB5A26F609ADA41CC7F80A039B17299
Resolved to: npm|"Windows_NT"|rbCoKv9PzjbAOWAsH9Pgr3Il2ZhErdZTzV08Qdl3Mz8=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session zzzzz
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `npm|"Windows_NT"|rbCoKv9PzjbAOWAsH9Pgr3Il2ZhErdZTzV08Qdl3Mz8=`
Information, There is a cache miss.
Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session zzzzz
Finishing: Cache
Post-job: Cache output:
Starting: Cache
==============================================================================
Task : Cache
Description : Cache files between runs
Version : 2.0.0
Author : Microsoft Corporation
Help : https://aka.ms/pipeline-caching-docs
==============================================================================
Resolving key:
- npm [string]
- "Windows_NT" [string]
- package-lock.json [file] --> 2F208E865E6510DE6EEAA6DB0CB7F87B323386881F42EB63E18ED1C0D88CA84E
Resolved to: npm|"Windows_NT"|OQo0ApWAY09wL/ZLr6fxlRIZ5qcoTrNLUv1k6i6GO9Q=
Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session zzzzz
Information, Getting a pipeline cache artifact with one of the following fingerprints:
Information, Fingerprint: `npm|"Windows_NT"|OQo0ApWAY09wL/ZLr6fxlRIZ5qcoTrNLUv1k6i6GO9Q=`
Information, There is a cache miss.
Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session zzzzz
##[error]The system cannot find the file specified
Finishing: Cache
How can I fix my build definition so the caching works?
#Levi Lu-MSFT was right in his comment but there's a gotcha.
#FLabranche has a working solution in his answer but I believe reasoning is not quite right.
The problem
npm install and #Cache task are looking for the npm cache at different locations. Consider the flow when pipeline runs for the first time:
#Cache task: does nothing since there's no cache yet.
npm i (or npm ci) task: installs packages in node_modules/ and updates the npm cache at default location. Default location is ~/.npm on Linux/Mac and %AppData%/npm-cache on Windows. On Linux hosted cloud agent the absolute path will be /home/vsts/.npm.
(... more tasks from your pipeline)
Post-job #Cache task (added implicitly): reads the npm cache found at user-provided location to store it for future reuse. User-provided location is set by the npm_config_cache: $(Pipeline.Workspace)/.npm variable. On Linux hosted cloud agent the absolute path will be /home/vsts/work/1/.npm.
As a result, #Cache task fails with tar: /home/vsts/work/1/.npm: Cannot open: No such file or directory.
Solution
Make npm install and #Cache task use the same npm cache location.
One option suggested by Levi Lu is to update the npm config with npm config set cache $(npm_config_cache) --global but it won't work in the pipeline (at least it didn't work for me in an Azure-hosted Linux agent): Error: EACCES: permission denied, open '/usr/local/etc/npmrc'
npm ci --cache $(npm_config_cache) updates the npm cache location for a single call and it does work in this case. It feels a bit hacky though since --cache option is not even documented on the npm website.
All in all this code worked for me:
variables:
NPM_CACHE_FOLDER: $(Pipeline.Workspace)/.npm
steps:
- task: Cache#2
displayName: Cache npm dependencies
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
restoreKeys: |
npm | "$(Agent.OS)"
npm
path: $(NPM_CACHE_FOLDER)
- script: npm ci --cache $(NPM_CACHE_FOLDER)
displayName: 'Install npm dependencies'
...
You can log into your Windows Server 2017 server and check if the folder $(Pipeline.Workspace)/.npm is created and the dependencies are stored inside.
I copied and tested your yaml. It worked both on local agent(win2019) and cloud agents. You can try to run your pipeline on the cloud agents or other agents with newer system to check if it is the agent that cause this error.
The keys generated with your package-lock.json differ between the two tasks.
It happens when the file is modified. Here, they're modified by your npm install task.
You can use the restoreKeys option when configuring the Cache task to fall back onto the latest cache entry.
And I think you don't need the 'npm install' task.
Could you try replacing this :
- task: Cache#2
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
path: $(npm_config_cache)
cacheHitVar: NPM_CACHE_RESTORED
- task: Npm#1
displayName: 'npm install'
inputs:
command: 'install'
condition: ne(variables.NPM_CACHE_RESTORED, 'true')
By this definition :
- task: Cache#2
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
restoreKeys: |
npm | "$(Agent.OS)"
npm
path: $(npm_config_cache)
displayName: Cache npm
- script: npm ci --cache $(npm_config_cache)
Yesterday, I was able to get it working with no issue at all on a self-hosted machine agent by using this:
- task: Cache#2
inputs:
key: '**/package-lock.json, !**/node_modules/**/package-lock.json, !**/.*/**/package-lock.json'
path: '$(System.DefaultWorkingDirectory)/node_modules'
displayName: 'Cache Node Modules'
Today, trying to work on a hosted agent today and this doesn't cut it at all. Aggh, Back to the grinding board. Anyhow, maybe could work for you on your self-hosted pipeline
This seems to be related to this open issue.
I have resolved the problem by switching the build agent pool to hosted and using windows-latest image.
pool:
vmImage: 'windows-latest'
I have configured a private agent in VSTS and have installed NPM there globally. When I'm trying to install NPM through my build task, it is still installing NPM packages for every build which is taking an aweful lot of time- approximately 12 minutes.
How can I cache the NPM installations so that the build time is reduced?
We use npm-cache, npm-cache is a node module that will calculate a hash of your package.json file for every hash it will create zip folder on your build server with the content of node_modules, now npm install is reduced to extracting a zip on every build (of course only in case you didn’t actually change package.json).
The idea is: in the first time the tool download the npm packages and save them locally, in the second time if the package.json not changed he takes the packages from the local disk and copy them to the build agent folder, only if the package.json changed he downloads the packages from the internet.
Install the npm-cache on the build machine:
npm install npm-cache -g
In the build definition add Command Line task (Tool: C:\Windows\User\AppData\Roaming\npm\npm-cache (or just npm-cache if you add the tool to environment path variables); Arguments:install npm; Working folder: $(Build.SourcesDirectory) (or where package.json located).
MS has finally implemented this feature (currently in beta) https://learn.microsoft.com/en-us/azure/devops/pipelines/caching/index?view=azure-devops#nodejsnpm
From there:
variables:
npm_config_cache: $(Pipeline.Workspace)/.npm
steps:
- task: CacheBeta#0
inputs:
key: $(Build.SourcesDirectory)/package-lock.json
path: $(npm_config_cache)
displayName: Cache npm
- script: npm ci
Unfortunately we cannot cache the NPM installations as no such a built-in feature for now.
However there's already a user voice submitted to suggest the feature : Improve hosted build agent performance with build caches, and seems the VSTS team are actively working on this now...
For now, you can try to speed Up NPM INSTALL on Visual Studio Team Services
Use Cache task
Caching is added to a pipeline using the Cache pipeline task. This
task works like any other task and is added to the steps section of a
job
With the following configuration:
pool:
name: Azure Pipelines
steps:
- task: Cache#2
inputs:
key: 'YOUR_WEB_DIR/package.json'
path: 'YOUR_WEB_DIR/node_modules/'
- task: Npm#1
inputs:
command: 'install'
workingDir: 'YOUR_WEB_DIR/frontend'
You can use key YOUR_WEB_DIR/package-lock.json too, but be aware that file might be changed by other next step like npm install so hash also will be changed.
I'm using Gitlab CI 8.0 with gitlab-ci-multi-runner 0.6.0. I have a .gitlab-ci.yml file similar to the following:
before_script:
- npm install
server_tests:
script: mocha
client_tests:
script: karma start karma.conf.js
This works but it means the dependencies are installed independently before each test job. For a large project with many dependencies this adds a considerable overhead.
In Jenkins I would use one job to install dependencies then TAR them up and create a build artefact which is then copied to downstream jobs. Would something similar work with Gitlab CI? Is there a recommended approach?
Update: I now recommend using artifacts with a short expire_in. This is superior to cache because it only has to write the artifact once per pipeline whereas the cache is updated after every job. Also the cache is per runner so if you run your jobs in parallel on multiple runners it's not guaranteed to be populated, unlike artifacts which are stored centrally.
Gitlab CI 8.2 adds runner caching which lets you reuse files between builds. However I've found this to be very slow.
Instead I've implemented my own caching system using a bit of shell scripting:
before_script:
# unique hash of required dependencies
- PACKAGE_HASH=($(md5sum package.json))
# path to cache file
- DEPS_CACHE=/tmp/dependencies_${PACKAGE_HASH}.tar.gz
# Check if cache file exists and if not, create it
- if [ -f $DEPS_CACHE ];
then
tar zxf $DEPS_CACHE;
else
npm install --quiet;
tar zcf - ./node_modules > $DEPS_CACHE;
fi
This will run before every job in your .gitlab-ci.yml and only install your dependencies if package.json has changed or the cache file is missing (e.g. first run, or file was manually deleted). Note that if you have several runners on different servers, they will each have their own cache file.
You may want to clear out the cache file on a regular basis in order to get the latest dependencies. We do this with the following cron entry:
#daily find /tmp/dependencies_* -mtime +1 -type f -delete
EDIT: This solution was recommended in 2016. In 2021, you might consider the caching docs instead.
A better approach these days is to make use of artifacts.
In the following example, the node_modules/ directory is immediately available to the lint job once the build stage has completed successfully.
build:
stage: build
script:
- npm install -q
- npm run build
artifacts:
paths:
- node_modules/
expire_in: 1 week
lint:
stage: test
script:
- npm run lint
From docs:
cache: Use for temporary storage for project dependencies. Not useful for keeping intermediate build results, like jar or apk files. Cache was designed to be used to speed up invocations of subsequent runs of a given job, by keeping things like dependencies (e.g., npm packages, Go vendor packages, etc.) so they don’t have to be re-fetched from the public internet. While the cache can be abused to pass intermediate build results between stages, there may be cases where artifacts are a better fit.
artifacts: Use for stage results that will be passed between stages. Artifacts were designed to upload some compiled/generated bits of the build, and they can be fetched by any number of concurrent Runners. They are guaranteed to be available and are there to pass data between jobs. They are also exposed to be downloaded from the UI. Artifacts can only exist in directories relative to the build directory and specifying paths which don’t comply to this rule trigger an unintuitive and illogical error message (an enhancement is discussed at https://gitlab.com/gitlab-org/gitlab-ce/issues/15530 ). Artifacts need to be uploaded to the GitLab instance (not only the GitLab runner) before the next stage job(s) can start, so you need to evaluate carefully whether your bandwidth allows you to profit from parallelization with stages and shared artifacts before investing time in changes to the setup.
So, I use cache. When don't need to update de cache (eg. build folder in a test job), I use policy: pull (see here).
I prefer use cache because removes files when pipeline finished.
Example
image: node
stages:
- install
- test
- compile
cache:
key: modules
paths:
- node_modules/
install:modules:
stage: install
cache:
key: modules
paths:
- node_modules/
after_script:
- node -v && npm -v
script:
- npm i
test:
stage: test
cache:
key: modules
paths:
- node_modules/
policy: pull
before_script:
- node -v && npm -v
script:
- npm run test
compile:
stage: compile
cache:
key: modules
paths:
- node_modules/
policy: pull
script:
- npm run build
I think it´s not recommended because all jobs of the same stage could be executed in parallel.
First all jobs of build are executed in parallel.
If all jobs of build succeeds, the test jobs are executed in parallel.
If all jobs of test succeeds, the deploy jobs are executed in parallel.
If all jobs of deploy succeeds, the commit is marked as success.
If any of the previous jobs fails, the commit is marked as failed and no jobs of further stage are executed.
I have read that here:
http://doc.gitlab.com/ci/yaml/README.html
Solved a problem with a symbolic link to a folder outside the working directory. The solution looks like this:
//.gitlab-ci.yml
before_script:
- New-Item -ItemType SymbolicLink -Path ".\node_modules" -Target "C:\GitLab-Runner\cache\node_modules"
- yarn
after_script:
- (Get-Item ".\node_modules").Delete()
I know this is a enough dirty solution but it saves a lot of time for build process and extends the storage life.
GitLab introduced caching to avoid redownloading dependencies for each job.
The following Node.js example is inspired from the caching documentation.
image: node:latest
# Cache modules in between jobs
cache:
key: $CI_COMMIT_REF_SLUG
paths:
- .npm/
before_script:
- npm ci --cache .npm --prefer-offline
server_tests:
script: mocha
client_tests:
script: karma start karma.conf.js
Note that the example uses npm ci. This command is like npm install, but designed to be used in automated environments. You can read more about npm ci in the documentation and the command line arguments you can pass.
For further information, check Caching in GitLab CI/CD and the cache keyword reference.