On our CI we continuously build UWP software which we also have to install on the CI clients to execute some smoke or UI tests.
Now in the process of speeding things up I noticed that some portion of time is essentially wasted by installing / uninstalling dependencies, e.g.
...\x86\Microsoft.NET.CoreFramework.Debug.2.2.appx
...\x86\Microsoft.NET.CoreRuntime.2.2.appx
...\x86\Microsoft.VCLibs.x86.Debug.14.00.appx
...\x64\Microsoft.NET.CoreFramework.Debug.2.2.appx
...\x64\Microsoft.NET.CoreRuntime.2.2.appx
...\x64\Microsoft.VCLibs.x64.Debug.14.00.appx
(don't mind debug, it's a test). What is unwelcome here is that we never deploy x86 but spent time on installing dependencies for it.
I am looking therefore for a way to skip the uninstall of the dependencies on our CI to speed at least this part up (which would also mitigate the installation of the unused x86 dependencies).
For some yet undiagnosed reason installing a dev packages takes about 30 seconds and doing this for 10 apps one by one slows things down.
My questions are
Why does my configuration install x86 when I only package x64?
Does somebody know how to avoid uninstalling implicit dependencies for an app in order to avoid installing them over and over?
Any further hints on how to speed this up?
By our testing, we packaged the app into app bundle and then we deleted the dependencies in the dependencies folder under package folder. After that, we installed the app, it will prompt a message showing we need to install those dependencies. So we installed the dependencies manually and installed the app again, it worked this time. At last we uninstall the app and tried to re-install it, it also worked well. So you can also try this way-manually install the dependencies to avoid installing dependencies over and over.
Related
I am currently staying at a location where internet and disk space are at a premium and yarn/npm constantly having to install module every single time isn't the most efficient use of both my disk space and internet data which bring me to my question,
I recently came across pnpm and it perfectly solves my problem (it install modules in a central location and symlink them to your projects), my question is this if i completely migrate to pnpm will that affect the project setup if i was working with someone using yarn/ npm for instance and if i publish a project will the users be forced to use pnpm or they can use any package manager
It is recommended to pick one package manager for a project and force its usage by everyone. pnpm, yarn, npm all have their own lockfiles.
If it is a small project and you don't commit the lockfile, then you may use different package managers. But I still don't recommend this.
A colleague and I are both working with serverless, both on windows 10, with the latest serverless version of 1.37.0 with the same npm version of 6.70.
Yet running the serverless command, his computer bundles up the node modules folder to upload a package of on average 8-9 MB.. yet mine bundles to 20Mb.. i cannot figure out what could be causing this.
Dev dependencies are being excluded so not sure why my laptop is packaging it up so inefficiently compared to his.
Does any one know of any reasons as to why this could be? or any other recommendations?
We are using TFS Build Tasks and one of the tasks runs npm install. This is NOT through a batch or powershell file.
It runs successfully but it looks like it completes and then hangs for about 3 to 4 minutes. I know this because the task summary says it successfully completed but it doesn't start the next task for 3 to 4 minutes.
Originally when I added the task, I don't think it would hang like this. I'm not sure what changed.
I tried using npm set progress=false, recommended from this forum and explained in this article. I haven't added npm-cache because it doesn't seem to be relevant; remember, the build task completes successfully and then hangs.
What could cause the npm task to hang after completing?
According to your description, it is make sense. NPM install is just wasting time because it takes 3-4 minutes to determine the packages are already installed.
Fist try to run you npm from the console to see the performance on TFS is normal or not. If all of your NPM tasks are taking long time, one possibility is related to nodejs version.
For instance, you are using a latest version such as nodejs (8.2.0) installed on the build agent. Then downgrade to the latest LTS (long term support) version (6.11.1) may resolve the issue for you. Details please have a look at this blog.
Another way is using npm-cache as an alternative way since you use on-premise build agents for build.
It is useful for build processes that run [npm|bower|composer|jspm]
install every time as part of their build process. Since dependencies
don't change often, this often means slower build times. npm-cache
helps alleviate this problem by caching previously installed
dependencies on the build machine. npm-cache can be a drop-in
replacement for any build script that runs [npm|bower|composer|jspm]
install.
How it Works
When you run npm-cache install [npm|bower|jspm|composer], it first
looks for package.json, bower.json, or composer.json in the current
working directory depending on which dependency manager is requested.
It then calculates the MD5 hash of the configuration file and looks
for a filed named .tar.gz in the cache directory ($HOME/.package_cache
by default). If the file does not exist, npm-cache uses the system's
installed dependency manager to install the dependencies. Once the
dependencies are installed, npm-cache tars the newly downloaded
dependencies and stores them in the cache directory. The next time
npm-cache runs and sees the same config file, it will find the tarball
in the cache directory and untar the dependencies in the current
working directory.
A sample for your reference: Speed up your npm dependent CI build
I recently got confused and almost installed a tool via brew install when in fact it was an npm package and all I needed to do was npm install -g.
So these tools are strangely similar yet obviously different.
What's the difference in crystal clarity?
NPM exists to resolve dependencies for application code, on a per app basis, allowing an app to be self-contained and portable. This means that (in its default mode of operation) it will install the same stuff many times, uniquely, repeatedly, and separately, for every app on your system that needs the same package, inside of that apps own directory and isolated from everything else.
Homebrew is not like this. The reason is it serves the system itself, not individual apps, so is more comparable to just the npm -g part of npm.
There is one extra bit to understand about homebrew, though - some system packages have specific dependencies and could even have conflicts. This means that for the global installs that homebrew provides, it also has to solve some nesting and conflict issues. It's a kind of magic?
I am building a HTML5 front-end using NPM-based tools (grunt).
One of the first steps of my continuous integration build process is to run an npm install.
npm install is SLOW. Even with a local NPM proxy caching artifacts (Sonatype's Nexus 3), it is still taking 4 minutes!
$> time npm install
real 4m17.427s
user 0m0.170s
sys 0m0.290s
If I follow my usual best practices for continuous integration, I would start from a pristine SCM repository and run the build. This means that each time the CI build will have to do a fresh npm install and take on the cost of 4 minutes.
This is a significant proportion of my build time. I am discontent that the build is taking so long.
The alternative seems to be to keep the node_modules around between builds. However, I've had problems with the build becoming unstable as a result.
Removing dependencies from package.json does not remove them from node_modules with a simple npm install. I can work-around this with an npm prune first.
What is considered to be best practice here?
Since March 5, 2018 and npm 5.7.1, you can use npm ci . This is much faster than a regular npm install because it omits some user-friendly features and installs packages for a userless CI environment.
The caveat here is that you'll need to make sure your package.json and package-lock.json files are in sync. If you install a new package, commit package.json but forget to do the same for package-lock.json, you'll get an error when running npm ci.
Considering that in order to build you must install new packages, you have no choice but to call install. As for pristine, I strongly believe they refer to the "build" process and not the "dependency management" process.
Why are they different? Let's go through an example to make it more apparent.
As a developer, when you first start your job, you MUST "install"
softwares that will enable to code. This is usually done once.
Afterwards, you can start coding. The later is the "build" part
as you are generating value for each feature your code produce. From
time to time, you can update your tool list by removing, adding or updating one.
In this example, installing your tools everyday you arrive at work before starting coding would be hell.
I would suggest you to make sure that the building process, which means producing an artifact (like a Jar for example), is decoupled from the dependency installation process. Meaning that installation is done once and building can proceed without trouble. You don't mention what will be built, but grunt can take care of the rest for sure.
Hence, I believe pruning and installing is a good strategy. You shouldn't worry for the fist times. Think of it as a cold start. Any system implemented with sub components working together as a pipeline have this "issue". Take a car for example. It will not be as fuel efficient when you start it as when you drive it after an hour.
Schedule a daily job to build a docker container with your dependencies. Run your CI job in the latest container. Artifact the CI job's build.
You should install npm packages offline in local machine or local network, you can found some tips here => Offline installation of npm packages
Have you considered using npm link or even symlinking your entire node_modules folder?
At least npm link could be used for your dev dependencies, which you normally want a controlled version of on the server anyway. This should speed things up a bit.