I have project that is using npm. Some of the dependencies exist in a registry that can only be accessed over a VPN. Getting on the VPN can be onerous for developers and build machines so we don't want to require people to be on it while running npm install.
In order to solve this, I had created a directory called node_modules_local. I installed the VPN-specific dependencies while connected to the VPN then moved them to the node_modules_local folder. I updated package.json look like this:
"dependencies": {
"core-js": "^2.6.5",
"dep_1": "file:node_modules_local/dep_1",
"dep_2": "file:node_modules_local/dep_2",
...
}
After that, running npm install installed everything without requiring VPN access. It seems to work, but is this a valid approach?
I'm using node 8.7.0 and npm 5.4.2 in this case.
My team and I use Yarn for this because it's easy to set up an offline mirror.
Pull down your packages into some directory (e.g. #angular-core-7.2.0.tgz)
Set up an offline mirror yarn config set yarn-offline-mirror C:\my\local\repo
Run yarn install --offline --freeze-lockfile to download the dependencies from the offline mirror
You may need to run yarn install --offline to generate a yarn.lock file before running with the --freeze-lockfile option.
Related
I have a project that I'm working on for a client where I have two private packages (which I can't get access to npm install) are inside the package.json.
I do however have access to clone the repos for those said packages. If I simply run an npm install I'll get a permission denied error. Same if I run npm link to the packages.
I've been working around this by removing the packages from the package.json then running npm install ../some-package. This works but isn't a great solution because if I wanted to add a new package I'd have to deal with a bit of a mess with the package.json.
Is there a better way than this?
I have tried running npm link ../some-package but I still get access denied. The only way I've managed to complete an install is by removing the packages then installing them from a local dir.
I don't know the details of your situation, but I see at least two potential solutions to explore.
Option 1: Install the package from the repo
I do however have access to clone the repos for those said packages.
You can install from a git repo and package.json will record that git repo as the source of the package rather than the npm registry.
From the docs at https://docs.npmjs.com/cli/v8/commands/npm-install:
npm install :
Installs the package from the hosted git provider, cloning it with git. For a full git remote url, only that URL will be attempted.
Option 2: Install from the local file system with --no-save
If that approach doesn't work for you, you can try npm install --no-save ../some-package as a build step. The --no-save makes it so it doesn't modify package.json.
I'm more familiar with the Python ecosystem at this point and have a question about how I can do something with npm that I'm used to doing with pip.
Let's say I have a wheel for a particular Python package, as well as a wheel file for each of the Python package's dependencies. And let's say I have all these wheel files in a folder called /path/to/wheel/files. To install this package and all of its dependencies, I could run something like pip install /path/to/wheel/files/*.whl --no-deps, where --no-deps keeps me from having to install the various dependencies in the proper order.
Does npm have an equivalent to this? I'm using npm-offline-packager to create a tarball that contains a Node package (as its own tarball) and all of its dependencies (as their own tarballs). I know I can tell npm install to install a particular tarball. However, when I do this, it tries pulling in the required dependencies from the online NPM registry instead of pulling in the dependencies from the tarballs I already have.
Ideally, I'd like npm install to use the tarballs to add the main package to my project's package.json while adding the package's dependencies to my project's package-lock.json. And of course, I'd also like the main package and all its dependencies to be installed to my project's node_modules directory as well.
TL;DR Does npm have something equivalent to pip install /path/to/wheel/files/*.whl --no-deps?
I'm responding to my own question here, but note that my answer is only applicable to my particular use case and may not be applicable in general.
For my use case, I have access to two computers: one that has access to the internet and one that doesn't. For the machine that doesn't have access to the internet, I was attempting to use Verdaccio as a way of creating a self-hosted NPM registry. However, publishing packages to Verdaccio wasn't working because it kept trying to pull in the package's dependencies from the public NPM repository. The solution was to remove all references to "npmjs" in Verdaccio's config file (which, for me, Verdaccio created at ~/.config/verdaccio/config.yaml).
So, in case anyone needs to do development on a machine that doesn't have access to the internet, the process for setting up Verdaccio looks something like this:
On the machine that has access to the internet, create an NPM project using npm init (I called my project "verdaccio_runner"). The reason I did this is because, without already having an NPM registry on the machine that doesn't have access to the internet, it was hard doing a global install of Verdaccio.
Run npm install verdaccio to install Verdaccio to the NPM project that was created in the previous step.
Transfer this project over to the machine that doesn't have access to the internet.
Once it's transferred over, run Verdaccio from the project like this: npx verdaccio.
Quit out of Verdaccio.
Remove all references to "npmjs" from the config file that Verdaccio created (again, mine was at ~/.config/verdaccio/config.yaml).
Run Verdaccio again to pull in those changes.
Tell NPM where your private registry is: npm config set registry http://localhost:4873/.
Add yourself as a user by running npm adduser and by then filling out the information you're prompted for.
And the process for publishing packages to Verdaccio on a machine that doesn't have access to the internet looks like this:
For the package you want to install, on the machine that has access to the internet, run npo fetch <package name> --no-cache (assuming you've already done a global install of npm-offline-packager on the machine that has internet access).
Bring the tarball that npo created for you over to the machine that doesn't have internet access.
Untar the tarball.
From the directory that's created, run for file in ./*.tgz; do npm publish $file; done.
The published packages can now be npm installed to projects on the machine that doesn't have internet access.
Note: in order for Verdaccio to be accessible to other machines on the private network, I also had to add the following to Verdaccio's config file:
listen:
0.0.0.0:4873
My project references mocha, phantomjs, etc, which takes a lot of time to download during npm install. This is not a problem in my local machine because I only download them once and can use them forever unless I decide to manually upgrade them.
However, in my CI machine, my jenkins server need to download them every time that I did a git commit and git push to do the testing and deploy.
So can I just speed up that process by set the npm not to download these slow packages from the remote server? Rather, install them from local cache or not to install them if I installed them globally?
Anyone knows how to configure that?
I found some packages that might be helpful
npm-install-changed will run npm install only if the contents of package.json's devDependencies and dependencies were changed, note that it assumes that node_modules persists across different builds which might not be helpful if your CI server always start from scratch
npm-install-cache runs npm install and then copies your current node_modules folder (to somewhere in \tmp), if you call the script again it will verify any changes to package.json (instead of changes done on devDependencies or dependencies), if it didn't change then it will copy the node_modules folder stored in \tmp, the only limitation I see is that it's not cross platform and that the cache folder is \tmp which is erased on reboot (or maybe even when a is process finished!)
The second package might not work as it is but it seems like a good place to start :)
You can specify all of the packages you want to use locally in devDependencies in package.json, and then running npm install -d will install those instead of the main dependencies.
When I deploy my app to AWS, it's copied into a new directory, so NPM will install all the same packages, during each deploy, which can take a lot of time. Most of these packages haven't changed between builds (if at all), so having it do a full npm-install seems like a waste.
My app server runs a bunch of different Node apps, so installing globally isn't an option. Instead I'd like to have the app store it's node packages in a location that isn't wiped out during deployment, but have the option to update packages as necessary during npm install.
Does NPM have a concept of an app-specific module directory that isn't located in a subfolder of an app? That way I can delete the app folder, and not have to reinstall the same packages over and over again.
I could achieve this by using symlinks, or migrating the current node_module directory.
If you lock down your dependencies versions, NPM is likely to cache the packages. So the installation wouldn't take much longer.
If you prefer not to do this, you can install dependencies globally and link them with the npm link command (which is basically creating a symlink yourself!). Then, it'll be up to you update the globally installed packages regularly.
I tried to maintain package.json with the list of node modules dependencies. when I call npm install it installs the node modules.and generates a folder for it in my app. I call the npm shrinkwrap. But this generates the dependency on the local node module
"dependencies": {
"async": {
"version": "0.2.5",
"from": "async#0.2.5",
"resolved": "https://registry.npmjs.org/async/-/async-0.2.5.tgz"
},
when I upload the app to the appfog server it can install from the npm-shrinkwrap.json. So Ideally I want to remove the node modules folder and just pass the shrinkwrap.json file. But it has this "from". I had in the past generated the shrinkwrap & it didn't have the "from" field in there.
How to generate without "from"/ can I just get a shrinkwrap file from package.json. so my app will be leaner. I can maintain all the node module globally.
Thanks
I'm a bit confused by your question.
Shrinkwrap does not install, package, upload or do anything to your dependencies.
All it does is scan your installed node_modules and record the versions (recursively) into a file. Invoking npm install after that file is defined becomes repeatable, which is a principle of software engineering.
"from" was introduced a few months back. The npm shrinkwrap command seems to set it to the URL from which a module was installed. This is probably for portability. npm install takes a module name, consults a registry (whose URL is configurable as an npm config setting) and installs it. I could take the same package.json and npm-shrinkwrap.json, put them on another machine and theoretically get a different result if that machine's npm config settings point it to a different registry. Therefore, embedding the resolved URL in the shrinkwrap file adds an additional level of repeatability to npm install
See the npm config man page for details of setting the registry parameters.
According to npm issue 3145 on github, the "from" setting is known to cause backwards-compatibility issues with pre-1.2.x npm systems. Upgrading is the only resolution.
https://github.com/isaacs/npm/issues/3145
I think that you are looking for shrinkpack: https://www.npmjs.com/package/shrinkpack
from the doc:
Shrinkpack complements the npm shrinkwrap command by maintaining a node_shrinkwrap directory in your project, containing the exact same tarballs that npm install downloads from https://registry.npmjs.org.
The rest of the npm install process is exactly the same. The only difference is that no network activity is necessary when installing and building your project. The node_shrinkwrap directory can be ignored in your editor (much like is done with the node_modules directory) but is instead checked into source control.