I've got an irritating little bug with AWS, where I create my workspace (in this case, npm install) and distribute it to a bunch of slaves. Part of the install lifecycle for npm creates a bunch of symlinks in ./node_modules/.bin. Unfortunately, S3 doesn't support symlinks (see this question).
Now, it's a bit unfortunate that I'm downloading a prebuilt ./node_modules from S3, but it's how it's gotta be done (outside the scope of the question). When I run npm install on the slave, node doesn't recreate the symlinks.
I could always add a pretest hook to recreate the symlinks manually, but that's what got me wondering: is there a way to invoke specific parts of the npm lifecycle manually? If not, why? Running only parts of the install could be useful -- at least, here.
Related
In Ubuntu, I have a directory with my code /code which has my packages.json and such. My NPM prefix is /usr via npm prefix -g. I am trying to install packages globally so as to not create a node_modules directory in /code, but really as long as node_modules is not in /code, that will also work.
While in /code, I run npm install -g. I see a .staging directory created under my global prefix (eg, /usr/lib/node_modules/.staging) but then NPM takes another step and begins creating a /code/node_modules directory.
If I instead just run npm install -g <package> it correctly ends up in /usr/lib/node_modules/<package>. But I do not want to have to run npm install -g <package> for every package I need.
Why is NPM doing that and how can I make it stop doing that? At the very least, I would like the node_modules directory to NOT be in my /code directory (this is due to some environment constraints I can't change).
Globally installed packages are (almost always) not going to be used by your code. That's not how node or npm (conventionally) works. There isn't a shared global node_modules that programs all look in. (OK, there kinda is, but you don't want to use it if you can avoid it. Read on.) Having a local node_modules helps you avoid dependency hell. Global modules are for CLI tools that end up in your PATH and things like that.
You don't explain why you don't want a local node_modules for your project, so I will mention that if it is simply because it somehow bothers you, then please please please just get over it. You will be vastly happier. Type npm install and move on.
That said, for legacy reasons, your code in /code will look in /node_modules if there is no /code/node_modules. This is true for CommonJS modules that use require(). If you are using ESM modules via import, you may need to read the docs.
So, if you are using CommonJS/require(), you could have a directory structure like /project-name/code. You could have your index.js or whatever other code in the code subdirectory, while your package.json could be in project-name. Run npm install from /project-name and it will create a node_modules directory there rather than in the code subdirectory, and your Node.js files in code will find those modules.
If you don't like that, then for other options, you can review the relevant docs. At the time of this writing, those options include a NODE_PATH environment variable, $HOME/.node_modules, $HOME/.node_libraries, and $PREFIX/lib/node. However it all comes with this caveat that I implore you to abide by if you can:
It is strongly encouraged to place dependencies in the local node_modules folder. These will be loaded faster, and more reliably.
Context
I have a .NET project and we use some npm packages for the UI. I use a pre-built check if the folder node_modules does not exist, run the command npm install.
When committing an update to package.json, npm install will not trigger for other people who are updating the repository. Because node_modules exists on their machine which leads to errors like Can't resolve....
Question(s)
Is the folder check obsolete? Is npm install smart enough to download only the necessary things and not all the dependencies? Or do i need a hash check on package.json?
There's no harm in running npm install multiple times as it will simply not do any operation in case there's nothing to do.
You don't need to check for the node_modules folder. npm will download and update any dependencies that are missing.
It's also important to run npm install since other machines may be running different systems and the dependencies may need to be compiled differently.
You can hash check package.json and/or package-lock.json for caching purposes, but it's not really necessary.
TLDR: How reliable is an assumption that a compatible version of make is installed on machines that my NPM package will be installed on?
I'm working for a client who is really fond of make and Makefiles. We need to integrate a JS package into another project. My go-to method would be npm install our-package. The package requires a build step. We want to be able to install versions that are not yet released to npm.js from GitHub. The build process is more than one line of shell script so I don't want to repeat it in Makefile and in package.json (it will go out of sync real quick). Straight forward way is to set scrpits.prepare to make in packgae.json. But I'm concerned about cross-platform compatibility (Windows, containers, etc).
Update: I've also discovered that it's extremely easy to create an endless loop where make calls npm ci or npm install and then npm calls make. With npm install there is --ignore-scripts option, but ci doesn't have an equivalent, see: https://npm.community/t/add-ignore-scripts-option-to-clean-install-ci-command/6322
My project references mocha, phantomjs, etc, which takes a lot of time to download during npm install. This is not a problem in my local machine because I only download them once and can use them forever unless I decide to manually upgrade them.
However, in my CI machine, my jenkins server need to download them every time that I did a git commit and git push to do the testing and deploy.
So can I just speed up that process by set the npm not to download these slow packages from the remote server? Rather, install them from local cache or not to install them if I installed them globally?
Anyone knows how to configure that?
I found some packages that might be helpful
npm-install-changed will run npm install only if the contents of package.json's devDependencies and dependencies were changed, note that it assumes that node_modules persists across different builds which might not be helpful if your CI server always start from scratch
npm-install-cache runs npm install and then copies your current node_modules folder (to somewhere in \tmp), if you call the script again it will verify any changes to package.json (instead of changes done on devDependencies or dependencies), if it didn't change then it will copy the node_modules folder stored in \tmp, the only limitation I see is that it's not cross platform and that the cache folder is \tmp which is erased on reboot (or maybe even when a is process finished!)
The second package might not work as it is but it seems like a good place to start :)
You can specify all of the packages you want to use locally in devDependencies in package.json, and then running npm install -d will install those instead of the main dependencies.
When I deploy my app to AWS, it's copied into a new directory, so NPM will install all the same packages, during each deploy, which can take a lot of time. Most of these packages haven't changed between builds (if at all), so having it do a full npm-install seems like a waste.
My app server runs a bunch of different Node apps, so installing globally isn't an option. Instead I'd like to have the app store it's node packages in a location that isn't wiped out during deployment, but have the option to update packages as necessary during npm install.
Does NPM have a concept of an app-specific module directory that isn't located in a subfolder of an app? That way I can delete the app folder, and not have to reinstall the same packages over and over again.
I could achieve this by using symlinks, or migrating the current node_module directory.
If you lock down your dependencies versions, NPM is likely to cache the packages. So the installation wouldn't take much longer.
If you prefer not to do this, you can install dependencies globally and link them with the npm link command (which is basically creating a symlink yourself!). Then, it'll be up to you update the globally installed packages regularly.