Running npm prepublish on an entire project - npm

I have a weird set of "local" npm modules that use TypeScript and depend on each other similar to:
A -> B, C
B -> C
C -> D
I need to run npm install and get all of my TypeScript compiled in order or it won't be able to find things properly. I'm under the impression I should use prepublish scripts to handle the TypeScript compile but it doesn't seem to cascade the prepublish request for local dependencies.
How am I supposed to set up a bunch of local modules with prepublish scripts such that they all get resolved appropriately when running npm install?
Another way to word what I am asking: How do I maintain multiple, local node modules and modify them at the same time? The modules have varying dependencies on each other and it is extremely inconvenient to modify them in isolation.

TypeScript compile but it doesn't seem to cascade the prepublish request for local dependencies.
Indeed prepublish only runs install for dependencies. Your dependencies should already be built (with prepublish) before putting them on NPM and installing them.

I figured out how to do what I needed. After updating to npm 3.3.9 and TypeScript 1.6, I was able to use a postinstall script to build things on the fly. The prototype lives here: https://github.com/MrHen/TypeScriptNpm
But the important pieces are:
// In the module's package.json
"scripts": {
"build": "gulp npmbuild",
"postinstall": "npm run build"
},
And:
// In the server's package.json
"dependencies": {
"hen-doodad": "file:../modules/hen-doodad",
"hen-widget": "file:../modules/hen-widget"
}
And:
// In the gulpfile
gulp.task('npmbuild', function() {
gulp_util.log('Detecting appropriate starting directory...', process.env.INIT_CWD);
var out = process.env.INIT_CWD + '/app';
var build = [process.env.INIT_CWD + '/**/*.ts', 'typings/tsd.d.ts', '!' + process.env.INIT_CWD + '/node_modules/**/*'];
var typings = 'typings/tsd.d.ts';
// ... do typescript build using above paths
Feels like a bit of a hack but this worked more consistently than prepublish. To run the whole thing, do an npm install inside of the server folder.
The gulp task is optional. Presumably you could use tsc directly.
It should be noted that this is most certainly not how you should be packaging up npm modules. The reason I had to do this involved details from a pre-existing build system.
Pitfalls from doing it this way:
Does not fit the standard publishing patterns used by npm.
The built files only ever live inside of the server's node_modules which can be awkward depending on what else you do with them.
Running npm install twice will not grab latest changes. You need to either remove the installed modules or update the version number on the module.
Each module gets its own TypeScript build instead of building en masse. If you can just build everything all at once you should do that instead.
Requires TypeScript 1.6 or greater due to how they auto-detect typings files included inside of node_modules.

Assuming that you are using TypeScript 1.6+ which support node module resolution. As you are trying to keep all modules synced, I guess your project is just starting.
I think symlinks would do the job, but if you have more concerns, please let me know.
More specifically, you may either create symlinks manually or take the advantages of npm link.
cd /path-of-a-module
npm link # this will create a link as global module
cd /path-of-your-app
npm link your-module-name
Then you may just maintain these modules happily.
As for dependencies configuration in your package.json file, it could be a git repository. But you could probably left the compiled *.js and *.d.ts files there and everything would just work.

Related

Patching a NPM package locally with patch-package, not working

I'm working on a vue.js frontend, and I need to patch a package to fit the special needs of the app. The package I'm trying to patch is 'vue-youtube' (not that it really matters). I'm trying to patch it with patch-package (https://www.npmjs.com/package/patch-package)
So basically :
I edited locally the /node_modules/vue-youtube/src/vue-youtube.js to fit my needs
I did add the postinstall script in my package.json : "scripts": { "postinstall": "patch-package" }
I did npm install patch-package --save-dev
Then I ran npx patch-package vue-youtube
It did create a vue-youtube+1.4.0.patch file in a /patches folder with my modifications
BUT, my modifications are not seen. When I do npm run serve and launch my webapp, the package used is still the one not edited. I tried running npm install before, without success. When I go to the /node_modules/vue-youtube/dist/vue-youtube.js (thankfully it is a small package so it is readable), I can see that indeed my modifications have not been "compiled".
What am I missing here ? I feel like I have followed eveything in the patch-package npm page..
Thanks
EDIT : Still investigating.. few more informations/questions :
my patch name is patches/vue-youtube+1.4.0.patch
when i run npm ls vue-youtube it returns just one element : vue-youtube#1.4.0
in my package.json the dependency listed is "vue-youtube": "^1.4.0", should it be different ? should it mention that it needs to be patched ?
EDIT 2 : I realized that I am not editing the node_modules/vue-youtube/dist/vue-youtube.js, but the node_modules/vue-youtube/src/vue-youtube.
If you edit the files in the dist folder, the patch works. (however I thought patch-package would allow me to edit the files in the src folder, in readable JS...)
WORKING SOLUTION :
If you edit the files directly in the dist/ folder of the package instead of the src/ folder, the patch works fine.
Adding below npm script in package.json after patching worked for me.
scripts: {
"prepare": "patch-package",
}
The lines from yarn documentation explains about prepare
For compatibility reasons, scripts called install, postinstall, prepublish, and prepare will all be called after your package has finished installing.
After adding this script in package.json, the changes of module file in patches folder has been patched into respective node module.
I was trying to do the exact same thing with some package, let's call it "some_package". When I saw the EDIT 2 my mind just connected the dots...
To test changes locally
Modify the files in node_modules/some_package/src folder and then, go to the node_modules/some_package and run:
$ npm install
$ npm run <name of the script that generates the dist folder>
No need to run npx patch-package nor postinstall step.
I think that this approach doesn't work for all packages, it depends on how the modified package's package.json is configured. Specifically, pay attention where the browser field is pointing (in my case ./dist/some_package.js).
CAVEAT: You will have to run npm install and npm run every time you make an update to the package.
To test changes and be able to share it among team members (when the package is on Github)
Make a fork of the package you want to modify.
Make all the changes you want to your forked version of the package.
Run the following to automatically update the package.json file to make the dependency point to your forked version:
$ npm install <github's user name>/<package's name of the forked repository>#<branch name> --save-prod
For instance, if your Github's user name is "johndoe", and you forked https://github.com/aurelia/framework, and you made a branch named "mycoolbranch" containing your changes, then it would be:
$ npm install johndoe/aurelia-framework#mycoolbranch --save-prod
Note that the --save-prod flag could be replaced with --save-dev if the dependency is just for development.
Take a look at this answer, it may help.
https://stackoverflow.com/a/71153240/9981565
For me it was happening because of version mismatch between package.json intended version of package and yarn.lock / package-lock.json

Install other package.json dependencies

Simple question : Is it possible, in a package.json, to reference another package.json, and install its dependencies ?
Thank you.
Yes, this is possible, and this is automatically done by npm install.
If you have pkg-a that depends on pkg-b, including pkg-a in your dependencies will install both pkg-a and pkg-b when running npm install. That is because dependencies are actually references to the package.json of other packages. NPM, upon running install, builds a dependency tree of all the packages that are indirectly required by your current project, and installs all of them in the node_modules directory, and keeps track of them all in package-lock.json.
Good question! but this is not possible as you cannot internally reference one json document from another (json is just a document format, it lacks any ability to process logic, import files etc), npm is configured to run using a single package.json file so your best best would be to put all your dependencies in a single package.json file or split your project into two directories with two separate package.json files, two npm installs etc, if for some reason you require your dependencies to be separate. You could then run your two node projects separately and connect via http if you wish.
The only way that you could come close to doing this would be to write an npm start script in the package.json that cds to another directory with a package.json and runs npm install, this would however only install the dependencies in the second directory node-modules/ folder

Transpile with babel after npm install

I installed this package: https://github.com/feathersjs/feathers-authentication-local (question not related specifically to this package). This package's source code is in ./src, and npm run compile puts the babel-transpiled code into ./lib, which is the main entry point.
My question is, after I do npm install feathers-authentication-local, how does npm know that it needs to run npm run compile? I thought of putting a postinstall script in package.json, this package doesn't have that.
Regarding what is uploaded to npm when publishing, there are two fields in package.json, files and directories, which are used to specify what should be uploaded.
Take a look as well to "main" property, it points to the files that will be used when importing a module in your aplication so:
import foo from 'foo'
Will look into node_modules/foo/$(main) which in this case points to lib/
The package actually doesn't compile after install on client's machines, but it's probably compiled on the mantainer's machine and then published on NPM.
Compiling process is fired by the prepublish script in package.json.

dependencies and devDependencies when using webpack

Does it make sense to put any modules into package.json dependencies when I use webpack?
When I want to develope a package, I use git clone <url> then npm install, then npm installs all dependencies and devDependencies from package.json file and it makes sense.
When I'm end-user and I just want to install some package into my node_modules to use it in my project, I run npm install package-name, then npm installs package-name with only its dependencies, and it makes sense too.
But does it make sense to put any modules into dependencies when I use webpack? The webpack will bundle all dependencies into eg. bundle.js, so, for me, there is no need to install dependencies then (while they're included into bundle.js file).
Let's assume that I put all neccessary modules into devDependencies (keep the dependencies object empty) for my project: my-project, bundle it with webpack and publish:
the developer-user will use git clone <url to my_project>, then run npm install, then npm will install devDependencies from package.json (and ommit empty dependencies object), then it is ready to develope.
the end-user will use npm install my-project, then npm will install my-project, do not install devDependencies (because this is for production) and do not install dependencies (because dependencies object in package.json remain empty). Putting anything into dependencies would double the dependencies: both the dependencies would be installed, and the same dependencies would be accessible in the bundle.js file.
Am I right?
You are correct that there may be no dependencies once it's been transpiled with webpack. However, some packages are multi-purpose and may be used in multiple ways, so in some circumstances dependencies may still be required.
If you look at the package.json specification, there are two possible entry points, 'main' and 'browser'. There is also the proposed 'module' entry point. It is currently under discussion about how to handle these in webpack and users seem to want webpack to prioritize them as module > browser > main, however browser is currently used by webpack first.
The idea behind prioritizing them in the order module > browser > main is presumably that browsers could use the pre-transpiled stuff directly in "browser", whereas another project calling require() or include() on your package would use non-transpiled code from the "module" entry. The "module" entry code could contain modern JavaScript with new features and the project/package requiring it could then transpile it to their own specifications, using "browserslist" for example.
I found this question because I was wondering the same thing...

How to shrinkwrap npm modules without local dependencies

I tried to maintain package.json with the list of node modules dependencies. when I call npm install it installs the node modules.and generates a folder for it in my app. I call the npm shrinkwrap. But this generates the dependency on the local node module
"dependencies": {
"async": {
"version": "0.2.5",
"from": "async#0.2.5",
"resolved": "https://registry.npmjs.org/async/-/async-0.2.5.tgz"
},
when I upload the app to the appfog server it can install from the npm-shrinkwrap.json. So Ideally I want to remove the node modules folder and just pass the shrinkwrap.json file. But it has this "from". I had in the past generated the shrinkwrap & it didn't have the "from" field in there.
How to generate without "from"/ can I just get a shrinkwrap file from package.json. so my app will be leaner. I can maintain all the node module globally.
Thanks
I'm a bit confused by your question.
Shrinkwrap does not install, package, upload or do anything to your dependencies.
All it does is scan your installed node_modules and record the versions (recursively) into a file. Invoking npm install after that file is defined becomes repeatable, which is a principle of software engineering.
"from" was introduced a few months back. The npm shrinkwrap command seems to set it to the URL from which a module was installed. This is probably for portability. npm install takes a module name, consults a registry (whose URL is configurable as an npm config setting) and installs it. I could take the same package.json and npm-shrinkwrap.json, put them on another machine and theoretically get a different result if that machine's npm config settings point it to a different registry. Therefore, embedding the resolved URL in the shrinkwrap file adds an additional level of repeatability to npm install
See the npm config man page for details of setting the registry parameters.
According to npm issue 3145 on github, the "from" setting is known to cause backwards-compatibility issues with pre-1.2.x npm systems. Upgrading is the only resolution.
https://github.com/isaacs/npm/issues/3145
I think that you are looking for shrinkpack: https://www.npmjs.com/package/shrinkpack
from the doc:
Shrinkpack complements the npm shrinkwrap command by maintaining a node_shrinkwrap directory in your project, containing the exact same tarballs that npm install downloads from https://registry.npmjs.org.
The rest of the npm install process is exactly the same. The only difference is that no network activity is necessary when installing and building your project. The node_shrinkwrap directory can be ignored in your editor (much like is done with the node_modules directory) but is instead checked into source control.