I'm coming from a background much more familiar with composer. I'm getting gulp (etc) going for the build processes and learning node and how to use npm as I go.
It's very odd (again, coming from a composer background) that a composer.lock-like manifest is not included by default. Having said that, I've been reading documentation on [shrinkwrap], [npm-lockdown], and [npm-seal]. ...and the more documentation I read, the more confused I become as to which I should be choosing (everyone thinks their way is the best way). One of the issues I notice is that npm-seal hasn't changed in 4 years and npm-lockdown in 8 months -- this all leads me to wonder if this because it's not needed with the newest version of npm...
What are the benefits / drawbacks of each?
In what cases would I use one over another in Project A, but use a different one in Project B?
How will each impact our development workflow?
PS: Brownie points if you include the most basic implementation example for each. ;)
npm shrinkwrap is the most standard way how to lock your dependencies. And yes, npm install does not create it by default which is a pity and it is something that npm creators definitely should change.
npm-lockdown is trying to do the same things as npm shrinkwrap, there are two minor points in which npm-lockdown is better: it handles optional dependencies better and it validates checksums of the package:
https://www.npmjs.com/package/lockdown#related-tools
Both these features seem not so relevant for me; I'm quite happy with npm shrinkwrap: For example, npmjs guarantees that once you upload certain package at certain version, it stays immutable - so checking sha checksums is not so hot (I've never encountered an error caused by this).
seal is meant to be used together with npm shrinkwrap. It adds the 'checksum checking' aspect. It looks abandoned and quite raw.
Good question - I'm going to skip everything but shrinkwrap because it is the de-facto way to do this, per NPM's docs.
Long story short the npm-shrinkwrap.json file is akin to your lock files you are used to in every other package manager, though NPM allows different versions of the same package to play nice together by isolation - literally scoping and copying different entire versions to node_modules at different levels of the tree. If two projects that are parent-child to each other use the exact same version, NPM will copy the version to only the parent and the child will traverse up the tree to find the package.
Best practice is simply to update package.json for your direct dependencies, run npm install, verify that things are working while developing, then run npm shrinkwrap when you are just about to commit and push. NOTE: make sure to rm npm-shrinkwrap.json before running npm install during active development - if your direct dependencies have changed, you want package.json to be used, and not the lock! Also include node_modules in your .gitignore or equivalent in your source control system. Then, when you are deploying and getting to run the project, run npm install like normal. If npm finds an npm-shrinkwrap.json file, it will use that to recursively pull all locked modules, and it will ignore package.json in both your project and all dependent projects.
You might find shrinkpack useful – it checks in the tarballs which npm install downloads and bundles them into your repository, before finally rewriting npm-shrinkwrap.json to point at that local bundle instead.
This way, your project is totally locked down, completely available offline, and much quicker to install.
Related
I've run into a situation where I need to access a transitive dependency in a stable way. In other words, I'd like to declare a dependency whose version is "whatever library Foo is already using".
Specifically, I'm setting up an Eleventy site and want to use markdown-it-anchor with it. Both libraries involve slugifying text, for which markdown-it-anchor allows you to specify a custom function. To keep everything consistent, I want to tell markdown-it-anchor to use the same slugifying function as Eleventy. Eleventy doesn't export its slugifying function, but it's just using #sindresorhus/slugify, so I can import that directly.
The problem came in when I added a direct dependency on #sindresorhus/slugify — I added a dependency with a splat version, on the assumption that npm would simply resolve it to the version of #sindresorhus/slugify which was already present in node_modules/. Instead, it resolved to the latest version. I tried playing around with editing package.json and even package-lock.json manually, but npm is very firm about installing Eleventy's version of #sindresorhus/slugify where I can't reach it and not installing the same version for my own use unless I duplicate the version specifier in my package.json.
What I want is to be able to freely update Eleventy in the future and have reasonable confidence that markdown-it-anchor will continue to be passed the correct version of #sindresorhus/slugify without having to manually verify each time that Eleventy hasn't bumped their dependency. Is there any way to accomplish that?
Well, I kind of got this to work. I say "kind of" because I'm depending on the splat version ("*") of my transitive dependency, which is pretty fragile. Getting it to work at all was pretty ugly, too, so there are multiple ways in which this isn't a "proper" solution.
Opened up package-lock.json and looked at the transitive dependency's dependencies (and transitive dependencies). Fortunately, for #sindresorhus/slugify, this isn't too bad.
Rearranged node_modules/ to move the transitive dependency to the top level (where my package can find it), and all of it's dependencies to where it can find them, without introducing new version conflicts. Again, in my case, this wasn't too bad.
Edited both package-lock.json and node_modules/.package-lock.json to reflect the moved packages' new locations.
Ran npm ci both to verify that I hadn't made any terrible mistakes and to make sure package-lock.json and node_modules/.package-lock.json were formatted the way it liked. (The only change it made was to reorder the packages to keep their directories sorted.)
Manually added a dependency on the (now formerly) transitive dependency with a splat version.
Ran npm install and verified that it didn't actually install or rearrange anything.
After all that, #sindresorhus/slugify works as expected when used directly from my site's build. There are a couple of serious caveats, though:
I'm not sure what npm's behavior will be if/when Eleventy updates its dependency on #sindresorhus/slugify. It may well simply update the latter where it's already located, in which everything will be fine. Otherwise, it probably won't.
I'm also not sure what npm's behavior will be if/when #sindresorhus/slugify gets added as a dependency anywhere else. It may well leave the existing version where it is and install new, conflicting versions under the …/node_modules/ folder of whatever packages require them, in which case everything will be fine. Otherwise, it probably won't.
In other words, I discovered a way to put a fair amount of effort into creating a situation which seems to work, but may not actually do what I originally wanted. 😅
NPM can be used for different things
front-end projects where the final artifacts often have all their dependencies included.
service side services where the node_modules directory is available to the runtime server code.
NPM currently hoists dependencies to the root node_modules folder in a workspace project. Even their new proposal https://github.com/npm/rfcs/blob/main/accepted/0042-isolated-mode.md still doesn't seem to allow for all dependencies to be installed just alongside the node_modules folder next to the package.json for that workspace folder. You either have to copy the root node_modules folder to a parent directory, which gets complicated releasing multiple services (since it's a shared resource), or you must merge the two which is also not trivial.
This becomes a problem when you want to build in a CI environment and then move the code and all dependencies somewhere else
Does anyone have thoughts on how they've solved/avoided/approached this problem?
I ended up going with Lerna, since it seems to handle this situation perfectly. Lerna basically treats every "sub-project" independently.
https://lerna.js.org/
I don't think npm workspaces fits this type of workflow, or it wasn't designed to be in the first place
I have a Javascript package (A) which uses one of our own packages (B) as a dependency. Neither is on the NPM registry. A is an application, and B is installed directly from github using specifiers like this: github:user/project#version
When A is built with npm install; npm run build we normally want it to use the version of B defined in package.json's dependencies. So set the production version here, say: github:user/B#semver:^1.0.0. All good.
What I would like is to be able to build a development version of A with everything the same but with the development branch of B: github:user/B#dev.
devDependencies does not seem to be designed for this, rather for dependencies which are entirely absent in production. NODE_ENV environment variable doesn't seem able to affect the behaviour of install.
Currently I edit A's package.json dependencies to use github:user/B#dev, and remember not to commit this to the master branch. And if do I commit it to the dev branch, then I can't simply have dev follow master in a fast-forwarding manner, instead I have to maintain two branches and merge dev across to master.
----+---+---+---> dev
\ \ \
v v v
------+---+---+--> master
This is inconvenient and potentially error-prone.
I would guess that my situation is not an unusual one, and that there might be a better solution for this out there.
How do other people solve this?
After a few comments I now understand your requirement. The answer is, it depends on how you plan to differentiate a dev and prod version of your application.
Different Deployments
One option, is to have two separate deployments of your application - one for development, and one for production. If the source code itself should be the same, the rather than having two separate repos, you could use a monorepo with two different package.json inside, one for development purposes, and the other for production.
Aliases
The other option, would be to make use of the npm alias functionality - npm now supports package aliasing like the following:
"dependencies": {
"prodPackage": "npm:mypackage#prod",
"devPackage": "npm:mypackage#dev"
}
If you then need to focus on how this is used in your code, the NODE_ENV variable could then be used to control this:
const Package = process.env.NODE_ENV == 'prod' ? require('prodPackage') : require('devPackage')
Or create a more specific env variable that defines your package, then change this in your environments to be which version you require, like so:
#in .env
MY_PACKAGE=devPackage #or prodPackage
// in your .js files
const Package = require(`${process.env.MY_PACKAGE}`)
Package.json & Source Control
The only other alternative that springs to mind (which is much less preferred), is to either remove package.json from source control and use local ones depending on dev or prod.
There is no official approach to this unfortunately, but there are ways to manage it - including the one you mentioned above. You can decide which one best suits you if any.
Either way, using the alias approach or different deployments, you wont have to worry about keeping separate branches. I hope any of this helps.
After updating NPM to version 5, I found package-lock.json file with package.json.
What is the difference between this two files?
What are the advantages of package-lock.json?
A package.json file: lists the packages that your project depends on. allows you to specify the versions of a package that your project can use using semantic versioning rules.
According to the npm docs,
package-lock.json is automatically generated for any operations where npm modifies either the node_modules tree, or package.json . It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates.
This file is intended to be committed into source repositories, and serves various purposes:
Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.
Provide a facility for users to "time-travel" to previous states of node_modules without having to commit the directory itself.
To facilitate greater visibility of tree changes through readable source control diffs.
Basically package-lock.json is used to optimize the installation process by allowing npm to skip repeated metadata resolutions for previously-installed packages.
Before npm 5.x.x, package.json was the source of truth for a project. What lived in package.json was law. npm users liked this model and grew very accustomed to maintaining their package file. However, when package-lock was first introduced, it acted contrary to how many people expected it to. Given a pre-existing package and package-lock, a change to the package.json (what many users considered the source of truth) was not reflected in the package-lock.
Example: Package A, version 1.0.0 is in the package and package-lock. In package.json, A is manually edited to version 1.1.0. If a user who considers package.json to be the source of truth runs npm install, they would expect version 1.1.0 to be installed. However, version 1.0.0 is installed, despite the fact that v1.1.0 is listed is the package.json.
Example: A module does not exist in the package-lock, but it does exist in the package.json. As a user who looks to package.json as the source of truth, I would expect for my module to be installed. However since the module is not present in package-lock, it isn’t installed, and my code fails because it cannot find the module.
Read more about package-lock.json in the Official npm Documentation!
package.json records only your direct dependencies and their versions.
package-lock.json records not only your direct dependencies and exact versions, but also those of all dependencies of your dependencies - the entire dependency tree, in other words, with exact versions.
It's the fact that package-lock.json records the exact versions of all dependencies of a project, including sub-dependencies, that ensures that builds will be identical each time. (This is why npm ci bases its build on package-lock.json, not package.json.)
Builds based on package.json (as with npm i) cannot guarantee that all sub-dependencies are the exact same versions each build (e.g., if the subdependency of one of your dependencies releases an update, but the version of your direct dependency doesn't change), even if exact version numbers for direct dependencies are specified in the package.json.
Context:
We're creating a new solution consisting of multiple app portals (for lack of a better term), each portal will need to leverage off of a base project that will already include some of my employed proprietary code, as well as any new features pertaining to that portal. Our current app leaves much to be desired, and as we're getting a fresh start, we'd like to go at it the right way. (Thus I'd like to rubberduck my thoughts somewhat)
I've thought of a few possible ways to solve this. Each with it's pro' and cons.
1. GIT Fork A Base Project:
This seems like the most straight forward way. Have a PortalCore
project, then have each project fork it in a downstream only fashion.
Con: If the base changes, we'll need to manually update all of the dependant projects.
Pro: Easier to implement initially, and I believe will reduce some of the other more "laborious" tasks. (Example, single build file that will travel with each new portal with our build requirements.)
The flow would be:
Fork PortalCore > Core will be kept up to date via updating via GIT master
2. Base Project NPM Package:
This seems like an ideal route, as with each deployment the latest version of our base package/project will be installed with each portal.
Con: From my research it seems like we're not able to have a npm package install outside of the npm folder (this pertains to my question). We'll need to share the build file via some other means if we want it to sit in the project root.
Pro: Updates automatically rolled out with the build process
The flow would be:
New Project > Add Portal Core npm > Make custom build task, or grab
from some central repo > Will be kept up to date via npm install >
Gulp Build
3. Combination of the above
Have a git project only containing our base npm modules, & build config. The build can then handle things like moving files to the right location (example. node_modles -> root)
The flow would be:
Fork PortalCore > Core will be kept up to date via npm install > Gulp Build
Questions:
Is there a way to have an npm package (or another package manager) install files to a specific location? (I have checked the npm forum, and this seems like a dead end. But I thought I'd try my luck here)
Are we frankensteining it? We don't want to create a new monster. Does this logic make sense ITO creating something that should be somewhat modular by design, but allows for easier maintenance. How do the big boys do this... if they do this?
The answer to this ended up being much simpler than I expected:
Put all shared services in separate common NPM packages (common components,shared / common services, etc.)
And create a few yeoman generators that assists with initial project
initialisation. (The only drawback is someone needs to maintain them,
should some new core dependency come along... but such is dev life.)