I have couple of NPM packages that requires binary files during it's installing process. (For example, during node-sass installation scripts, the package requires a binary file that could be found on node-sass releases page).
My team is working on private network environment (disconnected from github) and therefore we need to host/serve the binaries privately.
At the moment, we use the sass_binary_dir parameter which makes the install script to look for the files in a shared drive that contains the needed binaries.
That method is fine for node-sass but is not working for other packages that requires the real binary repository or another website / proxy but not filesystem location or directory.
I would like to know if there is a recommended way to host the files ? (Something like Verdaccio but for binary files).
I also thought about fileZilla but it seems as a bit uncomfortable solution.
Writing a server myself could be fine as a temporary solution, but in the future I belive it would have to be maintained by another more organized solution.
The solution there was to create a simple API that enables fetching the package over HTTPS / FTP.
Related
which tool among those in above is used to solve what sort of problem? The simplified and straight to the point answer the better.
You can find some pretty good descriptions of each of these with a basic Google search.
In short, npm is a software repository. Grunt is a tool used bring together multiple javascript tasks into single commands. Webpack is a powerful module bundler, allowing you to bring together javascript, css, html from various sources (one being npm) and bundle them in such a way that you can be left with a single javascript module containing all the code you require.
The World's Largest Software Registry (Library)
npm is the world's largest Software Registry.
The registry contains over 800,000 code packages.
Open-source developers use npm to share software.
What is NPM # W3Schools
Grunt is a JavaScript task runner, a tool used to automatically perform frequent tasks such as minification, compilation, unit testing, and linting. It uses a command-line interface to run custom tasks defined in a file (known as a Gruntfile). Grunt was created by Ben Alman and is written in Node.js. It is distributed via npm.
What is Grunt # Wikipedia
webpack is an open-source JavaScript module bundler.[5][6][7][8] It is made primarily for JavaScript, but it can transform front-end assets such as HTML, CSS, and images if the corresponding loaders are included.[9] webpack takes modules with dependencies and generates static assets representing those modules.[10]
Webpack takes the dependencies and generates a dependency graph allowing web developers to use a modular approach for their web application development purposes
What is Webpack # Wikipedia
difference between grunt and webpack
grunt is stream management tools that perform functions required by users through task configuration, such as format verification, code compression, etc. It is worth mentioning that the code processed by grunt is only replaced by local variable names and simplified. Nothing has changed, it's still your code.
While webpack has carried out a more thorough packaging process, and is more inclined to transform the module grammar rules. The main task is to break through the gap between browsers, analyze, compress, merge, and package various static files that are not originally recognized by the browser, and finally generate the code supported by the browser. Therefore, the code after webapck has been packaged. It’s not the code you wrote, maybe if you look at it again, you can’t understand it anymore.
npm is more like providing building enviroment
npm is a package management tool installed with NodeJS. It can solve many problems in NodeJS code deployment. Common usage scenarios are as follows:
Allow users to download third-party packages written by others from the NPM server to local use.
Allow users to download and install command line programs written by others from - the NPM server for local use.
Allow users to upload their own packages or command line programs to the NPM server for others to use.
npm is more like providing build enviroment, but grunt and webpack is working as building tools.
I am designing an environment for productive research, i.e. writing, data-analysis, publication, etc.
In order to share the final results with others, I need to find a way to package this and to set up the local installation.
The project depends on Anaconda, so conda as a package manager is available.
It also includes
Pandoc and some pandoc packages, some will have to be fetched from Github directly because some versions are not available via conda-forge (doable in conda)
Atom and Atom packages; they should be installed and configured by my script (this works on the CLI via the apm package manager)
Node.js and Mermaid and a few other JS packages, which require npm calls
Some file-system-level operations, like deleting parts from packages where I only need a portion from, creating symlinks and aliases etc.
Maybe some Python code for modifying yaml/json/ini files or reading therefrom.
The main project will reside in a Github repository. It will be fine for users to clone it from there and start a build script locally.
My idea is to write a Bash shell script that
creates a conda environment based on requirements.yaml for everything that can be done this way
installs other parts using CLI commands (wget/curl etc.)
does all necessary modifications using CLI commands, maybe using a few short Python scripts (e.g. for changing or reading JSON or yaml files).
My local usage will be on OSX Big Sur, Linux should be supported, Windows compatibility would be nice-to-have.
Before I start:
Is this approach viable? I think it will be pretty transparent, but of course also a bit proprietary.
Docker is likely overkill for my purpose, and I also read that the execution will be slow on OSX.
The same environment will likely be installed multiple times on the same users' machine, so it is important that I can control e.g. the usage of existing packages and files via aliases or symlinks. It is not important that the multiple installations are decoupled for the non-python/non-conda parts (e.g. atom, node.js, mermaid could be the same binaries for all installations; just the set of Python packages might vary by installation).
Thanks for your expertise!
A common issue I keep bumping into for web projects in net core, is the need for sharing javascript in easy to use modules from project to project. Often times large quantities of code written in VS project A could be very much used in project B, sometimes in the same solution.
Restrictions:
Must be self hosted, not publicly exposed, only within local network etc etc can access the libs/modules/packages/etc
Ideally can be performed via visual studio projects and make use of build tasks, powershell, msbuild, or other such automation tools to deploy and package, minify, bundle, etc etc the javascript libraries.
The absolute ideal is if this can all be hosted from just a network folder
NPM/Yarn
I'm not super familiar with either of these, but is there a way we can drag and drop javascript code we've built into some designated folder, perhaps modify some form of manifest, json or xml file or what have you, and then anyone can just npm install those packages? I guess what I'm wondering is, is there a way to tell npm "This folder now is a source of packages you can install from"?
Bonus points: If said "trust this folder" config can be set inside of the VS project, so if someone new grabs the git repo, it will just work "out of the box" and they dont need to go through steps configuring npm or yarn so it knows how to find those packages.
Libman
Same as above, but mostly I'm trying to figure out if there is any way at all to configure libman from VS. It's the default and what is currently in use, but it just has its four default CDNs it comes with that it trusts and I am not seeing any way at all to tell Libman "Here's a now resource for files to trust, add that to the selectable drop down"
But I am seeing basically zero configuration as an option for libman, which is quite disappointing.
Nuget
This is the other option that is already popular locally, but something about using nuget to deliver js files when NPM, Yarn, and Libman already exist sets my teeth on edge, but, we have I believe a locally hosted nuget server that could be used already, so the infrastructure I believe is already setup, if not, I know how to do it. I do like the fact that nuget 100% for sure could leverage actual projects and build steps and msbuild and etc for deploying.
Conclusion
What's the popular and easy way to do this nowadays? Best case scenario is if there's a way to go, "Put a manifest.json file in the folder root that points to all the modules inside, then add it as a trusted source to your package manager, and now you can install those packages"
I want to split up the code base of several of my project into isolated package like projects. Those should be easily usable by npm but they do not seem significant enough to be published to the global npm registry.
So, my question is if there is a middle way to handling them like local provided packages and installing them with their path and publishing them in the global repository.
Concerns:
cluttering the npm registry with packages which don't seem to be significant enough to take up the name
the need to document and to create tests for each package seems to be too much and I would not sleep well publishing packages which are not well documented and tested
I take up a name which might be more appropriate to be used by a more sophisticated package and maintainers
I still want other to be able to easily try / use this package, to see if it fits their needs
Alternatives:
A) creating a private npm repository (with CouchDB?)
+ is pretty much identical to the npm repository and would be easy to use
+ the versioning is identical just pure semver lookup
- every user needs to set up this repository if they want to use this package or need it as a child dependency in their (public npm) package (even though this is unlikely)
- Need to invest time into setting it up and maintaining it
B) Using my username npm namespace
+ would solve pretty much every problem
- namespaces seem to be meant for projects and its sub packages which wouldn't be the case for my packages since their only connection is the creator
- it seems arrogant to prefix your packages with your name, like you are tagging it with a big sign THIS WAS DONE BY ME
C) Using GitHub with a special detached branch which contains the (tagged) releases
+ you could use it like the global npm repository since the npm resolving strategy allows the repository url with a semver range in place of the version
- special case which is bound to break
- GitHub is not meant to provide npm packages, about no developer expects a git url instead of the versionrange, tools and firewalls might have problem with this
- workflow is really not meant this way neither for git nor for npm
D) using a local package and install package by its path
+ easy to setup and use
- no version management
- build steps must be done manually beforehand
- can not publish packages depending on those packages
- all dependencies have to be installed locally
E) making those packages more useful, implementing edge cases, writing documentation and testing the whole package
+ would resolve about all problems
- ALOT of extra work, primarily thinking about edge cases and giving the developer a good api
- sometimes you can't really get the name for you package (it collides with other) which results in weird
- it is your responsibility, you have to maintain it, be responsible (test it well, edge cases)
- cluttering of the npm repository
So those are all the alternatives which came to mind when I tried to find a solution. Please leave a comment / answer if you have another idea or maybe you can remove / reduce the importance of those contra points.
Maybe you could include your own experience, so I get a better view for the whole problem.
Currently I would just try to make the package more helpful to the greater majority but this does not work in all cases.
Thank you all for your time!
Installing from git is pretty standard feature in package managers. npm doesn't have Github-support, it's generic support for any git repo. Unless you can find some discussion about deprecating it from npm, I'd not worry about it. It's used internally in many companies for private packages.
Of course, there is still some trade offs: build artifacts and maybe a bit more clumsy workflow. Things like npm outdated doesn't understand git semver. For build artifacts, I have seen many projects to commit them to master branch to support direct git-install. If you look around older open source projects for example, that's the case quite often.
We went for a private repository with verdaccio running in a docker container, which is very similar to version A. It took some setup, but for our developers all it took was a single npm command to add the private repo "in front of" npm for all packages of the namespace we created. Granted, our packages are project specific, but in a private repository that does not really matter either way, does it?
We considered the local package option at first, but the drawbacks were just too big for us, even if it's very easy to setup.
I'm not sure this helps, but this is at least the setup we decided upon when we had the same issue a few months ago.
I need to distribute a webapp in an offline version on an USB. Electron seems like a great way to make the webapp feel native and work well on all platforms. There is (potentially) just one problem: The offline version requires several huge media files and therefore there is not enough space for each package to have its own folder with the media files. This can be solved by letting the packages share a common directory on the USB. Can you tell me if this is possible with Electron. And if so, how?
(Note that each package must be available on the USB.)