If I install stuff using brew or brew cask (on macOS) or npm or pip or composer or similar package/code/library managers, I have always wondered:
Where do they get their content from, who or what entity is managing or facilitating or hosting all those software or packages or modules or libraries?
Are there any checks, safety filters, audits, validation, or other mechanisms that prevent malware in the packages or libaries that are being distributed though these package managers?
Composer
List of packages is stored at https://packagist.org/, but there are only metadata. Packages are downloaded directly from related repositories (usually GitHub or GitLab), Packagist does not store or analyze its content. So while it may look a bit scary, the security model is based on trust to the vendor or direct code review. There is no magic solution which will pretend that it protects you from malware - you need to think what you're doing and what dependencies you're including into your project (or at least use some malware scanner on your own).
Related
I am following a Microservices course that creates a free NMP account and pushes a package he writes through the course into it and imports this package in other related files of the application like a common NPM module/package.
He also mentions there are other options like using Github or some local Git/version controller, etc.
As I am trying to make a commercial project for my own startup, I am curious to know what is a good/affordable option for a startup company in this situation? Is is still a good idea to use NPM free public account to push some packages and use it through our application?
I think that if you are creating a module that can be useful to the community or part of it, NPM is an interesting option to keep your packages.
If you're uploading not usable packages, but your whole application to npm, then that's messed up, NPM is a package registry. Github, Gitlab, and tools alike should be a more suitable option since those are general-purpose code repositories, not package registries.
Also, be aware that the packages you upload on a free npm account are available to any NPM user. see: NPM Private Packages
From my previous experience, both GitLab and Github offer good free small professional accounts that may be suitable for your case.
I would like to hard-pin my NPM dependencies. "Hard-pinning" would mean that an automated process would check my dependency list for certain packages with certain versions and if a package has been locally upgraded, an custom error message should be shown (ideally, this should be integrated in a pre-push Git hook).
The reasons for wanting this behavior could be:
external dependencies (e.g. other teams integrating with your project, requiring certain versions)
broken or unwanted behavior because of certain issues (e.g. "wait until #124 is fixed")
known non-obvious migration effort for major upgrades
upgrade incompatibilities (e.g. newest version requires, but does not enforce, a newer peer dependency).
Normal pinning does not cut it in this case: it's trivial to update pinned packages anyway, comments do not work with package.json without extra effort and sometimes the reasons are too important not to be displayed explicitly.
How can I achieve such "hard-pinning"?
I am designing an environment for productive research, i.e. writing, data-analysis, publication, etc.
In order to share the final results with others, I need to find a way to package this and to set up the local installation.
The project depends on Anaconda, so conda as a package manager is available.
It also includes
Pandoc and some pandoc packages, some will have to be fetched from Github directly because some versions are not available via conda-forge (doable in conda)
Atom and Atom packages; they should be installed and configured by my script (this works on the CLI via the apm package manager)
Node.js and Mermaid and a few other JS packages, which require npm calls
Some file-system-level operations, like deleting parts from packages where I only need a portion from, creating symlinks and aliases etc.
Maybe some Python code for modifying yaml/json/ini files or reading therefrom.
The main project will reside in a Github repository. It will be fine for users to clone it from there and start a build script locally.
My idea is to write a Bash shell script that
creates a conda environment based on requirements.yaml for everything that can be done this way
installs other parts using CLI commands (wget/curl etc.)
does all necessary modifications using CLI commands, maybe using a few short Python scripts (e.g. for changing or reading JSON or yaml files).
My local usage will be on OSX Big Sur, Linux should be supported, Windows compatibility would be nice-to-have.
Before I start:
Is this approach viable? I think it will be pretty transparent, but of course also a bit proprietary.
Docker is likely overkill for my purpose, and I also read that the execution will be slow on OSX.
The same environment will likely be installed multiple times on the same users' machine, so it is important that I can control e.g. the usage of existing packages and files via aliases or symlinks. It is not important that the multiple installations are decoupled for the non-python/non-conda parts (e.g. atom, node.js, mermaid could be the same binaries for all installations; just the set of Python packages might vary by installation).
Thanks for your expertise!
I don't know how it got this bad. I'm a web developer, and I use Ubuntu, and here are just some of the package managers I'm using.
apt-get for system-wide packages
npm for node packages
pip for python packages
pip3 for python 3 packages
cabal for haskell packages
composer for php packages
bower for front-end packages
gem for ruby packages
git for other things
When I start a new project on a new VM, I have to install seemingly a dozen package managers from a dozen different places, and use them all to create a development environment. This is just getting out of control.
I've discovered that I can basically avoid installing and using pip/pip3 just by installing python packages from apt, like sudo apt-get install python3-some-library. This saves from having to use one package manager. That's awesome. But then I'm stuck with the Ubuntu versions of those packages, which are often really old.
What I'm wondering is, is there a meta-package manager that can help me to replace a few of these parts, so my dev environment is not so tricky to replicate?
I had a thought to make a package manager to rule them all for that very reason. Never finished it though, too much effort required to stay compatible. For each package manager you have a huge community supporting it's upkeep.
Best advice I have is to try to reduce your toolchain for each type of project. Ideally you shouldn't need to work in every language you know for each project you work on. How many projects are you using that use both python 2 and python 3 simultaneously?
Keep using apt for your system packages and install git with it. From there try to stick to one language per project. AFAIK all of the package managers you listed support installing packages from git. The languages you mentioned all have comparable sets of tooling, so use the toolchain available for the target language.
I worked with a team that was using composer, npm, bower, bundler, maven, and a tar.gz file for frontend SPAs because those are the tools they knew. On top of all of that, they were using vagrant simply as a deployer. We considered our toolchain and described our need and realized that it could be expressed in a single language once we adopted appropriate tooling for the task at hand.
I have been reading page after page after page about the benefits of using YUM package installer and how NOBODY should built installs from source files (which again makes no sense to me) yet the repositories and source builders always package files in Tarball format, leaving a TON of work (which usually ends up going wrong) to the individual instead of formatting SRPMs for the end user.
Has the world gone mad? I feel like I am taking crazy pills!
Well, first of all there's more to life than just RPM and YUM. An SRPM would be (somewhat) useless to Debian, for instance.
As for why you'd use a package repository over building everything yourself, well I don't know about you, but I've much rather just run (I'm using Ubuntu so I have apt-get instead of yum):
# apt-get install firefox
Than trying to figure out all the dependencies, as well as all the dependencies dependencies, make sure I have the correct versions of everything, download/build/install any that I don't have (or are out of date: if updating existing dependencies, make sure the newer versions don't break any existing software that I have and make sure I don't end up with 15 different versions of the same thing), and only after all that then download/configure/build/install firefox.
Then realise I'll also want Open Office or MySQL and start all over again!
That said, there are some packages that I install the latest version of from source. For example, I run my media centre off MythTV and I always like to build the latest version of that from Subversion. But even then, with a package manager, that's as easy as:
# apt-get build-dep mythtv
> cd ~/src/mythtv/
> svn co <svn repo of mythtv>
> configure && (etc)
That is, the package management software already knows all the dependencies for MythTV and it can download and install them automatically. Why spend hours tracking it all down manually?
In the end, it sounds to me like maybe you'd prefer a distro like Gentoo... that's the benefit of Linux, of course. If you don't like how things are run in the Fedora/RedHat distribution, you can just choose a different one.
There are a few reasons to use a packaging infrastructure (like yum):
Creating "installations" is much easier to do, due to automatic dependency installation. From the simple yum install blah to creating chroots with mock/--installroot, or live CDs, etc.
Managing those installations. From the obvious yum update to operations which are much harder to do otherwise like: yum --security update, yum --bz=1234 update-minimal, yum --disablerepo=testing distro-sync.
Auditing those installations. The obvious examples here being yum history (not available in plain RHEL-5 atm.) and yum verify.
...however speed is not a factor, for instance Fedora rawhide moves as fast as gentoo.
RHEL-5 does not move that quickly, because it's 3 years old and is not supposed to break ... not because it's managed using yum/rpms. There are third party providers, like iuscommunity, which release co-installable newer releases for various packages. Or if you need to you create your own.
Or you can run a production server on Fedora rawhide or gentoo, both will have the latest packages really quickly ... I would not recommend that option though.
Among other things, tarballs are system independent and YUM appears to be RPM-based and thus mostly usable by Linux only (plus Netware and AIX, so as I said, Linux only :) )