Related
A common issue I keep bumping into for web projects in net core, is the need for sharing javascript in easy to use modules from project to project. Often times large quantities of code written in VS project A could be very much used in project B, sometimes in the same solution.
Restrictions:
Must be self hosted, not publicly exposed, only within local network etc etc can access the libs/modules/packages/etc
Ideally can be performed via visual studio projects and make use of build tasks, powershell, msbuild, or other such automation tools to deploy and package, minify, bundle, etc etc the javascript libraries.
The absolute ideal is if this can all be hosted from just a network folder
NPM/Yarn
I'm not super familiar with either of these, but is there a way we can drag and drop javascript code we've built into some designated folder, perhaps modify some form of manifest, json or xml file or what have you, and then anyone can just npm install those packages? I guess what I'm wondering is, is there a way to tell npm "This folder now is a source of packages you can install from"?
Bonus points: If said "trust this folder" config can be set inside of the VS project, so if someone new grabs the git repo, it will just work "out of the box" and they dont need to go through steps configuring npm or yarn so it knows how to find those packages.
Libman
Same as above, but mostly I'm trying to figure out if there is any way at all to configure libman from VS. It's the default and what is currently in use, but it just has its four default CDNs it comes with that it trusts and I am not seeing any way at all to tell Libman "Here's a now resource for files to trust, add that to the selectable drop down"
But I am seeing basically zero configuration as an option for libman, which is quite disappointing.
Nuget
This is the other option that is already popular locally, but something about using nuget to deliver js files when NPM, Yarn, and Libman already exist sets my teeth on edge, but, we have I believe a locally hosted nuget server that could be used already, so the infrastructure I believe is already setup, if not, I know how to do it. I do like the fact that nuget 100% for sure could leverage actual projects and build steps and msbuild and etc for deploying.
Conclusion
What's the popular and easy way to do this nowadays? Best case scenario is if there's a way to go, "Put a manifest.json file in the folder root that points to all the modules inside, then add it as a trusted source to your package manager, and now you can install those packages"
If I install stuff using brew or brew cask (on macOS) or npm or pip or composer or similar package/code/library managers, I have always wondered:
Where do they get their content from, who or what entity is managing or facilitating or hosting all those software or packages or modules or libraries?
Are there any checks, safety filters, audits, validation, or other mechanisms that prevent malware in the packages or libaries that are being distributed though these package managers?
Composer
List of packages is stored at https://packagist.org/, but there are only metadata. Packages are downloaded directly from related repositories (usually GitHub or GitLab), Packagist does not store or analyze its content. So while it may look a bit scary, the security model is based on trust to the vendor or direct code review. There is no magic solution which will pretend that it protects you from malware - you need to think what you're doing and what dependencies you're including into your project (or at least use some malware scanner on your own).
I'm looking for a "repository" to store derived information (build artifacts).
We have a repository (currently Mercurial) to store our source code. When something is pushed to the source repository the code goes through a continuous integration server and we do an incremental build and as a result some dlls will be changed. This should be added to some "repository" so that everybody can use that version without needing to do the build again.
I'm looking for the following features:
It should be easy to update the source code and get the corresponding binaries (we could probably make a script for that)
You should easily get all binaries at once (not only those that changed during the last incremental build.
Binaries that weren't changed should only be stored once in the repository.
When updating the source code and the binaries only the changed binaries should be transferred (and not all binaries). This is similar to what happens for source code.
When updating to some version, only that version should be stored locally, not the complete history.
We should be able to remove certain versions from the binary "repository" after a while. However if the dlls are still necessary for subsequent incremental builds, these dlls should of course not be completely removed from the "repository"
What would fit these requirements?
I agree with Manfred, what you are looking for is a binary repository manager. Besides the Nexus repository manager you should consider Artifactory.
As for the feature list you asked about:
As you have mentioned the CI server should be responsible for identifying a change in the version control and starting a build process which creates the binaries. The CI server/build tool should also deploy the generated binaries to the repository manager, in case the build was successful. Artifactory offers a build integration feature which takes care of deploying the binaries together with the build metadata.
Using the build integration feature of Artifactory, you can get a list of all the binaries generated by a specific build and download them as an archive. Artifactory provides a REST API for those actions.
There are different approaches for storing the artifacts in a repository manager. Some tools stores a multiple copies of the same binary. Other, for example Artifactory, use a checksum based storage which keeps only one copy per binary (based on its checksum). This pays of if you keep multiple copies of the same binary in different repositories, especially if you are dealing with large binaries (war files, docker images, ISOs etc.). Another benefit are cheap copies/moves between repositories which is a common practice for promotion workflows.
The Artifactory build integration uses checksum based deployment which deploys only binaries which does not exist in Artifactory. For binaries which do exist and have not changed, it only created a new reference to the existing binary saving the need to send the actual bytes.
Artifactory provides multiple option of cleaning up binaries, including built in cleanup policies and the option to develop your own custom logic using user plugins and the Artifactory query language (AQL)
In addition, I highly recommend to take a look at the binary repository comparison matrix.
Disclaimer: I am working for JFrog the company behind Artifactory
You are basically asking for a repository manager like the Nexus Repository Manager as you have correctly identified with the tags.
In terms of specific requirement from your questions here are a couple of ideas.
binary components are typically identified via some coordinates that most of the time includes some sort of name and version. A release and build process changes those and deploys them to the repository. This allows you to match source code with binaries. You can also embed information like git refs in the produced binaries.
accessing the binaries is typically done via HTTP, so its easy. You then just have to determine what it means to get "all binaries".
not duplicating binaries that are essentially the same can be supported by the underlying file system or the build tool. I have seen both processes to work. Often it is however not worth the effort since storage is cheap.
there are various ways to automatically clean up repositories including scheduled tasks that do it regularly. Worst case you have to implement your own logic in an extension
Disclaimer: I work as community advocate and trainer for the Nexus Repository Manager with Sonatype.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Every time a new developer joins the team or the computer a developer is using changes, the developer needs to do lots of work to setup the local development environment to make the current project work. As a SCRUM team we are trying to automate everything including deployment and tests so what I am asking is: is there a tool or a practice to make local development environment setup automated?
For example to setup my environment, first I had to install eclipse, then SVN, Apache, Tomcat, MySQL, PHP. After that I populated the DB and I had to do minor changes in the various configuration files etc... Is there a way to reduce this labor to one-click?
There are several options, and sometimes a combination of these is useful:
automated installation
disk imaging
virtualization
source code control
Details on the various options:
Automated Installation Tools for automating installation and configuration of a workstation's various services, tools and config files:
Puppet has a learning curve but is powerful. You define classes of machines (development box, web server, etc.) and it then does what is necessary to install, configure, and keep the box in the proper state. You asked for one-click, but Puppet by default is zero-click, as it checks your machine periodically to make sure it is still configured as desired. It will detect when a file or mode has been changed, and fix the problem. I currently use this to maintain a handful of RedHat Linux boxes, though it's capable of handling thousands. (Does not support Windows as of 2009-05-08).
Cfengine is another one. I've seen this used successfully at a shop with 70 engineers using RedHat Linux. Its limitations were part of the reason for Puppet.
SmartFrog is another tool for configuring hosts. It does support Windows.
Shell scripts. RightScale has examples of how to configure an Amazon EC2 image using shell scripts.
Install packages. On a Unix box it's possible to do this entirely with packages, and on Windows msi may be an option. For example, RubyWorks provides you with a full Ruby on Rails stack, all by installing one package that in turn installs other packages via dependencies.
Disk Images Then of course there are also disk imaging tools for storing an image of a configured host such that it can be restored to another host. As with virtualization, this is especially nice for test boxes, since it's easy to restore things to a clean slate. Keeping things continuously up-to-date is still an issue--is it worth making new images just to propagate a configuration file change?
Virtualization is another option, for example making copies of a Xen, VirtualPC, or VMWare image to create new hosts. This is especially useful with test boxes, as no matter what mess a test creates, you can easily restore to a clean, known state. As with disk imaging tools, keeping hosts up-to-date requires more manual steps and vigilance than if an automated install/config tool is used.
Source Code Control Once you've got the necessary tools installed/configured, then doing builds should be a matter of checking out what's needed from a source code repository and building it.
Currently I use a combination of the above to automate the process as follows:
Start with a barebones OS install on a VMWare guest
Run a shell script to install Puppet and retrieve its configs from source code control
Puppet to install tools/components/configs
Check out files from source code control to build and deploy our web application
I stumbled across this question and was very suprised that no one has mentioned Vagrant yet.
As Pete TerMaat and others have mentioned, virtualization is a great way to manage and automate development environments. Vagrant basically takes the pain away from setting up these virtual boxes.
Within minutes you can have a completely fresh copy of your favourite Linux distro up and running, and provisioned exactly the same way your production server is.
No more fighting with OSX or Windows to get PHP, MySQL, etc. installed. All software lives and runs inside the virtual machine. You can even SSH in with vagrant ssh. If you make a mistake or break something, just vagrant destroy it, and vagrant up to start over fresh.
Vagrant automatically creates a synced folder to your local file system, meaning you don't need to develop within the virtual machine (ie. using Vim). Use whatever your editor of choice is.
I now create a new "Vagrant box" for almost every project I do. All my settings are saved into the project repository, so it's easy to bring on another team member. They simply have to pull the repo, and run vagrant up, and they are literally ready to go.
This also makes it much easier to handle projects that have different software requirements. Maybe you have some projects that rely on PHP 5.3, but some newer ones that run PHP 5.4. Just install the version you want for that project.
Check it out!
One important point is to set up your projects in source control such that you can immediately build, deploy and run after checkout.
That means you should also checkin helper infrastructure, such as Makefiles, ant buildfiles etc., and settings for the tools, such as IDE project files.
That should take care of the setup hassle for individual projects.
For the basic machine setup, you could use a standard image. Another option is to use your platform's tools to automate installation. Under Linux, you could create a meta-package that depends on all the packages you need. Under Windows, a similar thing should be possible using MSI or the like.
Edit:
Ideally, instead of checking in helper infrastructure, you check in the information that allows the build to generate the helper infrastructure. This is the approach taken by e.g. the GNU build system (autotools etc.), or by Maven. This is even more elegant, because you can (theoretically) generate infrastructure for any (supported) build environment, thus you are not bound to e.g. one specific IDE, and settings in the helper infrastructure (paths etc.) don't need to duplicate the main project settings.
However, this also a more complex approach, so if you can't get it to work, I believe checking in stuff like IDE files directly is acceptable.
I like to use Virtual PC or VMware to virtualize the development environment. This provides a standard "dev environment" that could be shared among developers. You don't have to worry about software that the user could add to their system that may conflict with your development environment. It also provides me a way to work to two projects where the development environments can't both be on one system (using two different versions of a core technology).
Use puppet to configure both your development and production environment. Using a top-notch automation system is the only way to scale your ops.
There's always the option of using virtual machines (see e.g. VMWare Player). Create one environment and copy it over for each new employee with minimal configuration needed.
At a prior place we had everything (and I mean EVERYTHING) in SCM (clearcase then SVN). When a new developer can in they installed ClearCase|SVN and sucked down the repository. This also handles the case when you need to update a particular lib/tool as you can just have the dev teams update their environment.
We used two repo's for this so code and tools/config lived in separate places.
I highly recommend Blueprint from DevStructure. It's open-source and your use case is actually the exact reason we originally wrote the software. Our goals have somewhat changed, but it still is the perfect tool for what you are describing. In short, you can create reusable server configs - dead simple configuration management. I hope this helps!
https://github.com/devstructure/blueprint (Blueprint # Github)
I've been thinking about this myself. There are some other technologies that you could throw into the mix. Here's what I'm currently setting up:
PXE based pre-seeded installation images (Debian Squeeze). You can start up a bare-metal machine (or new virtual appliance) and select the image from the PXE boot menu. This has the major advantage of being able to install your environment on physical machines (in addition to virtual appliances).
Someone already mentioned Puppet. I use CFEngine but it's a similar deal. Essentially your configuration is documented and centralized in policy files which are continually enforced by an agent on the client.
if you don't want a rigid environment (i.e. developers may choose a combination of tool-sets) you can roll your own deb packages so new devs can type sudo apt-get install acmecorp-eclipse-env or sudo apt-get install acmecorp-intellij-env, for example.
Slightly off-topic, but if you run a Debian based environment (i.e. Ubuntu), consider installing apt-cacher (package proxy). In addition to saving bandwidth, it will make your installations much faster (since packages are cached on your local network).
If you're using OSX and working with Rails. I'd suggest either:
https://github.com/platform45/let-there-be-light
https://github.com/thoughtbot/laptop
If you use machines in a standard configuration, you can image the disk with a fresh perfectly configured install -- that's a very popular approach in many corporations (and not just for developers, either). If you need separately configured OS's, you can tar-bz2 all the added and changed files once a configured OS is turned into your desired setup, and just untar it as root to make your desired environment from scratch.
if you're using a linux flavor, you've probably got a package management system: thinks .rpm for fedora/redhat, or .deb for ubuntu/debian. many of the things you describe already have packages available: svn, eclipse, etc. you could roll your own packages for company specific software, create a repository (perhaps only available on the local network) and then your setup could be reduced to a single bash script which would add the company repo to /etc/apt/sources.list (debian/ubuntu) and then call a command like,
/home/newhire$ apt-get update && apt-get install some complete package list
you could use buildbot to then automate regular builds for company packages that change often.
Try out DevScript at http://nsnihalsahu.github.io/devscript .
Its one command like ,
devscript lamp or devscript laravel or devscript django . In around a few minutes ,depending on the speed of your internet co
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
We are using Maven for a large build process (> 100 modules). We have been storing our external dependencies in source control, and using that to update a local repo.
However, we are ready to graduate to a local repo that can cache central so that we don't have to proactively download all 3rd parties (but we can still have a local repo to pull from). In addition we want to publish our internal build artifacts from a nightly build so that developers don't have to build the world.
We are considering Nexus and Artifactory. What are the reasons for preferring one over the other? Are there others we should be considering?
I'm sure that if you only talk about storing binaries from "mvn deploy" both will do fine.
We use Artifactory very extensively with all upgrades along the way. Lots of projects, numerous snapshots deployed and external repos proxied. Not a single problem. I find it hard to explain how other people experience issues with its DB, indexing or anything else. Nothing like that ever happened to us. Also, Artifactory allows to store data on a disk and only use a DB for storing metadata, it is quite flexible (see more here).
What makes those applications very different is their approach towards integration with other build tools and technologies. Nexus and Sonatype are pretty much locked on Maven and m2eclipse. They ignore anything else and only recently started to work on their own proprietary Hudson integration (see their Maven 3 webinar).
EDIT: This is not true anymore as of 2017 Nexus gives a much larger support for other build tools End of Edit
Artifactory provides an awesome Hudson, TeamCity and Bamboo integration, and Gradle / Ivy support. So while Nexus gives you nothing once you step out of Sonatype "comfort zone" (Maven, m2eclipse), Artifactory embraces and collaborates with all major build tools.
In fact, being able to deploy build artifacts from Hudson, when job has finished, and not by "mvn deploy" is a huge difference: Artifactory Hudson plugin makes an atomic-like deploy of all artifacts at once, only when a build job finished successfully. "mvn deploy" runs after each module and can deploy a partial set of artifacts if a build job fails in the middle. Deploying from Maven on module completion and not from a build server on job completion is really a bad thing to do.
As you see, Artifactory thinks "outside the box" while Nexus thinks "inside the box" and only cares about Maven and Maven artifacts.
Something else that makes Artifactory more accessible is their cloud-based Artifactory Online solution. For about $80 a month you have your own Artifactory instance, no need to dedicate any server for it.
Artifactory has a simple and straightforward REST API, don't know how it works for Nexus.
Edit Nexus has also a REST API that you can use easily as well.
To summarize, for basic storage of Maven artifacts I think both are fine. But while Nexus stops there being strictly a "Maven repository manager", Artifactory goes on and on, being a general "Binaries storage" for binaries of any kind, from any build tool and CI server.
I don't know about Artifactory but here are my reasons for using Nexus:
Dead simple install (and since 1.2, dead simple upgrade, too)
Very good web UI
Easy to maintain, almost no administrative overhead
Provides you with RSS feeds of recently installed, broken artifacts and errors
It can group several repositories so you can mirror several sources but need only one or two entries in your settings.xml
Deploying from Maven works out of the box (no need for WebDAV hacks, etc).
it's free
You can redirect access paths (i.e. some broken pom.xml requires "a.b.c" from "xxx"). Instead of patching the POM, you can fix the bug in Nexus and redirect the request to the place where the artifact really is.
Artifactory supports both file-system and database storage backends. Storage is checksum based and identical binaries are stored only once, no matter how many times they appear in the repo, which makes Artifactory more efficient storage-wise. Move and copy are also very cheap because of this architecture (in Nexus there's no REST for move/copy - you have to move stuff on the file system, then run corrective actions on the repo to let it know content has changed).
Another important differentiator is Artifactory has unique integration with Hudson and TeamCity for capturing information about deployed artifacts, resolved dependencies and environment data associated with build runs, which provides full build traceability.
Artifactory stores the artifacts in a database, which means that if something goes wrong, all your artifacts are gone. Nexus uses a flat file for your precious artifacts so you don't have to worry about them all getting lost.
If you need the "Pro" features of either (e.g. Staging repos, artifact promotion, NuGet), , then you need to consider the different pricing models, which are displayed on their websites.
http://www.jfrog.com/home/v_pricing
http://www.sonatype.com/nexus/purchase
In summary:
Artifactory Pro
you pay per server
you can pay more for increased service hours
Nexus Pro
you pay per seat, i.e. how many developers downloading artifacts
support service is Mon-Fri 0800-2000 ET only, no matter what you pay
No matter how many users you have, Nexus Pro offers a support service that's broadly equivalent to Artifactory's $7,450/year "Silver Value Pack".
$7,450/year will buy you approximately 67 Nexus Pro seats (1-50 # $108, the rest # $120).
On price and support alone then, Nexus Pro makes sense until you get to 67 users, at which point Artifactory becomes the cheaper option.
If you're doing all the support in-house; however, that magic point is about 23 users (Artifactory's most basic support offering is $2,750/year).
I made some research recenly about Artifactory 2 and Nexus 1.3. I'll list here the main differences I found:
Artifactory stores metadata and optionally files in DB, Nexus writes directly to file system. There are pros. and cons. for each approach. DB supports transactions, while in FS stored files can be accessed directly.
Artifactory has higher system requirements especially for disk space.
The most complete comparison: http://binary-repositories-comparison.github.io/
You should use Artifactory
Its latest version was a real jump
You can backup incrementally your repositories , which means you can have all your artifacts saved and maintain
Its has a easy to use web ui
and is really easy to set up
i enjoyed it a lot
check out its new version 2.0
From a learners point of view I note some specific differences between the two.
Sonatype .war deployment is not supported on Jboss application server at the time, although it does run under Tomcat.
Sonatype does not offer me an Amazon Machine Image (AMI), at present, that I could quickly stand up and test.
An Artifactory AMI is provided by Bitnami and takes a only a few minutes to stand up and a few more minutes to configure, maybe several tens of minutes dependant upon what you're trying to achieve.
Artifactory offer a SaaS version of Artifactory in the cloud so you can focus on getting things done rather than infrastructure.
I've no experience with Nexus but I've found Artifactory very intuitive and easy to configure, at least initially.
Added - I do note that the Artifactory User Guide, which may be OK for a seasoned pro, is a bit light on for some in depth explanations. For instance, starting out, one unzips and then addes a Repository, say RedHat's Jboss EAP Enterprise Repo. All goes fine but then when I tried to view the artifacts that were imported Artifactory reports zero artifacts? No errors or warnings so I'm now looking for an explanation. Is this normal or not normal? A simple explanation in the doco can quickly point one in the right direction. Being a good contributor I'm adding these comments to the project for the benefit of other starters.
All politics/religion aside, licensing makes a difference for some organizations.
Nexus is GPL now AGPLv3 and now Eclipse Public License (EPL).
Artifactory is Apache licensed LGPLv3 licensed as of version 2.1 of the product.
You may also want to consider Archiva, just for comparison's sake. It's Apache 2.0 licensed.
I see that Nexus usage is growing, while Artifcatory usage is generaly staying flat.
Picture is taken from here http://blog.sonatype.com/2014/11/42000-nexus-repository-managers-and-growing/
There is also matrix-comparison http://docs.codehaus.org/display/MAVENUSER/Maven+Repository+Manager+Feature+Matrix
Both Artifactory and Nexus have more or less similar feature set but Artifactory's LDAP support makes it more attractive over Nexus. Though Nexus also have LDAP support but in paid version :-(
Hmmm...my experience with artifactory is awful...but I'm a relative newbie so take it with a grain of salt. My overall complaint is that jar files recently uploaded to Artifactory do not seem to get indexed right away - as in for hours - and there does not seem to be a good way to force it. I've tried various things that appeared as if they should have worked, but didn't. I have been working with m2eclipse, adding dependencies to a project that i'm converting from ant. When I try to add a jar that I have just added to artifactory, I expect it to show up as a choice in the selector but it does not.
a coworker told me that they had installed nexus and so far they like it...but I can't vouch for it yet. I'm about to install that on a Linux box as soon as IT can find me one.