Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Every time a new developer joins the team or the computer a developer is using changes, the developer needs to do lots of work to setup the local development environment to make the current project work. As a SCRUM team we are trying to automate everything including deployment and tests so what I am asking is: is there a tool or a practice to make local development environment setup automated?
For example to setup my environment, first I had to install eclipse, then SVN, Apache, Tomcat, MySQL, PHP. After that I populated the DB and I had to do minor changes in the various configuration files etc... Is there a way to reduce this labor to one-click?
There are several options, and sometimes a combination of these is useful:
automated installation
disk imaging
virtualization
source code control
Details on the various options:
Automated Installation Tools for automating installation and configuration of a workstation's various services, tools and config files:
Puppet has a learning curve but is powerful. You define classes of machines (development box, web server, etc.) and it then does what is necessary to install, configure, and keep the box in the proper state. You asked for one-click, but Puppet by default is zero-click, as it checks your machine periodically to make sure it is still configured as desired. It will detect when a file or mode has been changed, and fix the problem. I currently use this to maintain a handful of RedHat Linux boxes, though it's capable of handling thousands. (Does not support Windows as of 2009-05-08).
Cfengine is another one. I've seen this used successfully at a shop with 70 engineers using RedHat Linux. Its limitations were part of the reason for Puppet.
SmartFrog is another tool for configuring hosts. It does support Windows.
Shell scripts. RightScale has examples of how to configure an Amazon EC2 image using shell scripts.
Install packages. On a Unix box it's possible to do this entirely with packages, and on Windows msi may be an option. For example, RubyWorks provides you with a full Ruby on Rails stack, all by installing one package that in turn installs other packages via dependencies.
Disk Images Then of course there are also disk imaging tools for storing an image of a configured host such that it can be restored to another host. As with virtualization, this is especially nice for test boxes, since it's easy to restore things to a clean slate. Keeping things continuously up-to-date is still an issue--is it worth making new images just to propagate a configuration file change?
Virtualization is another option, for example making copies of a Xen, VirtualPC, or VMWare image to create new hosts. This is especially useful with test boxes, as no matter what mess a test creates, you can easily restore to a clean, known state. As with disk imaging tools, keeping hosts up-to-date requires more manual steps and vigilance than if an automated install/config tool is used.
Source Code Control Once you've got the necessary tools installed/configured, then doing builds should be a matter of checking out what's needed from a source code repository and building it.
Currently I use a combination of the above to automate the process as follows:
Start with a barebones OS install on a VMWare guest
Run a shell script to install Puppet and retrieve its configs from source code control
Puppet to install tools/components/configs
Check out files from source code control to build and deploy our web application
I stumbled across this question and was very suprised that no one has mentioned Vagrant yet.
As Pete TerMaat and others have mentioned, virtualization is a great way to manage and automate development environments. Vagrant basically takes the pain away from setting up these virtual boxes.
Within minutes you can have a completely fresh copy of your favourite Linux distro up and running, and provisioned exactly the same way your production server is.
No more fighting with OSX or Windows to get PHP, MySQL, etc. installed. All software lives and runs inside the virtual machine. You can even SSH in with vagrant ssh. If you make a mistake or break something, just vagrant destroy it, and vagrant up to start over fresh.
Vagrant automatically creates a synced folder to your local file system, meaning you don't need to develop within the virtual machine (ie. using Vim). Use whatever your editor of choice is.
I now create a new "Vagrant box" for almost every project I do. All my settings are saved into the project repository, so it's easy to bring on another team member. They simply have to pull the repo, and run vagrant up, and they are literally ready to go.
This also makes it much easier to handle projects that have different software requirements. Maybe you have some projects that rely on PHP 5.3, but some newer ones that run PHP 5.4. Just install the version you want for that project.
Check it out!
One important point is to set up your projects in source control such that you can immediately build, deploy and run after checkout.
That means you should also checkin helper infrastructure, such as Makefiles, ant buildfiles etc., and settings for the tools, such as IDE project files.
That should take care of the setup hassle for individual projects.
For the basic machine setup, you could use a standard image. Another option is to use your platform's tools to automate installation. Under Linux, you could create a meta-package that depends on all the packages you need. Under Windows, a similar thing should be possible using MSI or the like.
Edit:
Ideally, instead of checking in helper infrastructure, you check in the information that allows the build to generate the helper infrastructure. This is the approach taken by e.g. the GNU build system (autotools etc.), or by Maven. This is even more elegant, because you can (theoretically) generate infrastructure for any (supported) build environment, thus you are not bound to e.g. one specific IDE, and settings in the helper infrastructure (paths etc.) don't need to duplicate the main project settings.
However, this also a more complex approach, so if you can't get it to work, I believe checking in stuff like IDE files directly is acceptable.
I like to use Virtual PC or VMware to virtualize the development environment. This provides a standard "dev environment" that could be shared among developers. You don't have to worry about software that the user could add to their system that may conflict with your development environment. It also provides me a way to work to two projects where the development environments can't both be on one system (using two different versions of a core technology).
Use puppet to configure both your development and production environment. Using a top-notch automation system is the only way to scale your ops.
There's always the option of using virtual machines (see e.g. VMWare Player). Create one environment and copy it over for each new employee with minimal configuration needed.
At a prior place we had everything (and I mean EVERYTHING) in SCM (clearcase then SVN). When a new developer can in they installed ClearCase|SVN and sucked down the repository. This also handles the case when you need to update a particular lib/tool as you can just have the dev teams update their environment.
We used two repo's for this so code and tools/config lived in separate places.
I highly recommend Blueprint from DevStructure. It's open-source and your use case is actually the exact reason we originally wrote the software. Our goals have somewhat changed, but it still is the perfect tool for what you are describing. In short, you can create reusable server configs - dead simple configuration management. I hope this helps!
https://github.com/devstructure/blueprint (Blueprint # Github)
I've been thinking about this myself. There are some other technologies that you could throw into the mix. Here's what I'm currently setting up:
PXE based pre-seeded installation images (Debian Squeeze). You can start up a bare-metal machine (or new virtual appliance) and select the image from the PXE boot menu. This has the major advantage of being able to install your environment on physical machines (in addition to virtual appliances).
Someone already mentioned Puppet. I use CFEngine but it's a similar deal. Essentially your configuration is documented and centralized in policy files which are continually enforced by an agent on the client.
if you don't want a rigid environment (i.e. developers may choose a combination of tool-sets) you can roll your own deb packages so new devs can type sudo apt-get install acmecorp-eclipse-env or sudo apt-get install acmecorp-intellij-env, for example.
Slightly off-topic, but if you run a Debian based environment (i.e. Ubuntu), consider installing apt-cacher (package proxy). In addition to saving bandwidth, it will make your installations much faster (since packages are cached on your local network).
If you're using OSX and working with Rails. I'd suggest either:
https://github.com/platform45/let-there-be-light
https://github.com/thoughtbot/laptop
If you use machines in a standard configuration, you can image the disk with a fresh perfectly configured install -- that's a very popular approach in many corporations (and not just for developers, either). If you need separately configured OS's, you can tar-bz2 all the added and changed files once a configured OS is turned into your desired setup, and just untar it as root to make your desired environment from scratch.
if you're using a linux flavor, you've probably got a package management system: thinks .rpm for fedora/redhat, or .deb for ubuntu/debian. many of the things you describe already have packages available: svn, eclipse, etc. you could roll your own packages for company specific software, create a repository (perhaps only available on the local network) and then your setup could be reduced to a single bash script which would add the company repo to /etc/apt/sources.list (debian/ubuntu) and then call a command like,
/home/newhire$ apt-get update && apt-get install some complete package list
you could use buildbot to then automate regular builds for company packages that change often.
Try out DevScript at http://nsnihalsahu.github.io/devscript .
Its one command like ,
devscript lamp or devscript laravel or devscript django . In around a few minutes ,depending on the speed of your internet co
Related
recently I tried to install an apache webserver on an ubuntu machine. I discovered two ways to do that.
The first one is to install it like it is described at this source apache docs using ./configure, make, make install to the downloaded apache sources.
And the second way is to install it via apt-get update && apt-get install apache2.
I also noticed that when I run the installation via apt-get it seems that there is a different configuration of the apache for example the directory /etc/apache2/* is only available over this way of installation. So when I install it manually the directories sites-available, sites-enabled, ... are just missing.
Is there also a way to get these folders when running the manual installation?
Where do this differences come from?
This is nothing that can be summed up in a few sentences. You basically ask: how does software management work under Linux systems? In short:
using the apt utility on a Ubuntu based system you are using the systems package management system, that should nearly always be what you want. That way the system can take care to keep your apache installation up to date, you can remove software again without leaving artefacts, the software is guaranteed to work with the system libraries already installed in your system. Potential conflicts are resolved. You can be certain that the configuration matches your system.
using the build system is a more generic (archaic) way: you do not only install the software, you build it from scratch from the software sources prior to installing it. That is only possible for OpenSource software, obviously. This certainly allows for more flexibility. But you are responsible yourself for a whole lot of things, starting with first setting up a complete build system, then configuring the package, select what you actually want to build and last but not least you are yourself responsible to update software you installed that way. This rarely is a good idea, with two exceptions:
you absolutely cannot find a pre build package for your Linux distribution or
you want to make own modifications to the software itself
The difference in the folder layout of the apache configuration is a separate thing. There are two aspects that come into play here:
you can change that layout using the uncounted build options offered by the build system (namely the options you can hand over to the configure utility on the command line).
typically the configuration of complex software packages (like the apache http server with all it's modules) is broken up into
various sub folders so that you can keep an overview and
allows additional packages to drop their additional configuration in place (this is mostly useful for prebuild packages)
Long story short: in 99,8% of all cases you want to use prebuild packages prepared for your distribution. That is the power of the software management systems under Linux (that still have no comparable counterpart in other operating systems).
I am working on a college project along with a group of people. Our goal is to add features to an already existing application that runs on the web. Currently, I'm in the process of getting the source code to run on my machine. This consists of cloning a bunch of repos, installing MySQL and some (very old and outdated :-| ) versions of Python, and running some scripts. The process sounds straightforward but it isn't; there are a lot of dependancies that need to be met for the code to run, which means that I need to spend a lot of time looking at error logs trying to figure out what package is missing and needs to be installed or downgraded. But that's not the point of this question.
I'd like to make it easier for people to pick up the project in the future and work on it without having to spend hours just to get the code to compile. I'd like to get the project set up on a Linux VM (something I know how to do using VirtualBox) and then somehow share (?) that VM so that other people can simply set it up and be able to immediately have the code compiling (something that I don't know how to do, or if it is even possible).
Additionally, I'd like to be able to do all the coding on the host OS if possible, and only do the compiling/running on the VM (something I also don't know how to do). I would like some help/pointers with all the "I don't know" 's, as I don't know much about VM's other than how to set one up using VirtualBox.
You can use Vagrant to automate the provisioning of the VM, and setup all your tools and dependencies using Docker.
There are many good tutorials and sample vagrantfiles online to get you started. There is a learning curve involved, but well worth the effort. Many companies use Vagrant to quickly provision dev environments.
Vagrant can automatically download a specific distro/version of a VM from the web if one is not already locally installed. It can also provision a Docker container, in which you can install any required dependencies, tools, etc. You can store the vagrantfile, dockerfile, scripts, etc. in GitHub for easy access by your colleagues. All they would have to do is install Vagrant and run vagrant up from the command line.
If you want to write code on the host machine and compile/test it on the VM, you will need to setup a shared folder in the VM using Guest Additions (see here). Be VERY careful with line endings if you are working in Windows and running in Linux. You can setup the shared folder with Vagrant as well (see here).
I have read what the Docker is but having hard time finding of what are the real scenarios of using Docker?
It would be great to see here your usages.
I'm replicating production environment with it, on commit on project with jenkins after building binaries i deploy there, launch the required daemons and run integration tests, all in a very short time (a few seconds over the time that takes the integration tests). Having no need to boot, and little overhead on memory/cpu/disk is great for that kind of things.
I could extend that use for development (just adding a volume where the code resides to my git repository, at least for scripting languages) to have the production environment with the code im actually editing, at a fraction of what virtualbox would require.
Also needed to test how to integrate some 3rd party code into a production system that modified DB. Cloned the DB in a container, installed the production system in another, launched both and iterated the integration until i did it well, going back to zero to try again in seconds, and faster, cheaper and more scriptable than doing it with VMs+snapshots.
Also run several desktop browser instances on containers, with their own plugins, cookies, data storage and so on separated. The docker repository example for desktop integration is a good start for it, but planning to test subuser to extend this kind of usage.
I've used Docker to implement a virtualized build server which any user could ask to run a build off their personal git branch in our canonical environment.
Each SSH connection made to the server was connected to a new container, ensuring that all builds were isolated from each other (a major pain point in the past), ensuring that the container's state couldn't be corrupted (since changes were all isolated to that single instance), and ensuring that even developers on platforms such as Windows where Docker (and other tools in our canonical build environment) couldn't be run locally would be able to run builds.
We use it for the following uses:
We have a Jenkins Container which we can use to bring up our Jenkins server. We mount the workspace using volumes so we can migrate the server easily just by copying the files and launching the container somewhere else.
We use a Jetty container to easily deploy our war files in our production and development environment.
We use a whole host of other monitoring tools such as Uptime which we have containers for so that we can bring them up and down on various hosts with a single command.
I use docker to build and test our software on several different Linux distributions (RHEL 4/5/6/7, Ubuntu 12.04, 14.04).
Docker makes it easy and fast to create minimalistic and consistent build environments.
Docker gives you the benefits that other virtualization solutions give you to a fraction of the recourse needed.
In a project we a forced to use IBM RAD and Webspher Application Server (6.1).
Setting up the development environment is currently described in about 10 pages of wiki documentation and takes about a day if you don't do any mistake. The main parts are:
Installing the IBM Installer;
Use it to install RAD
Install a patch to the Installer;
use it to install half a dozen patches to RAD
create a network drive pointing to ...
checkout project source to ...
install WAS
configure the a WAS instance with two jdbc drivers, 6 datasources, a queue ...
I think you get the idea
I'd like to automate that process (or at lest 95% of it) to something like.
start script x.
On prompt enter a directory with at least yGB of memory available.
Get yourself a cup of coffee
start working.
What are the proper tools to get this working? Should I use something like puppet and chef? Or is that overkill and I can just zip the installation directory and change 2 registry entries?
Has anybody experience with this? Any pointers to get started?
You can script the configuration of WAS using wsadmin:
http://pic.dhe.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=%2Fcom.ibm.websphere.base.doc%2Finfo%2Faes%2Fae%2Fwelc6topscripting.html
It is some effort to learn how to do so but in the end it saves a lot of time. You need to use Jython or Jacl to do so.
WAS profiles can be created headless with a response file. Use manageprofiles.bat in bin directory of WAS to do so.
Regarding RAD installation you can install the IBM Installation Manager version you need to install the patches right away and then install everything in one shot. Add the fixes you need as Repositiories right from the beginning. The fixes will be installed instead of the old versions in this case. You should have the base images and all fixes on the local disk to do so.
The installation of RAD itself can also run in headless mode but I don't have any experience in doing this.
The configuration of the RAD workspace is the next thing you want to automate. This is not so simple to do. The simplest thing you can do is to export the workspace preferences of a workspace that contains all settings to an eclipse preference file (.epf). File -> Export
This is not a complete solution but may help you a bit. Be sure to keep all settings in just one file and import that into a fresh workspace.
Use Notepad++ TextFX plugin to sort the settings in the epf file. You can then figure out which settings you need just by looking at them.
More control over the workspace settings and automated conifiguration requires accessing eclipse internal APIs and some coding.
Regarding the the project sources it depends on the SCM you are using.
I want to play around and check out Apache's couchdb as a possible back-end for a web-app that I am designing. Therefore I want to have an instance of couchdb, but also to be able to throw it away when the testing is done. The development computer is an Ubuntu laptop (not server). The problems are:
The Ubuntu repository has couchdb 1.0, but the couchdb website strongly recommends to install 1.1, built from source.
I have Erlang built and installed from source, because the Erlang distro from the repository is defective. I don't see the point in installing another Erlang aside it.
couchdb has a lot of dependencies, including a whole bunch of perl libs, that I really don't need, and prefer to throw away when I'm done.
So I am looking for a way to either:
Install couchdb 1.1 as a package that can be easily uninstalled, or
Build couchdb from source, with as few as possible installed dependencies, so when I'm done I can just delete it. Preferably, do this without building another Erlang distro, but configuring it to use the existing one.
Is any of these possible, and how? Thanks in advance.
Btw, I am aware of the build-couchdb project, but from what I read, it requires installing all the build dependencies in advance, which is undesirable, because it will leave a whole bunch of dangling packages in my system, without being a dependency of a couchdb package. It also fetches a copy of Erlang, which is redundant for me.
(Dear moderators: This questions combines issues that relate not only to programming, but also to server administration, Unix software, and, particularly, Ubuntu Linux issues. Therefore, it might be suitable for a few other stack exchange sites. I recon it is most likely to be answered here, since this kind of hackery is often done by programmers. However, if I am wrong, feel free to migrate it, and I apologies in advance for your troubles.)
You could install CouchDB into a chroot jail
A chroot is a way of isolating applications from the rest of your computer, by putting them in a jail. This is particularly useful if you are testing an application which could potentially alter important system files
From the Ubuntu instructions on creating a chroot jail
Another option, assuming your laptop has the appropriate hardware virtualization support, is to use KVM.
The KVM option might be more helpful in the long run as you could move the VM's disk image onto a server.