I was looking for a uniform configuration management tool for remote installation of OS on remote servers(similar to puppet/chef) having wide range of platform support. I think we can use PXE/kickstart for remote installation. I am not sure that can be used to install OS on multiple servers in parallel? Other way to spin up the EC2 instance from AWS and pay amazon for the usage. I was wondering is there any other best option for this requirement?
Regards
Bubunia
You can consider ansible as a strong candidate for this.
Some of its features:
Open source with large development community
Number of modules which can help you building flexible solutions.
Cloud focused development modules
Ansible inventory which can help you automate things end to end on the basis of tags to your instances
Agent less
Easy to write, read and understand yaml format
Pre-builded modules for multiple installations available in open source community
Work with multiple OS
It is efficient as well I am using it from last 1 year and found it very good.
Sparrow6 cm supports quite a range of platforms/os. You can choose Sparrowdo to run configuration jobs in push manner over ssh.
Related
I would like to know if there is any open source load generator for stress testing. I am using JMeter script to run the test. I know there is one called flood.io but think it is not free. Can you please recommend me some option and documentation on how to use it properly? Thanks!!
Well, JMeter is open source, you can observe source repositories so it matches your needs.
The point is where to run JMeter so you need to have your own bare metal machines (or normal desktop/laptop/virtual machine). If you're looking for an open-source operating system there is a wide range of free and open source Linux distributions, FreeBSD, illumos, etc.
If you don't have any machines and any money to invest into these machines there are some free cloud options like AWS Free Tier or Oracle Cloud Free Tier, also some SaaS service providers offer free pricing plans, for example BlazeMeter gives you 50 virtual users, LoadFocus gives you 20 concurrent users, etc.
I like the Docker Hub with dockerfiles idea very much.
Is there a similar way to get a small working linux VirtualBox instance in a few commands, that could also be controlled from a command line?
Vagrant is a great tool that does just what you want and much more! It's a ruby application written for fast and simple setup of minimal development environments.
By default it creates VirtualBox images, but it supports VMWare and many others too. The whole setup of a box is managed by a single Vagrantfile! Your vm options, network settings and provisioning is done there.
Setting up a virtualbox box is as easy as executing just two shell commands. Checkout the Getting Started Guide for an example using Ubuntu.
You can use a vast range of prepared images from the Hashicorp Atlas or build your owns.
Also, vagrant doesn't limit you to one virtual machine per development setup, it enables you to model cluster setups on a single machine using multiple vms. I myself use docker for that part though.
Edit: fixed a typo :<
Azure and EC2 are optimized for running servers. Lots and lots of servers. Both platforms attempt to manage tons of things for you -- in Azure's case, it wants to manage even the target operating system.
However, I'd like to use such a service for a different reason: Testing.
I've got a ton of operating systems I need to support. My tests don't actually take that long, but running them on every platform is time consuming. I was going to just use a cloud service for this, thinking that these machines would be running for much less than an hour, and it wouldn't cost all that much.
The problem is that the major cloud services won't run client versions of Windows -- Windows Server only.
Is there a cloud service which would let me run every client and server version, and every service pack level, of Windows released starting with Windows 2000 SP4 to the present day?
Try CloudSigma, Defiantly can upload your own ISO's and run any x86 and 64bit OS you like on it. They have their in-house versions to get started but you can bring your own OS versions.
Based in Switzerland but they would have also the servers in the US, performance i've expected to quite good.
https://www.cloudsigma.com/
There is also a free trail on at the moment
https://cs.cloudsigma.com/accounts/signup/
The list of Open Virtualization Alliance members may have some candidates for you.
A search on the page for "operating system" suggests the following possibilities (in addition to the already-mentioned CloudSigma):
ElasticHosts
stepping stone GmbH (I'm less sure about this one)
Sublime IP
No, commercial cloud services like Azure and Amazon EC2 are themselves virtual, so you don't get a great deal of control over the operating system.
An option may be to consider renting a full physical server (colocated, or managed) and then use a battery of virtual machines to run the tests. Something like VMWare's snapshot feature sounds perfect: spin up a clean virtual machine, deploy the test code, then throw away changes to the disk once the tests have been completed.
Or, indeed, as #Stuart suggests - run the tests locally.
This definitely isn't something Azure offers - I think all of Azure's images are based near to Windows Server 2008 R2.
For EC2 you could set up images for Server 2003 through to 2008R2 - but nothing else. There are also some services out there to assist with this - e.g. VaasNet http://www.vaasnet.com/catalog
For testing the other Windows operating systems, I simply don't think there's a cloud service available to let you do this. I don't even think there are any cloud services where you can run "Virtual PC" type applications on top of the hosted operating system - as I think most of the virtualization APIs are disabled in the cloud environments (virtualization within virtualization not supported!)
Sorry to say this, but your best bet may be local test hardware running VirtualPC images.
It appears that the Xen Cloud Platform might do what you're after. This page ends with:
Guest Operating Systems: the XCP binary distribution is delivered with a wide range of Linux and Widnows guests. Check out the release notes for a complete list.
And their PDF document Xen Cloud Platform Virtual Machine Installation Guide (Release 0.1, Published October 2009) says that Windows 2000 Server has "No known issues."
(I don't have any affiliation with Xen)
In conjunction with the above, there is also a list of Xen VirtualPrivateServerProviders, several of which say they include Windows.
Buy time on an EC2 instance and use it to host VirtualBox VMs with VMs set up for each operating system you want to test for. Use a RDP client or VNC or some other means to control the guest OS. This forum post seems to point to that being possible. But yes it is not a cloud service itself and you would have todo some initial setup and configuration work yourself.
What tools or strategies are you using for automation of EC2 activities?
I need to be able to bring up a number of EC2 instances, provision various software to it (primarily Python packages), interact with S3 (primarily download data), and run various jobs. I'll be doing this both on-demand and on a scheduled basis.
I'm trying to decide if I should:
Create an AMI with all my software loaded on it
or
Launch a plain vanilla linux AMI instance and scp my software to it
For the provisioning and automation Boto looks pretty good. Or I could write something with Paramiko. Recommend either or anything else I should be looking it?
Basically I'm looking for advice / success stories, let me know what's working for you.
To answer your bullets about selecting AMIs, I would say that it depends on how much software you're installing.
I have been successful with a hybrid approach, where I build an AMI and load my heavyweight and more stable software. This is the stuff that needs to run an installer, or takes considerable time to install (remember that if you re-install a package every time as part of your startup process, you're paying for the install every time). Then, I upload the small and volatile software at provisioning/startup time. In this bucket goes most of the application code, data, etc. That way, I can change my app and not have to touch the AMI.
The benefits of this approach:
Don't have to pay for running the same software install thousands of times.
AMI can stay fairly stable over time.
Can use software that requires intervention or GUI interaction to install.
Major drawbacks:
Your AMI's OS version will become stale over time.
Your AMI may not be flexible as to the instance type/architecture it will run on. For instance, you may create it on a 32-bit OS and thereby prevent it from running on the High CPU instance types, or vice versa. So you may lock yourself into a pricing scheme.
I don't use Python, so I can't comment on either of the APIs you referenced.
AWS just released the Systems Manager suite, which includes an Automation service that will (among other things) handle your use cases around AMIs.
This question was asked some time ago now but I believe my answer could be useful to other users. I believe the best automations tools available on the market are provided by Cloud Management platforms. For example they offer auto-scaling, configuration software integration (Chef/Puppet), databases replications, dns management...
The most popular cloud management softwares are Scalr (disclaimer: I work there), RightScale and enStratus. Scalr is open-source and released under the Apache 2 license.
Regarding your specific question on AMIs, cloud Management platforms usually provide pre-configured AMIs (at Scalr, we call them roles). If you want to create your own AMI built on an existing instance, you'll be able to create snpashots and use them as a template for future instances.
What are the key use cases for the use of virtualization -- that is, running one or more "virtual PCs" using software such as VMWare and Microsoft Virtual PC -- for software development?
Also -- are there other instances/uses of virtualization that aren't covered by my definition above (use of a tool like MS Virtual PC or VMWare), and that are useful to developers?
My impetus for asking is this StackOverflow comment by Metro Smurf asserting "You'll wonder how you ever developed without it!", regarding use of virtualization.
(Please include just one use case per response. Thanks!)
Application testing in multiple environments is one obvious use of virtualization that I'm aware of. Testing your application on other operating systems (without requiring additional physical computers to do so), as well as testing that involves software that generally only allows you to install a single version on a given machine (such as the Internet Explorer browser; running both IE6 and IE7 on the same machine is not an officially supported configuration), are good candidates for virtual machine usage.
If your build-server is running in a VM, you can make a snapshots of it for every software release in order to be 100% sure that you can recreate the build environment (in case you want to make patches to old releases, for example).
If you set up snapshots of your development environment (and back them up) it can be very easy to resume productivity if your computer breaks down. When your machine breaks down right before your release - and you can resume immediately with all your tools installed and configured, it can be a lifesaver.
The simplest case which applies to my current situation is that we have a complex client-server environment and with virtualization every developer can very quickly get a baseline set of operating systems to deploy their local build to and verify end to end functionality.
Locally you have your dev box, and N client boxes which get re-initialized as fresh OSes each time you want to try a build. Essentially it's the test environment equivalent of a 'make clean' where even the client workstation gets replaced with a new OS.
Quickly distributing environments between team members is a very nice use case to for virtualization especially if you have a lot of various components, tools, etc.. This can save you a ton of time with new hires, contractors, or other individuals who need an environment quickly.
Many presenters use a VM for presentations - it allows them to revert immediately to reset the presentation for the next day, transfer all presentation materials quickly between computers, and not have to show your attendees your messy My Documents folder.
Using virtualization for sales activities is also a great use case. You can take a snapshot at a particular time that you can save as your demo baseline. Then once you run through the demonstration and change the data, etc. you can restore back to your previous baseline for future demonstrations. You can also capture multiple baselines and pick and choose which baseline best fits the upcoming demo.
Test environments. If you have more than one setup that a system needs to be targeted for (e.g. Windows & Linux, XP & Vista) then a machine with lots of RAM and VMWare (or on of the others) is a good way to manage the environments.
Another is developing on one system and targeting another. For example, at one point I did some J2EE work on a workstation running Linux where the client was I.E. 5.5. A VM with Windows 2000 and IE 5.5 would let me test the application.
Reasons I use virtual machines for development.
Isolate different development environments.
Testing environments.
Easy recovery due to computer hardware failure/upgrade.
Ability to "roll-back" changes to your development environment if something corrupts it.
Currently, I am using VirtualBox for my VM setup. I used to use VirtualPC, but I REALLY hated not having any type of "snapshot" feature (like VMware and VirtualBox have).
We develop software for use in our SaaS application, our production environment has a large number of servers and their software environment needs to be absolutely predictable; we can't have ANYTHING installed extra, or absent from our development machines.
Moreover, our application requires a number of different server types in order to function properly (at least 7 last time I counted); mostly they can't be installed on the same (virtual) machine - at least, not without violating the "same software as production" requirement.
In order to have a consistent environment, it's necessary to use VMs. I don't know how anyone ever manages without them.
Snapshots and rollbacks are nice too, but I use them only occasionally (really useful during installation / upgrade tests).
Suppose you're developing a new version of your software, and checking that the upgrade from the previous version works correctly... how long does it take to do a test cycle without being able to rollback the box? Do you have to reinstall the OS then the old version? Can you guarantee that the uninstall really uninstalls everything?
Being able to test/retest your deployment process is a huge savings.
Developing Add-Ins for different versions of Microsoft Office (using Visual Studio Tools for Office).
My main work machine has Office 2007. When I work with Add-Ins for Office 2003 I use a virtual machine with Visual Studio and Office 2003.
I'm suprised that nobody has mentioned the VMware record/replay feature (awesome video demo) which is great for debugging.
I have a headless server running ESXi which runs various machines for building installers (so I don't have to give up processing power on my desktop), automated testing (server is faster than any desktop) and various test environments (about 20 different configurations) so that the support team can easily jump onto a configuration that closely matches a customers system.
When you have one really beefy server running VMs that can be shared between support, test and dev teams, you introduce huge cost savings. In all, we're running ~25 VMs on ESXi (dual-quad core Xeon 2.5G + 8Gb RAM) shared between 5-10 people, some of the developers use Virtual PC and then I use VMware Workstation on my desktop. All of the Mac users here use VMware Fusion as well
I am surprised that no one has mentioned the benefit of increased security by isolating, for example, the database server and web server in different VM's.
Some server applications can use VMs too. When one vm is not used much, the server can locate the resources to other vms.
Some sort of test environment: if you are debugging malware (either writing or developing a pill against it) it is not clever to use the real OS. The only possible disadvantage is that the viruses can detect that they are being run in the virtualization. :( One of the possibilities to do it is because the VM engines can emulate a finite set of hardware.