Automatic packing of server-side product as Docker and OVA image - virtual-machine

We develop a server-side solution and to ease its deployment we would like to provide our cutomers with two options:
1. Docker image
2. VM image in OVA format
The images should be automatically created by our build machine.
As of today, we use packer for this purpose. First we create docker image and then update that image in preconfigured virtual machine image (using 'virtualbox-ovf' builder). This works pretty well, but there are some problems with this solution.
First, our vm includes docker framework and two OSes (host's and docker's), so our VM image is ~twice bigger than docker. Second, to base our solution on another linux distro, we should manually configure new VM machine.
We are looking for 'Dockerfile'-style solution to create and configure VM automatically and then export it in OVA format. 'virtualbox-iso' builder is the obvious way to do this, but the building process will be much longer.

If you are willing to use Debian as your base OS then you could look at TurnKey Linux's TKLDev. It's probably a bit of a learning curve initially but it's a pretty cool thing IMO (although I'm very biased - see below disclaimer). TKLDev will build you a TurnKey (Debian based) ISO with your software installed on top. Then using Buildtasks you can convert the ISO to OVA, VMDK, LXC, Docker, OpenStack, etc...
Unfortunately Buildtasks is not very well documented but significant chunks of it are in bash so if you are handy with a Linux commandline you could work it out. Otherwise ask on the TurnKey forums.
The initial development (from Packer to TKLDev) may take a little while, but once the heavy lifting is done the creation of an ISO (in a guest VM on a moderm multicore CPU PC) takes about 10-15 mins and the OVA probably another ~5; Docker another ~5.
If you wanted to make it build automatically then you could use a hook to trigger a fresh TKLDev build (including the buildtasks image creation) everytime a commit was made to a repo. I know that git supports this but I assume that other version control systems allow something similar.
Also if the appliance that you are making is open source then perhaps it could be added to the TurnKey Linux library?
Disclaimer: I work with TurnKey Linux. :)
FWIW this is essentially the process we use to create our library of appliances in most virtualisation formats known to human kind!

Related

Vagrant and / or Docker workflow with full OS X filesystem integration for seamless local feel?

Recently I've been dabbling with vagrant and docker. These are quite interesting tools, but I haven't been able to convince myself that it's the way to go quite yet on my OS X machine. Being an old Unix hat, I have to say that I like having a consolidated and sandboxed environment for development purposes.
I've seen a lot of chatter and a number of friends have been using vagrant with just stock vim for editing. I'm not really a fan of that approach and would probably prefer to use the vm provider's sharing mechanism OR, more likely, NFS.
Personally I'd like to be able to edit directly in TextMate, SublimeText, Emacs (on OS X), or even perhaps use RubyMine and its various IDE features, etc.
Is there any way to really get the workflow down so that such an environment will be essentially like working on a local environment without having to pull a lot of additional background strings to make things work out?
I suppose a few well placed scripts could go a long way, but I've not found any solid answers on really making this a seamless environment.
What actually worked for me was to use boot2docker which makes it easy to install a lightweight virtual machine (with VirtualBox) that will host your docker deamon and images. The only thing you need in order to run docker commands is to run $(boot2docker shellinit) when you open a new Terminal.
If you need to also have your files on an OS X folder and share them with a running docker image, you need some additional setup, but once you do it, you won't have to do it again.
Have a look here for a nice walkthrough on how to do it. The steps in short are:
Get a special boot2docker image that allows you to use shared folders for VirtualBox
Configure VirtualBox to share a folder:
VBoxManage sharedfolder add boot2docker-vm -name home -hostpath /Users
This will share your /Users folder with the boot2docker image that hosts docker.
From you Mac share the folder you need with a folder in a docker image like:
docker run -it -v /Users/me/dev/my-project:/root/src:rw ubuntu /bin/bash
One small annoyance that I haven't found how to overcome is that you do not longer access your software through localhost because it actually runs on boot2docker instance. You have to run boot2docker ip and access that ip.
Hope that helps!

Does VirtualBox have any advantages over VMWare Player?

I've been using VMWare Player for ages now for both Windows development on my Linux box and (more importantly) automated testing of Windows applications.
Basically what I do is to:
have my development VM running and I build my code and automatically transfer the install package to Linux.
when this shows up at Linux, automatically copy a "known-state", snapshot VM to my test work area (I say snapshot but it's really just a backup copy of the whole directory, not a real VMWare snapshot).
also automatically start the VM in the work area once it's copied.
the VM has a single never-changing startup script which pulls a real startup script from Linux and runs it.
that startup script is responsible for getting down the install package and doing a silent install.
it then runs a test suite and uploads results back to Linux where I have automated scripts which check them.
So, it's basically a one-button test process.
Now I notice more and more people seem to be using VirtualBox.
First off, I'd like to confirm that it can also do a similar thing, primarily being able to backup and restore whole VMs and having shared folders between VirtualBox and Linux.
Secondly, and this is the crux: I'd like to know if that has any concrete advantages over VMWare Player, especially for the automated testing jobs.
I switched to VirtualBox because of one concrete advantage, I wasn't able to setup the network as I wanted to in player. I don't remember if it was bridging or port-forward or whatever that didn't work, but something didn't work the way I wanted it to with the network-setup (cause I needed the pay-version for that) and thus I switched. Personally I've found that both have good and bad sides, but I still use virtualbox cause of that network-thing.

Suggestions on Setting up Development Environment

After referring to many posts, opinions and feedback from SO, I have just bought a Lenovo Thinkpad T410 . x64 Win 7 Prof, 500 GB # 7200 RPM , Core i7 620M processor, 4GB DDR3 RAM.
I am now setting up my development environment on the new machine. I need your suggestions in setting up a clean, structured and risk-free Development Environment.
Something about what I intend to do on this machine:
I am Entrepreneur bootstrapping my Startup. So I will have business related purposes (presentations) other than Coding.
I do coding on Microsoft stack currently for some of my other projects. But will start coding in other technologies such as RoR. So I need to have MS products (VS, IIS) and other OSS'
This machine also doubles up as production environment on top of Development Environment.
I don't have a separate Desktop for doing heavy lifting. This is my whole and sole workstation.
I have read a lot about VMwares here and how they help to keep the machine clean and ordered which you can just wipe out clean and have reinstalled as you wish. Is it a good thing to have VMs each for Microsoft stact, RoR stact and so on or have all of them installed on my main machine itself.
Also, apart from this, it would be great if someone can suggest some good options for Firewall+Antivirus+Malware stack (considering that this is a Win 7 machine)
ThanQ
This machine also doubles up as
production environment on top of
Development Environment.
You are setting yourself up for a world of hurt.
Aside from that, it is just common sense that you should use some form of VCS (I recommend Git) and store all your code NOT on your development machine. You should be able to checkout out your code, run a script, and be up and running. You are bootstrapping a startup; pay the $12/month to keep your code safe.
VMs are a good idea if you need to support different environments, for example Win7, Win Vista, and perhaps some flavor of linux. If you take my suggestion and use a remote VCS setup, you can checkout from the remote source onto your VMs.
Another benefit of VMs is you can set up a base install, with all the software you need, and create an image of it, so from that point on you can pass the image around, and you won't need to install the baseline software again.

Best setup for Linux development from Windows? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
What's the best setup for developing Linux apps from a Windows workstation? Right now I'm connected via SSH to our Linux development server and am using Eclipse, forwarded over SSH via PuTTY, to the public domain version of Xming running on my Windows workstation. It works, but it's not great; Eclipse's response times are far from snappy (noticeably worse than Eclipse running natively on my much slower Windows workstation), I can't resize some dialog boxes, and I haven't figured out a good way to reconfigure my fonts.
Is there a better setup available?
Edit: This is for C/C++ development.
Options for Linux on Windows:
Tools Only
Given you're using Eclipse I'm going to assume you want a full IDE, but if you can get by with just the GNU/Linux tools, there are a few choices.
cygwin gives you a bash shell with lots of tools, including an X11 server. This has been around awhile and is mature.
msys is a smaller, lightweight alternative to cygwin.
GNU utilities for Win32 is another lightweight alternative. These are native versions of the tools, as opposed to cygwin which requires a cygwin DLL to fake out its tools into thinking they are running on Linux.
Linux in a Windows Process
There are several packages that will run Linux as a Windows process, without simulating an entire PC as virtualization does. They use Cooperative Linux, a.k.a. coLinux, which is limited to 32-bit systems. These don't have the overhead of virtualizing, and they start up faster since you're not booting a virtual PC. This is a little more on the experimental side and may not be as stable as some of the virtualization options.
Portable Ubuntu
andLinux
Virtualization
Virtualization software lets you boot up another OS in a virtual PC, one that shares hardware with the host OS. This is pretty tried-and-true. There are nice options here for taking snapshots of your Virtual PC in a particular state, suspend/resume a virtual PC, etc. It's nice to be able to experiment with a virtual PC, add a few packages, then revert to a previous snapshot and "start clean".
VMWare
VirtualBox
VirtualPC
In my case...
Sounds like your environment has different performance characteristics, but here's my situation: I started out with Eclipse on my Windows laptop (doing Rails development), found this sluggish, and switched to using putty to ssh into a fast Linux box. I do my editing via an emacs running on the Linux server, displayed on Windows using Xming. Or I use native emacs on Windows, editing the files shared via NFS. The latter is slower in my environment due to sluggish saves.
When working from home, I ditch X because it is too slow with remote clients, and just run emacs -nw within a putty window. I then use GNU screen so that I have multiple "windows", and so that I can easily resume where I left off if my network connection flakes out.
The best approach that I've found is to:
keep your code portable
develop natively on your desktop
verify any OS dependencies (minimize these as much as possible)
deploy to your target regularly, test & debug there
I know that this isn't a direct answer, but using an IDE for development through X is painful with most of the free tools. The only way that I've been productive doing work this way was when I was running a UNIX-like on my desktop so X was native. If you are going to use this approach, try a commercial X solution on the desktop.
Other than that, consider ditching the IDE and doing your development and debugging via SSH, a terminal editor (e.g., vi, pico, ee, emacs), make/ant, and gdb.
The best approach for you is going to be driven by your programming language and the type of application you're developing. If you are doing GUI applications, then using X might be the only approach that is acceptable. If you are doing back-office/daemon development, then the SSH and terminal approach will probably work though you probably want to get really comfortable with either vi or emacs.
EDIT: just noticed that you are doing C/C++ development. Consider using a cross platform framework if you aren't already. Using something like Qt, APR, ACE, or Poco should make it possible to natively develop under Windows with a deploy/debug step to your Linux environment.
For development I usually use a Linux virtual machine on my Windows box. It will probably send Linux users running to the bathroom to wash their hands, but I do all of my development in Visual Studio, and I have a custom Visual Studio plugin that invokes G++ through the virtual machine and pipes the output into the VS output window. With a quick change of a Combo box I can build and test for Windows or Linux.
An easy to setup option would be to run Eclipse natively in windows but deploy the code via a Samba share on the Linux machine (which you can mount as another drive) (or SSH/SCP if SMB is not an option) and then run it there via SSH console.
Another easy to setup option is to simply develop on Linux via freenx or a similar tool instead of a full blown X session, check this answer: https://serverfault.com/questions/11367/remote-desktopping-from-windows-to-linux/11372#11372
The other options (Virtualization, Linux running inside windows, Cygwin) are indeed valid but have their drawbacks, like being more machine demanding, harder to setup, or not equivalent enough to the actual linux environment, but may very well be worth your while if you have the machine and the scenario justifies their use.
Doing everything on the Linux side will always have some drawbacks
if your machine is Windows.
I personally have a Linux box where everybody else has Windows and
do Windows dev inside a VM, but it has costed me a lot of RAM and some network setup pains.
I find coLinux tremendously helpful when developing on Windows for Linux, it's basically a linux system running in parallel to your Windows OS (i.e. as a service) and can be configured to simply show up on your LAN, basically like a virtual machine does. Also, it's much more full featured than CygWin, and its performance is really remarkable - I can easily run non-trivial stuff under coLinux, and still run simulators at 90+ fps.
Also, coLinux can be easily set up to run X11 and window managers like gnome/KDE, so that you can for example use something like vnc to access your linux desktop.
Cooperative Linux is the first working free and open source method for optimally running Linux on Microsoft Windows natively. More generally, Cooperative Linux (short-named coLinux) is a port of the Linux kernel that allows it to run cooperatively alongside another operating system on a single machine
. For instance, it allows one to freely run Linux on Windows 2000/XP, without using a commercial PC virtualization software
such as VMware, in a way which is much more optimal than using any general purpose PC virtualization software.
(source: colinux.org)
There are multiple solutions, I'd recommend No. 1
A VM (Virtual Machine) running a flavor of linux as a guest operating system inside Windows. Start with VirtualBox which is free.
To make managing it easier you can use a tool like Vagrant. Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity. So you code in your Windows PC and compile/run the application on a Linux system using Vagrant. Vagrant is free! Similar tool: Docker can be used too. For this setup you can use any IDE, I'd recommend VSCode its quite handy for C/C++ with intellisense but Eclipse should work too.
Web based tool like Nitrous.io which is discontinued, but you can host your own open-source version of the Nitrous IDE called Nitrous Solo which lets you host your own instance of the Nitrous IDE on your preferred cloud provider.
Windows 10 provides provides Windows Subsystem for Linux, try using that to compile and run your project. This requires a 64-bit version of Windows 10 Anniversary Update or later (build 1607+).
Cygwin / MinGW are popular bash tools for Windows, they might be able to compile and/or run your application.
Cygwin might be helpful.
I've done what you want to do for exactly the same reason: full control over the output (you're having font issues with your current solution) and much slower Windows machine than the remote Linux development box.
Most answers are bogus: having a "Linux development environment" is not just "having an IDE". It's about having the whole Un*x power at your fingertips.
Is it a local or remote Linux server? bandwith issues? Because on a LAN, even an old 100 MBit/s LAN, FreeNX flies. How's the load on that Linux server?
Setup the free FreeNX on the Linux system, install the free FreeNX client on the Windows machine and bingo, you've got your Linux development environment at your fingertips.
FreeNX is much more efficient than VNC, it's night day (VNC is actually pretty bad perfs wise, even compare to Windows's Remote Desktop... But FreeNX flies).
Regarding speed, a long time ago, I set up my main Linux workstation (it was a Pentium 4 / 2GB of memory back in the days) on which I was developing full-time using IntelliJ IDEA (another IDE), to serve a full X session (complete with a window manager etc.) that another developer was displaying remotely to... run another IntelliJ instance (and access all the Un*x niceties). It was on a LAN 100 Mbit/s and it was as if the app was local for the other developer.
Anyway, on today's hardware I cannot imagine how this could not work: I now have here a Core 2 Duo / 4GB of ram as my main desktop and a Gigabit LAN.
Such a setup was working perfectly 4 years ago, it would work perfectly today.
Now if you tell me you have bandwith issues or that the Linux machine you've got your account on is under heavy load or that it's not on the LAN, then things may be different...
How the younger developers who want a powerful Un*x system do it at the company I'm consulting for nowadays (that only has Windows desktops)? Most of them bring their shiny MacBook Pro and use that to develop ;)
I'm using xming as well and suffer from the same problems with Eclipse. Apparently, neither switching to cygwin makes it fast enough. Eventually I switched to developing in vim via xming. It doesn't take as much time as I feared to get used to all the key combinations, and the performance is absolutely smooth. Actually, now sometimes I use vim even when working natively.
Either a Virtual Machine with a Linux-based dev environment, or a local copy of some toolchain-agnostic IDE (e.g. Notepad++, with testing done via MinGW or CygWin as far as you can), or just write in Notepad++ and keep uploading to your dev machine and testing there, which is what I do.
You might try other X servers on Windows such as xwin32 and hummingbird. Note that these are commercial implementations.
Another solution is to install a VM server on your Windows box and install Linux on the VM. Options include VMware (non-free) and Microsoft Virtual PC (free download). VMware is much nicer than VirtualPC (64-bit support, more incentive to support Linux client OSes, etc.).
EDIT: In the last 13 years since this post was originally made, Cygwin/X (and Xming) has gotten a lot better. It's worth trying again. I now use it for my everyday work again.
You could take a look at setting up a svn server on the linux box and then using something like TeamCity todo a build on commit. You could write your code locally and do a commit when you want it to be compiled.
I don't know if there's a more modern route, but the standard way in my time was to run X Windows in Microsoft Windows, that way you can run any number of applications on your Ubuntu machine and control them and display them in Microsoft Windows
Check Check out.
You could try using any of the linux distros for windows, even windows-store have ubuntu, SUSE etc for windows and this could help reduce your coding efforts. This linux distros contain linux shell, kernel etc so you won't be needing linux system everytime debugging or testing your code.
You could also use Visual Studio Code which is far better and fast compared to eclipse and is even supported in linux and mac.
Check this for ubuntu distro on windows store.
Linux distros can also be downloaded from other sources but microsoft urges to use the one from Windows-Store.
Use Linux! I usually have the other problem: developing win under linux.
There is no reason for not doing so: I have win running on a virtual box now almost all the time.
Linux comes with a lot of development tools.
The problem is:
is it a graphical interface?
If no you will have no problems as soon as your code STD/portable.
(X allows you simple stuff too but for an nice application today you need a bit more.)
If Yes then you will have a lot of problems when you actually port the code
on the running platform.
Is it supposed to be portable/exchangeable between linux and windows?
if not, just develop on the native OS. Way less pain. You have Eclipse for both
platforms. Even if you think to port the code on a later stage,
just do the work for one first.
I developed a couple of graphical application under linux which are actually right now
used only under windows. My recipe is: GTK/GNOME. I made it running with cygwin and mingw.
But I guess that Qt has the same usable environment too.
My code went on win with no changes!
[ok.. a couple of touching on file paths... but was a bug..]
There is no way to develop under win and hope to be running on linux unless you are sure
not to use any win libs. That is: in a graphical application almost no chance. Or a lot of
checking... Or you will not be using any win facility. Forget Visual Studio.
Check indeed wine and the winehq pages.
Unless the problem is another, like: using team sharing facilities, or svn or whatever.
Which is not a code development problem but a bit more on the organizational side.
Bottom line:
It is way easier to port a free code on win then a proprietary code on the free market.

Automate CentOS installation with VMware for testing

Is is possible to automate the installation of an OS using VMware or any other virtualization product?
One of our products consists of a customized version of CentOS that installs the OS and our application on a server. It's much like any CentOS/RHEL installation where you choose a mode that corresponds to different kickstart options, and then you choose your keyboard type. The rest of the installation is automatic.
What I'd like to have is an automated system that will create a new guest VM, boot it with the ISO image of our product, start the installation (including choosing the keyboard), wait for the reboot, and then launch a set of automated tests.
I know that there are plenty of ways to automate the creation of new VM guests from existing templates/images, and I know you can use the VIX API to interact with virtual machines, but the VIX API seems to require that VMware tools is already running (which won't be the case when you're booting from the CentOS install disk).
This answer (Automating VMWare or VirtualPC) indicates that you can script VMware to boot from an ISO that does an unattended installation, but I would really like to test the same process that our customers will be using.
Another option might be to use Xen's fully-virtualized mode and see if scripting it over the serial port will work.
TIA,
Jason
I have a very very similar question, it is on superuser:
https://superuser.com/questions/36047/moving-vmware-os-image-as-primary-os-on-a-system
You can also use VirtualBox instead of VMWare. The VirtualBox SDK allows you to directly control the keyboard, the mouse the serial port and the parallel port of the guest without the virtualbox guest tools installed.
Unfortunately it doesn't offer a text console interface but the serial port can be connected to a local pipe file and that can probably be worked with just as well.
This may not be exactly what you need:
I have done something similar with a Ubuntu-based install. We used preseeding (Debian's form of kickstart), to answer all the questions during the install - providing the preseed file and the installer via tftp.
In addition to the official Ubuntu mirror we added the apt-server with our own packages in the preseed file. We put a .deb version of vmware-tools on the apt-server and added it to the packages to be installed.
The .deb of vmware tools just contained the .tar.gz and a postinstall script that would extract it to /tmp and run the vmware install script (which has a switch to be run unnattended, so it does not ask any questions).
So after the reboot vmware-tools were up and running and we could use vix to script the rest (which was not very reliable).
If you should encounter problems with running vmware-config.pl during boot, you could make a custom package that just extracts the tools and an init script that installs them on first boot, disables itself and reboots.
Maybe you can use this strategy (replacing apt by yum, preseed by kickstart and tftp by a remastered iso). If you really need to test that your users choose a keyboard in the installer (which is not very different from kickstart) this would obviously not work for you..