I was wondering if it is possible to generate a "core" file, copy if to another machine and then continue execution of the a core file on that machine?
I have seen the gcore utility that will make a core file from a running process. But I do not think gdb can continue execution based on a core file.
Is there any way to just dump the heap/stack and and restore those at a later point?
it's called process migration.
mosix and OpenMosix used to be able to do that. nowadays it's easiest to migrate a whole VM.
On modern systems, not from a core file, no you can't. For freezing and restoring an individual process on Linux, CryoPID and the new Kernel-based checkpoint and restart are in the works, but their abilities are currently quite limited. OpenVZ and other virtualization-like softwares can freeze and restore an entire system.
Also checkout out the Condor project. Condor can do that with parallel jobs as well. Condor also include monitors that can automatically migrate your process when some, for example, starts using their workstation again. It's really designed for utilizing spare cycles in networked environments.
This won't, in general, be sufficient to let an arbitrary process continue on another machine. In addition to the heap and stack state, there may also also open I/O handles, allocated hardware resources, etc. etc.
Your options are either to explicitly write your software in a way that lets it dump state on a signal and later resume from the dumped state, or to run your software in a virtual machine and migrate that to the alternate host - Xen and Vmware both support freeze/restore as well as live migration.
That said, CryoPID attempts to do precisely this and occasionally succeeds.
As of Feb. 2017, there's a fairly stable and mature tool, called CRIU that depends on updates to the Linux Kernel made in version 3.11 (as this was done in Sep. 2013, most modern distros should have those incorporated into their kernel versions).
It can be installed via aptitude by simply calling sudo apt-get install criu.
Instructions on how to use it.
In some cases, this can be done. For example, part of the Emacs build process is to load up all the Lisp libraries and then dump the memory image on disk for quick loading. Some other language interpreters do that too (I'm thinking of Lisp and Scheme implementations, mostly). However, they're specially designed for that kind of use, so I don't know what special things they have to do to allow that to work.
I think this would be very hard to do for a random program, but if you wrote a framework where all objects supported serialisation/deserialisation, you can then serialise all objects used by your program, and then ship that elsewhere, and deserialise them at the other end.
The other people's answers about virtualisation are on the spot, too.
Depends on the machine. It's very doable in a very small embedded system, for instance. I think it's also implemented somewhat in Beowulf clusters and other supercomputeresque apps.
There are lots of reasons you can't do what you want very easily. For example, when you restore the core file on the other machine how do you resolve file descriptors that you process had open? What about sockets, named pipes, semaphores, or any other OS-level resource? Basically unless your system is specifically designed to handle such an operation you can't naively dump a core file and move it to another machine.
I don't believe this is possible. However, you might want to look into virtualization software - e.g. Xen - which make it possible to freeze and move entire system images fromone machine to another.
Related
I want to play around and check out Apache's couchdb as a possible back-end for a web-app that I am designing. Therefore I want to have an instance of couchdb, but also to be able to throw it away when the testing is done. The development computer is an Ubuntu laptop (not server). The problems are:
The Ubuntu repository has couchdb 1.0, but the couchdb website strongly recommends to install 1.1, built from source.
I have Erlang built and installed from source, because the Erlang distro from the repository is defective. I don't see the point in installing another Erlang aside it.
couchdb has a lot of dependencies, including a whole bunch of perl libs, that I really don't need, and prefer to throw away when I'm done.
So I am looking for a way to either:
Install couchdb 1.1 as a package that can be easily uninstalled, or
Build couchdb from source, with as few as possible installed dependencies, so when I'm done I can just delete it. Preferably, do this without building another Erlang distro, but configuring it to use the existing one.
Is any of these possible, and how? Thanks in advance.
Btw, I am aware of the build-couchdb project, but from what I read, it requires installing all the build dependencies in advance, which is undesirable, because it will leave a whole bunch of dangling packages in my system, without being a dependency of a couchdb package. It also fetches a copy of Erlang, which is redundant for me.
(Dear moderators: This questions combines issues that relate not only to programming, but also to server administration, Unix software, and, particularly, Ubuntu Linux issues. Therefore, it might be suitable for a few other stack exchange sites. I recon it is most likely to be answered here, since this kind of hackery is often done by programmers. However, if I am wrong, feel free to migrate it, and I apologies in advance for your troubles.)
You could install CouchDB into a chroot jail
A chroot is a way of isolating applications from the rest of your computer, by putting them in a jail. This is particularly useful if you are testing an application which could potentially alter important system files
From the Ubuntu instructions on creating a chroot jail
Another option, assuming your laptop has the appropriate hardware virtualization support, is to use KVM.
The KVM option might be more helpful in the long run as you could move the VM's disk image onto a server.
I am quite new to the embedded linux programming and did not really understand this concept very well.
Can anyone explain the essence of the "host-target" relation? Is this model only specific to the "cross-compilation"? Is it used just because "executable code will be run on another enviroment"? and what matters with the linux kernel on the target? E.g., the "building the embedded linux system" book mentioned this, but did not explain its motivation or goal of this type of development.
Thanks a lot.
The 'motivation' for this model is that seldom is an embedded target a suitable platform for development. It may be resource constrained, have no operating system, have no compiler that will run on the target, have no filesystem for source files, have no keyboard or display, no networking, and may be relatively slow or anything else you might need to develop effectively.
If your embedded system is suited to running Linux, it is possible that not all of the above limitations apply, but almost certainly enough of them to make you want to avoid developng directly on the target. If this were not so, they it hardly qualifies as an embedded system perhaps.
http://www.landley.net/writing/docs/cross-compiling.html
Seems pretty clear. What specific questions do you have?
Linux since its very origin was written in very portable way. It runs on a whole range of machines with very different CPUs, and it is considered the Good Thing to write in a portable way, so that, for example, package maintainer can easily port your program to some embedded ARM or Cygwin, or Amiga, or...
So, yes, the model is "only" specific to cross-compilation, but actually about every compilation on Linux is a (variant of) cross-compilation, just that by default build, host and target are automatically set to the same value, the same as the machine you run on.
Still, even then, you can take a Linux-i386 compiled compiler, sources for it, and "cross-compile" it for Linux-amd64. And the resulting binary will run much faster on a 64bit CPU.
It IS quite essential in embedded programming though. Mostly because you write programs for weak CPUs that are not capable of running a compiler or would run it at a snail pace. So you take a cross-compiler on a fast CPU (say, some multi-core Intel) and cross-compile for the embedded CPU (say, some low-end ARM).
"In different environment" is putting things very mildly. What you're doing when cross-compiling for embedded is working with entirely different instruction set, different memory access modes, different resource access methods and so on and so on. A machine of entirely different construction than the build host. Your build host may be a Windows PC running Cygwin. Your target may be a chip inside a smartphone. The binary will look nothing like the Cygwin .exe files.
As a direct consequence, -everything- must be compiled for the target from scratch. The kernel, the system utilities, the system libraries, all the tools the target must be running. Thing is, if the target is a ticket selling booth, there is really no sense cross-compiling Eclipse, GCC and Gnome for it, then developing in "local" environment, typing your code on a ticket booth keyboard. Instead, you just cross-compile the essentials of the OS, and your specific applications. You keep the development environment on the build machine, and cross-compile everything you need on the embedded device.
[in practice, you get a Linux distro for the target, and just compile whatever you need modified].
Suppose I have a server that runs on Linux on which I would like to develop software (mainly OCaml, C/C++ and Java).
Is there a way to "remote develop" these things? I mean an IDE that allows me to modify files remotely (they are then uploaded when modified and saved) and to compile through SSH (basically invoking make or omake).
I was looking for something that makes this process transparent to the developer, without caring of doing things by hand. I'm used to use Eclipse so I wonder if a plugin to achieve this exists or if are there other choices?
Mind that it may happen that the local machine it not able to build software I intend to (for example for OCaml) so it should rely just on remote connection.
Thanks in advance
You can use X11 forwarding. Even if you are connecting from a Windows machine.
If you are on Linux, connecting with ssh -Y might work right out of the box for you:
ssh -Y user#your_server
eclipse &
Well the simplest idea I can think of, though it is rather brute force would be to just open up a file share to the server and then edit the file directly through Eclipse.
If that doesn't work for Java at least you could make use of Maven to do some of those tasks. I am less certain about invoking Make though.
I think your answer is IDE-centric.
KDE's ioslaves support access over both SFTP and SSH (using fish, which uses a Perl script uploaded to the remote machine). I believe Gnome also has a virtual file system (gvfs) which supports remote filesystem access.
My recommendation, therefore, is to choose an IDE which supports a virtual filesystem that can operate over SSH/SFTP and allows you to specify the build command. You would then only need to specify the build command which would get its output from the remote make command (for example, vim has a makeprg option which can be set to any arbitrary command).
Depending on how 'remote' this is; why not ssh in and run the IDE remotely over X?
Using a build tool (Hudson for example) you could put a build agent on your remote server, check your changes into your repository as normal, and have it do a build when you check in changes (it will either do a repository hook or poll for changes, probably). Your build process will be the same, it will simply be automated. :-)
emacs has tramp, which lets you both open and save remote files, and open a shell on a remote system. Working with tramp is almost exactly like working with local files, except for the filename. To open 'foo.c' on the machine 'bork' as user 'joe' I open it with the standard emacs commands, giving it the pathname /joe#bork:foo.c
I use vim for remote development. (Well, I use vim also non-remote.)
If building is the problem, have you thought about simply using an automated build system where you commit to svn and the system then automatically builds the software? I've heard many good things about these sorts of systems, although I haven't quite tried any out myself.
As for remote development, a SVnDAV solution might be reasonable. It basically commits your every save and is completely transparent to the text editor you're using. However a probably much nicer solution would simply be to use a networked drive/directory and edit files remotely. On all unix-based systems this should work completely transparently to both the developer and the text editor.
Your choice of IDE will have the most impact on the answer to "can I?". If your IDE of choice is CLI based than you can always just SSH in, fire up screen (so that your CLI session is persistent across SSH sessions), and have at it!
Use vim or emacs since they will offer you speed. I know there is a learning curve associated with these editors; but once you get comfortable in any of them; you will be able to work on them as good as with Eclipse or any other IDE.
If you already have a linux server then I would suggest setting up a simple VPN server. I have done this in the past and it works pretty well. This way you can connect and modify/build your files with any "local" OS. I did this cause I use mac, pc and linux through various parts of the day and in multiple locations, so the VPN allowed me to edit files remotely w/out having to allow file sharing over the internet.
There are plenty of tutorials about how to achieve this even if you are newer to linux. I use ubuntu server on my linux box and here are the tutorial I have used.
http://www.ubuntugeek.com/howto-pptp-vpn-server-with-ubuntu-10-04-lucid-lynx.html
Netbeans 7.3 has a new feature which addresses your problem (and mine). Here's the tutorial.
https://netbeans.org/kb/docs/cnd/remotedev-tutorial.html
note: I realize it has been 3 years since this question was asked so the answer may be irrelevant to #Jack now.
One IDE that supports exactly your language set is Nuclide. It adds some packages to Atom and is used internally in Facebook exactly as you have described - full-fledged remote development in C++, Java, and Ocaml.
If a friendly file editor is enough for you then I'd recommend to use Jupyter.
Super fast installation
Built in server/file editor that starts with one command
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Every time a new developer joins the team or the computer a developer is using changes, the developer needs to do lots of work to setup the local development environment to make the current project work. As a SCRUM team we are trying to automate everything including deployment and tests so what I am asking is: is there a tool or a practice to make local development environment setup automated?
For example to setup my environment, first I had to install eclipse, then SVN, Apache, Tomcat, MySQL, PHP. After that I populated the DB and I had to do minor changes in the various configuration files etc... Is there a way to reduce this labor to one-click?
There are several options, and sometimes a combination of these is useful:
automated installation
disk imaging
virtualization
source code control
Details on the various options:
Automated Installation Tools for automating installation and configuration of a workstation's various services, tools and config files:
Puppet has a learning curve but is powerful. You define classes of machines (development box, web server, etc.) and it then does what is necessary to install, configure, and keep the box in the proper state. You asked for one-click, but Puppet by default is zero-click, as it checks your machine periodically to make sure it is still configured as desired. It will detect when a file or mode has been changed, and fix the problem. I currently use this to maintain a handful of RedHat Linux boxes, though it's capable of handling thousands. (Does not support Windows as of 2009-05-08).
Cfengine is another one. I've seen this used successfully at a shop with 70 engineers using RedHat Linux. Its limitations were part of the reason for Puppet.
SmartFrog is another tool for configuring hosts. It does support Windows.
Shell scripts. RightScale has examples of how to configure an Amazon EC2 image using shell scripts.
Install packages. On a Unix box it's possible to do this entirely with packages, and on Windows msi may be an option. For example, RubyWorks provides you with a full Ruby on Rails stack, all by installing one package that in turn installs other packages via dependencies.
Disk Images Then of course there are also disk imaging tools for storing an image of a configured host such that it can be restored to another host. As with virtualization, this is especially nice for test boxes, since it's easy to restore things to a clean slate. Keeping things continuously up-to-date is still an issue--is it worth making new images just to propagate a configuration file change?
Virtualization is another option, for example making copies of a Xen, VirtualPC, or VMWare image to create new hosts. This is especially useful with test boxes, as no matter what mess a test creates, you can easily restore to a clean, known state. As with disk imaging tools, keeping hosts up-to-date requires more manual steps and vigilance than if an automated install/config tool is used.
Source Code Control Once you've got the necessary tools installed/configured, then doing builds should be a matter of checking out what's needed from a source code repository and building it.
Currently I use a combination of the above to automate the process as follows:
Start with a barebones OS install on a VMWare guest
Run a shell script to install Puppet and retrieve its configs from source code control
Puppet to install tools/components/configs
Check out files from source code control to build and deploy our web application
I stumbled across this question and was very suprised that no one has mentioned Vagrant yet.
As Pete TerMaat and others have mentioned, virtualization is a great way to manage and automate development environments. Vagrant basically takes the pain away from setting up these virtual boxes.
Within minutes you can have a completely fresh copy of your favourite Linux distro up and running, and provisioned exactly the same way your production server is.
No more fighting with OSX or Windows to get PHP, MySQL, etc. installed. All software lives and runs inside the virtual machine. You can even SSH in with vagrant ssh. If you make a mistake or break something, just vagrant destroy it, and vagrant up to start over fresh.
Vagrant automatically creates a synced folder to your local file system, meaning you don't need to develop within the virtual machine (ie. using Vim). Use whatever your editor of choice is.
I now create a new "Vagrant box" for almost every project I do. All my settings are saved into the project repository, so it's easy to bring on another team member. They simply have to pull the repo, and run vagrant up, and they are literally ready to go.
This also makes it much easier to handle projects that have different software requirements. Maybe you have some projects that rely on PHP 5.3, but some newer ones that run PHP 5.4. Just install the version you want for that project.
Check it out!
One important point is to set up your projects in source control such that you can immediately build, deploy and run after checkout.
That means you should also checkin helper infrastructure, such as Makefiles, ant buildfiles etc., and settings for the tools, such as IDE project files.
That should take care of the setup hassle for individual projects.
For the basic machine setup, you could use a standard image. Another option is to use your platform's tools to automate installation. Under Linux, you could create a meta-package that depends on all the packages you need. Under Windows, a similar thing should be possible using MSI or the like.
Edit:
Ideally, instead of checking in helper infrastructure, you check in the information that allows the build to generate the helper infrastructure. This is the approach taken by e.g. the GNU build system (autotools etc.), or by Maven. This is even more elegant, because you can (theoretically) generate infrastructure for any (supported) build environment, thus you are not bound to e.g. one specific IDE, and settings in the helper infrastructure (paths etc.) don't need to duplicate the main project settings.
However, this also a more complex approach, so if you can't get it to work, I believe checking in stuff like IDE files directly is acceptable.
I like to use Virtual PC or VMware to virtualize the development environment. This provides a standard "dev environment" that could be shared among developers. You don't have to worry about software that the user could add to their system that may conflict with your development environment. It also provides me a way to work to two projects where the development environments can't both be on one system (using two different versions of a core technology).
Use puppet to configure both your development and production environment. Using a top-notch automation system is the only way to scale your ops.
There's always the option of using virtual machines (see e.g. VMWare Player). Create one environment and copy it over for each new employee with minimal configuration needed.
At a prior place we had everything (and I mean EVERYTHING) in SCM (clearcase then SVN). When a new developer can in they installed ClearCase|SVN and sucked down the repository. This also handles the case when you need to update a particular lib/tool as you can just have the dev teams update their environment.
We used two repo's for this so code and tools/config lived in separate places.
I highly recommend Blueprint from DevStructure. It's open-source and your use case is actually the exact reason we originally wrote the software. Our goals have somewhat changed, but it still is the perfect tool for what you are describing. In short, you can create reusable server configs - dead simple configuration management. I hope this helps!
https://github.com/devstructure/blueprint (Blueprint # Github)
I've been thinking about this myself. There are some other technologies that you could throw into the mix. Here's what I'm currently setting up:
PXE based pre-seeded installation images (Debian Squeeze). You can start up a bare-metal machine (or new virtual appliance) and select the image from the PXE boot menu. This has the major advantage of being able to install your environment on physical machines (in addition to virtual appliances).
Someone already mentioned Puppet. I use CFEngine but it's a similar deal. Essentially your configuration is documented and centralized in policy files which are continually enforced by an agent on the client.
if you don't want a rigid environment (i.e. developers may choose a combination of tool-sets) you can roll your own deb packages so new devs can type sudo apt-get install acmecorp-eclipse-env or sudo apt-get install acmecorp-intellij-env, for example.
Slightly off-topic, but if you run a Debian based environment (i.e. Ubuntu), consider installing apt-cacher (package proxy). In addition to saving bandwidth, it will make your installations much faster (since packages are cached on your local network).
If you're using OSX and working with Rails. I'd suggest either:
https://github.com/platform45/let-there-be-light
https://github.com/thoughtbot/laptop
If you use machines in a standard configuration, you can image the disk with a fresh perfectly configured install -- that's a very popular approach in many corporations (and not just for developers, either). If you need separately configured OS's, you can tar-bz2 all the added and changed files once a configured OS is turned into your desired setup, and just untar it as root to make your desired environment from scratch.
if you're using a linux flavor, you've probably got a package management system: thinks .rpm for fedora/redhat, or .deb for ubuntu/debian. many of the things you describe already have packages available: svn, eclipse, etc. you could roll your own packages for company specific software, create a repository (perhaps only available on the local network) and then your setup could be reduced to a single bash script which would add the company repo to /etc/apt/sources.list (debian/ubuntu) and then call a command like,
/home/newhire$ apt-get update && apt-get install some complete package list
you could use buildbot to then automate regular builds for company packages that change often.
Try out DevScript at http://nsnihalsahu.github.io/devscript .
Its one command like ,
devscript lamp or devscript laravel or devscript django . In around a few minutes ,depending on the speed of your internet co
We need to interface to 3rd party app, but company behind the app doesn't disclose message protocol and provides only Windows DLL to interface to.
Our application is Linux-based so I cannot directly communicate with DLL. I couldn't find any existing solution so I'm considering writing socket-based bridge between Linux and Windows, however I'm sure it is not such a unique problem and somebody should have done it before.
Are you aware of any solution that allows to call Windows DDL functions from C app on Linux? It can use Wine or separate Windows PC - doesn't matter.
Many thanks in advance.
I wrote a small Python module for calling into Windows DLLs from Python on Linux. It is based on IPC between a regular Linux/Unix Python process and a Wine-based Python process. Because I have needed it in too many different use-cases / scenarios myself, I designed it as a "generic" ctypes module drop-in replacement, which does most of the required plumbing automatically in the background.
Example: Assume you're in Python on Linux, you have Wine installed, and you want to call into msvcrt.dll (the Microsoft C runtime library). You can do the following:
from zugbruecke import ctypes
dll_pow = ctypes.cdll.msvcrt.pow
dll_pow.argtypes = (ctypes.c_double, ctypes.c_double)
dll_pow.restype = ctypes.c_double
print('You should expect "1024.0" to show up here: "%.1f".' % dll_pow(2.0, 10.0))
Source code (LGPL), PyPI package & documentation.
It's still a bit rough around the edges (i.e. alpha and insecure), but it does handle most types of parameters (including pointers).
Any solution is going to need a TCP/IP-based "remoting" layer between the DLL which is running in a "windows-like" environment, and your linux app.
You'll need to write a simple PC app to expose the DLL functions, either using a homebrew protocol, or maybe XML-RPC, SOAP or JSON protocols. The RemObjects SDK might help you - but could be overkill.
I'd stick with a 'real' or virtualized PC. If you use Wine, the DLL developers are unlikely to offer any support.
MONO is also unlikely to be any help, because your DLL is probably NOT a .NET assembly.
This is a common problem. Fortunately, it now has a solution. Meet LoadLibrary, developed by Tavis Ormandy:
https://github.com/taviso/loadlibrary
I first stumbled across LoadLibrary in an article on Phoronix by Michael Larabel:
A Google researcher has been developing "LoadLibrary" as a means of
being able to load Windows Dynamic Link Libraries (DLLs) that in turn
can be used by native Linux code.
LoadLibrary isn't a replacement for Wine or the like but is intended
to allow Windows DLL libraries to be loaded that can then be accessed
by native Linux code, not trying to run Windows programs and the like
on Linux but simply loading the libraries.
This project is being developed by Tavis Ormandy, a well known Google
employee focused on vulnerability research. He worked on a custom
PE/COFF loader based on old ndiswrapper code, the project that was
about allowing Windows networking drivers to function on Linux.
LoadLibrary will handle relocations and imports and offers an API
inspired by dlopen. LoadLibrary at this stage appears to be working
well with self-contained Windows libraries and Tavis is using the
project in part for fuzzing Windows libraries on Linux.
Tavis noted, "Distributed, scalable fuzzing on Windows can be
challenging and inefficient. This is especially true for endpoint
security products, which use complex interconnected components that
span across kernel and user space. This often requires spinning up an
entire virtualized Windows environment to fuzz them or collect
coverage data. This is less of a problem on Linux, and I've found that
porting components of Windows Antivirus products to Linux is often
possible. This allows me to run the code I’m testing in minimal
containers with very little overhead, and easily scale up testing."
More details on LoadLibrary for loading Windows DLLs on Linux via
GitHub where he also demonstrated porting Windows Defender libraries
to Linux.
Sometimes it is better to pick a small vendor over a large vendor because the size of your business will give you more weight for them. We have certainly found this with AV engine vendors.
If you are sufficiently important to them, they should provide either a documented, supported protocol, a Linux build of the library, or the source code to the library.
Otherwise you'll have to run a Windows box in the loop using RPC as others have noted, which is likely to be very inconvenient, especially if the whole of the rest of your infrastructure runs Linux.
Will the vendor support the use of their library within a Windows VM? If performance is not critical, you might be able to do that.
Calling the DLL's functions themselves is of course only the tip of the iceberg. What if the DLL calls Win32, then you'd have a rather massive linking problem. I guess Wine could help you out there, not sure if they provide a solution.
IMO, the best bet is to use Sockets. I have done this previously and it works like a charm.
An alternate approach is to use objdump -d to disassemble the DLL, and then recompile/reassemble it. Don't expect to be able to recompile the code unedited. You might get pure, unadulterated rubbish, or code full of Windows calls, or both. Look for individual functions. Functions are often delimited by a series of push instructions and end with a ret instruction.