I am using valgrind --track-fds=yes to look for leaks. With my recent move to libuv, I am now getting lots of leaked file descriptors, especially with respect to the worker threads that are created. It appears that none of these are actual leaks as I am properly going through a shutdown of the loop using uv_loop_close() as well as uv_walk() in order to track down any stragglers. I'm pretty certain that all of these pertain to the libuv shared state. Is there any way to force the libuv shared state to clear itself out?
Related
Following is terrible for pthread users include me:
"Why threads can't fork | Hacker News" https://news.ycombinator.com/item?id=8449164
However Glib documents say that it can fork/exec under enabling pthread:
"Threads: GLib Reference Manual" https://developer.gnome.org/glib/stable/glib-Threads.html
"Spawning Processes: GLib Reference Manual" https://developer.gnome.org/glib/stable/glib-Spawning-Processes.html
Why/how can Glib do that?
You can fork() from a pthread, but it may be tricky to do the details right. For example, the forked process memory will contain copies of all the condition variables and mutexes in the original process. Here is a related question, and some guidelines on how to e.g. cleanly release all mutexes held by various threads.
fork in a multithreaded process with lots of state held in threads
Note that the general idea is to fork(), do some cleanup, and then exec().
For a list of async-signal-safe functions that can be called between fork and exec, see this manpage (signal-safety(7))
But my general advice is to make your threaded application simpler - forking with a lot of state to handle in varioius threads is asking for trouble. For example, exec() by design preserves open file descriptors, so if any thread had an open file descriptor, the exec():ed process may unnecessarily carry around that descriptor. Resource leaks are a code smell and generally a sign of sloppy design. Use RAII if you can.
I am evaluating different multiprocessing libraries for a fault tolerant application. I basically need any process to be allowed to crash without stopping the whole application.
I can do it using the fork() system call. The limit here is that the process can be created on the same machine, only.
Can I do the same with MPI? If a process created with MPI crashes, can the parent process keep running and eventually create a new process?
Is there any alternative (possibly multiplatform and open source) library to get the same result?
As reported here, MPI 4.0 will have support for fault tolerance.
If you want collectives, you're going to have to wait for MPI-3.something (as High Performance Mark and Hristo Illev suggest)
If you can live with point-to-point, and you are a patient person willing to raise a bunch of bug reports against your MPI implementation, you can try the following:
disable the default MPI error handler
carefully check every single return code from your MPI programs
keep track in your application which ranks are up and which are down. Oh, and when they go down they can never get back. but you're unable to use collectives anyway (see my opening statement), so that's not a huge deal, right?
Here's an old paper (back when Bill still worked at Argonne. I think it's from 2003):
http://www.mcs.anl.gov/~lusk/papers/fault-tolerance.pdf . It lays out the kinds of fault tolerant things one can do in MPI. Perhaps such a "constrained MPI" might still work for your needs.
If you're willing to go for something research quality, there's two implementations of a potential fault tolerance chapter for a future version of MPI (MPI-4?). The proposal is called User Level Failure Mitigation. There's an experimental version in MPICH 3.2a2 and a branch of Open MPI that also provides the interfaces. Both are far from production quality, but you're welcome to try them out. Just know that since this isn't in the MPI Standard, the function prefixes are not MPI_*. For MPICH, they're MPIX_*, for the Open MPI branch, they're OMPI_* (though I believe they'll be changing theirs to be MPIX_* soon as well.
As Rob Latham mentioned, there will be lots of work you'll need to do within your app to handle failures, though you don't necessarily have to check all of your return codes. You can/should use MPI error handlers as a callback function to simplify things. There's information/examples in the spec available along with the Open MPI branch.
I have VB.NET application in which one of the form has IE control in it, the application starts initially with memory size consumed around 9 MBs, but when IE form is launched, the memory consumed rises to 27 MB, and when that form is closed, the memory reduces merely by 3-4 MBs, so why memory allocated to IEFrame is not de-allocated automatically? is there any work around to solve this issue? if possible, launching the form as a separate process would be helpful.
If you make sure to dispose the form properly, the garbage collector should free up that memory eventually. Running the IE control in a separate process should not be necessary.
However, if you are using IE 7, you might want to read this question about memory leaks.
Why not just put that form in a separate application if this is an issue? There are plenty of ways you can pass whatever data between the two apps.
The still allocated memory might not be an issue at all. If you have sufficient available memory in the computer the .NET Garbage Collector will not run to clean up. Only when you need the memory the GC will kick in.
If you want to make sure it is a leak you could do the following:
Make sure you have no references to the form in any way.
Call GC.Collect()
See if the memory is still claimed
Do not put the GC.Collect() in the final build; it's just to make sure you are not hunting ghosts.
I'd like to have a small (not doing too damn much) daemon running on a little server, watching a directory for new files being added to it (and any directories in the main one), and calling another Clojure program to deal with that new file.
Ideally, each file would be added to a queue (a list represented by a ref in Clojure?) and the main process would take care of those files in the queue on a FIFO basis.
My question is: is having a JVM up running this little program all the time too much a resource hog? And do you have any suggestions as to how go about doing this?
Thank you very much!
EDIT: Another question I should ask: should I run this as its own instance (using less memory) and have it launch a new JVM when a file is seen, or have it on the same JVM the Clojure code that will process the file?
As long as it is running fine now and it has no memory leaks it should be fine.
From the daemon terminology I gather it is on a unix clone, and in this case best is to start it from an init script, or from the rc.local script. Unfortunately details differ from OS to OS to be more specific.
Limit the memry using -Xmx=64m or something to make sure it fails before taking down the rest of the services. Play a bit with the number to find the lowest reliable size.
Also, since clojures claim to fame is its ability to deal with concurrency it make a lot of sense to only run one JVM with all functionality running on it in multiple threads. The overhead of spawning new processes is already very big and if it is a JVM which needs to JIT and warm up its memory management, doubly so. On a resource constrained machine could pose a problem. and on a resource rich machine this is a waste.
I always found that the JVM is not made to quickly run something script like and exit again. It is really not made for that use case in my opinion
.
Whether this is possible I don't know, but it would mighty useful!
I have a process that fails periodically (running in Windows 2000). I then have just one chance to react to it before having to restart it and painfully wait for it to fail again. I didn't write the process so don't have the source to debug. The failure is seemingly random.
With a snapshot of the process I could repeatedly and quickly test reactions to the failure.
I had thought of running inside a VM but this isn't possible in this instance.
EDIT:
#Jon Cage asked:
When you say a snapshot, you mean capturing a process when it's about to fail (including memory, program state etc. etc.) ...and then replaying it's final few seconds repeatedly to see what effect it has on some other component?
This is exactly what I mean!
I think minidump is what you are looking for.
You can also used Userdump:
The User Mode Process Dumper
(userdump) dumps any running Win32
processes memory image (including
system processes such as csrss.exe,
winlogon.exe, services.exe, etc) on
the fly, without attaching a debugger,
or terminating target processes.
Generated dump file can be analyzed or
debugged by using the standard
debugging tools.
This article shows you how to use it.
My best bet is to start the process in a debugger (OllyDbg being my preferred tool).
The process will pause on an exception, and you can try to figure out what happened shortly before that.
This needs some understanding of assembler and does not allow to create a snapshot of the process for later analysis. You would need to write your own debugger for that - it should be theoretically possible.