I'm looking for a process pool management library, maybe with an interface similar to Python's concurrent.futures.
My goal is to open N processes to execute a task, and when one finishes, create a new one in its stead. So at any single point in time, there are N running processes.
Is there something in existence?
Ended up writing my own class.
I hope it can be of use to someone: https://bitbucket.org/snippets/dorfire/8jdLn
Related
I have a server which has two JVM's and I have a class which creates dummy threads.This runs on lets say JVM A.How can I create these threads on JVM B programatically.
After some research I came across http://docs.oracle.com/javase/6/docs/jdk/api/attach/spec/index.html. I am not sure if this can suffice my requirement.
You need JVM A to contact JVM B to tell it to start some threads. You will also need to pass any data JVM B need to run and possibly pass back any results. A simple example is using RMI or RPC.
I have written a task script using vb.net that have thread used in the code, the problem is how i can know when will be finished all the threads so i can return the success result.
Thanks alot.
i think you need to use a waitHandle object and the waitAll method
more info here: http://msdn.microsoft.com/en-us/library/system.threading.waithandle.aspx
That being said, I suspect you can refactor the design of your package to let the script task handle the execution, and let SSIS handle the execution scheduling. this gives you the parallelism you want without any of the hassle of multi threaded programming in .net.
a simple setup would be n foreach loops (which execute in serial) each running a partitioned chunk of the work load.
Another simpler option is have the package driven by variables and spawn multiple executions of the package. This could occur across 1-N servers to scale out.
I'd like to have a small (not doing too damn much) daemon running on a little server, watching a directory for new files being added to it (and any directories in the main one), and calling another Clojure program to deal with that new file.
Ideally, each file would be added to a queue (a list represented by a ref in Clojure?) and the main process would take care of those files in the queue on a FIFO basis.
My question is: is having a JVM up running this little program all the time too much a resource hog? And do you have any suggestions as to how go about doing this?
Thank you very much!
EDIT: Another question I should ask: should I run this as its own instance (using less memory) and have it launch a new JVM when a file is seen, or have it on the same JVM the Clojure code that will process the file?
As long as it is running fine now and it has no memory leaks it should be fine.
From the daemon terminology I gather it is on a unix clone, and in this case best is to start it from an init script, or from the rc.local script. Unfortunately details differ from OS to OS to be more specific.
Limit the memry using -Xmx=64m or something to make sure it fails before taking down the rest of the services. Play a bit with the number to find the lowest reliable size.
Also, since clojures claim to fame is its ability to deal with concurrency it make a lot of sense to only run one JVM with all functionality running on it in multiple threads. The overhead of spawning new processes is already very big and if it is a JVM which needs to JIT and warm up its memory management, doubly so. On a resource constrained machine could pose a problem. and on a resource rich machine this is a waste.
I always found that the JVM is not made to quickly run something script like and exit again. It is really not made for that use case in my opinion
.
I want to copy a file to a FTP server using wxFTP, but I would like to do this without blocking the UI, and much better, while displaying a progress bar. Can I do this without extra thread?
I'm using wxLua, but I can adapt a solution written in any language as long as it uses a wxWidgets binding.
Try using wx.lib.delayedresult. It's available in wxPython, but maybe also it is in your wxWidgets library too. It creates separate worker thread and is called with a consumer function that is called once worker thread finishes his job. Quite useful thing.
See wxPython docs for details.
What's wrong with starting your own Thread for this?
You could check the streams canRead() method periodically (through a timer or in the event loop, maybe) and only read when it returns true, but it'll probably be a lot more complex than just starting a separate thread.
Whether this is possible I don't know, but it would mighty useful!
I have a process that fails periodically (running in Windows 2000). I then have just one chance to react to it before having to restart it and painfully wait for it to fail again. I didn't write the process so don't have the source to debug. The failure is seemingly random.
With a snapshot of the process I could repeatedly and quickly test reactions to the failure.
I had thought of running inside a VM but this isn't possible in this instance.
EDIT:
#Jon Cage asked:
When you say a snapshot, you mean capturing a process when it's about to fail (including memory, program state etc. etc.) ...and then replaying it's final few seconds repeatedly to see what effect it has on some other component?
This is exactly what I mean!
I think minidump is what you are looking for.
You can also used Userdump:
The User Mode Process Dumper
(userdump) dumps any running Win32
processes memory image (including
system processes such as csrss.exe,
winlogon.exe, services.exe, etc) on
the fly, without attaching a debugger,
or terminating target processes.
Generated dump file can be analyzed or
debugged by using the standard
debugging tools.
This article shows you how to use it.
My best bet is to start the process in a debugger (OllyDbg being my preferred tool).
The process will pause on an exception, and you can try to figure out what happened shortly before that.
This needs some understanding of assembler and does not allow to create a snapshot of the process for later analysis. You would need to write your own debugger for that - it should be theoretically possible.