Preface
I'm writing an web-server that gives to users an access to some program written on C (I'm using an Python wrapper over this C-program, it is PyCLIPS). To serve a lot of users, the web-server has to start a lot of copies of this C-program, because one copy can serve very few users in the same time, about 1-3 users. In addition, each user should work only with his own copy, therefore should be a lot of copies of the C-program.
This C-program is a CLIPS engine if it'll help to understand.
So, to solve this design problem, I want write a Twisted TCP server that will be like a pool of long-running-processes. Each of long-running-processes is a small Twisted TCP server that gives an access to one copy of the C-program.
In example, a user ask the pool-server to reserve a long-running-process for him, then the pool-server create and run a long-running-process that starts listening on some port, then the pool-server return the host and port of this long-running-process to user. Now, user can communicate with this long-running-process directly.
Questions
How start these long-running-process from the pool-server? The pool-server and each of long-running-processes are should be separate Twisted servers.
Is Twisted a good choice for these aims?
Maybe there are another ways how solve this design problem?
Thanks a lot.
Using Twisted for this sounds like it makes sense. The low-level API that supports running processes in Twisted is reactor.spawnProcess. Some examples of its usage are given in the process howto document.
This API is well suited for dealing with many long running processes, and as with all I/O APIs in Twisted, it works well when combined with another event source, such as a web server users can use to request new processes be started up.
reactor.spawnProcess is more like the standard library subprocess module than like the multiprocessing package. It gives you a way to launch a child process running a particular executable, with particular arguments, etc. It does not provide a high-level API for running a particular Python function in another process. However, it's not too hard to build such a thing (at least for a particular case). Consider this approach:
from sys import executable
from os import environ
from twisted.internet import reactor
implementation = """\
from yourapp import somefunction
somefunction()
"""
reactor.spawnProcess(executable, [executable, "-c", implementation], env=environ)
reactor.run()
This just launches a new Python interpreter (whichever one you happen to be running) and uses the -c option to specify a program on the command line for it to run.
Related
Background
A while ago I wrote a Raku module to test out some ideas I had for making external process pipelines (e.g., grep raku | wc -l). I followed the traditional way of creating Pipes and setting up I/O redirections for the external processes.
Since there's no fork available in Raku, nor is it recommended due to threads in MoarVM, I resorted to using Proc::Async to start a wrapper process for each external command in a pipeline, and the wrapper would then sets up I/O redirections to connect the command processes and the pipes, and then close other FDs before exec-ing the real command.
Problem
This all seems to work well except there's still the possibility that other parts of the program using the module might start a sub process (e.g., via run, shell, Proc, or Proc::Async) in another thread after the pipe FDs are created but before they are closed, thus leaking them and causing the command processes in the pipeline to block on read / write.
Question
How would you prevent FDs from leaking to any child processes started in another thread in Raku?
Thanks
I have a simple twisted application which I run using a systemd service, executing a script, which subsequently executes a .tac file.
The application is structured as a JSON RPC endpoint (fastjsonrpc), built into a t.w.r.Resource, which is in a t.w.s.Site, and served t.a.i.TCPServer, and the whole thing packed into a t.a.Application. This works fine.
Where I do run into trouble is when I try to warm up caches at startup. This warm-up process is pretty slow (~300 seconds), and makes systemd timeout and kill the process. Increasing the timeout is not really a viable option, since I wouldn't want this to block system boot.
Analogous code is used in a separate stack running on Flask from within Apache and wsgi. That server starts itself off and lets systemd go on while it takes its time building the caches. This behaviour is fine for me.
I've tried calling the warmup function using the following within the setup function of the t.w.r.Resource:
reactor.callLater(1, ep.warmup, None)
I've not yet tried using this from within systemd, and have been testing it from twistd directly on the command line. The server does work as expected, however it no longer responds to SIGINT (^C). Removing the callLater is all that's needed to let the server respond to SIGINT.
If the warmup function is called directly (not by callLater, i.e., the arrangement which makes systemd give up while waiting for warm up to complete), the resulting server also continues to respond to SIGINT.
Is there a better / good way to handle this sort of long-running warmup code?
Why would twistd / the reactor not respond to SIGINT? Am I missing something here?
Twisted is a single-threaded thing. It sounds like your "cache warmup" code is blocking the reactor for those 300 seconds. One easy way to fix this would be using deferToThread to let it run without blocking the reactor.
I'm experimenting with the Distributed Shell example in YARN 2.2 and am hoping that someone can clarify what the difference between a managed and and an un-managed application manager is?
For example the following lines appear in the client code
// unmanaged AM
appContext.setUnmanagedAM(true);
but I am unable to find documentation explaining the difference this line makes to the execution behaviour.
Many thanks.
The setUnmanagedAM(true) is used for debugging purposes i.e. it runs an application manager in local mode and does not submit it to a cluster so it is easier to step into code and debug.
You can see it in use in the hadoop-yarn-applications-unmanaged-am-launcher.jar that ships with yarn
Check the respective JIRA tickets: JIRA-420 and JIRA-419 (client side)
Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
Blog post with more implementation details on unmanaged AM: click-me
Example of how Impala manages its resources with the help of unmanaged applications: Llama
I am completely new to the DDS world. I understand basic concepts like publish and subscribe, and the stuff that can be gained from the documentation. I am attempting to use OpenSlice DDS, and am able to get through the tutorial without much difficulty. However, I want to get two different computers on the same network to talk to each other, which seems like a relatively simple task, but i can find no documentation on it.
For example, the message chat room tutorial... how would i get the message board running on one machine, and the chatter on another machine?
Thanks!
Found it! http://opensplice.org/pipermail/developer/2009-July/000094.html.
To summarize from the link:
Setup your environment on node 1 by running the release file in the OSPL_HOME directory (release.bat)
start the opensplice daemon on node 1 (ospl start)
run the messageboard application on node 1
Setup your environment on node 1 by running the release file in the OSPL_HOME directory (release.bat)
start the opensplice daemon on node 2 (ospl start)
run the chatter application on node 2
Is there a way to know the return code or process ID of the process which gets executed when the privileged helper tool is installed as a launchdaemon and launched via SMJobSubmit().
I have an application which to execute some tasks in privileged manner uses the SMJobSubmit API as mentioned here.
Now in order to know whether the tasks succeeded or not, I will have to do one of the following.
The best option is to get the return code of the executable that ran.
Another option would be if I could create a pipe between my application and the launchd.
If the above two are not possible, I will have to resort to some hack like writing a file in /tmp location and reading it from my app.
I guess SMJobSubmit internally submits the executable with a launchdaemon dictionary to the launchd which is then responsible for its execution. So is there a way I could query launchd to find out the return code for the executable run with the label "mylabel".
There is no way to do this directly.
SMJobSubmit is a simple wrapper around a complicated task. It also returns synchronously despite launching a task asynchronously. So, while it can give you an error if it fails to submit the job, if it successfully submits a job that fails to run, there is no way to find that out.
So, you will have to explicitly write some code to communicate from your helper to your app, to report that it's up and running.
If you've already built some communication mechanism (signals, files, Unix or TCP sockets, JSON-RPC over HTTP, whatever), just use that.
If you're designing something from scratch, XPC may be the best answer. You can't use XPC to launch your helper (since it's privileged), but you can manually create a connection by registering a Mach service and calling xpc_connection_create_mach_service.