Background
A while ago I wrote a Raku module to test out some ideas I had for making external process pipelines (e.g., grep raku | wc -l). I followed the traditional way of creating Pipes and setting up I/O redirections for the external processes.
Since there's no fork available in Raku, nor is it recommended due to threads in MoarVM, I resorted to using Proc::Async to start a wrapper process for each external command in a pipeline, and the wrapper would then sets up I/O redirections to connect the command processes and the pipes, and then close other FDs before exec-ing the real command.
Problem
This all seems to work well except there's still the possibility that other parts of the program using the module might start a sub process (e.g., via run, shell, Proc, or Proc::Async) in another thread after the pipe FDs are created but before they are closed, thus leaking them and causing the command processes in the pipeline to block on read / write.
Question
How would you prevent FDs from leaking to any child processes started in another thread in Raku?
Thanks
Related
I have a simple twisted application which I run using a systemd service, executing a script, which subsequently executes a .tac file.
The application is structured as a JSON RPC endpoint (fastjsonrpc), built into a t.w.r.Resource, which is in a t.w.s.Site, and served t.a.i.TCPServer, and the whole thing packed into a t.a.Application. This works fine.
Where I do run into trouble is when I try to warm up caches at startup. This warm-up process is pretty slow (~300 seconds), and makes systemd timeout and kill the process. Increasing the timeout is not really a viable option, since I wouldn't want this to block system boot.
Analogous code is used in a separate stack running on Flask from within Apache and wsgi. That server starts itself off and lets systemd go on while it takes its time building the caches. This behaviour is fine for me.
I've tried calling the warmup function using the following within the setup function of the t.w.r.Resource:
reactor.callLater(1, ep.warmup, None)
I've not yet tried using this from within systemd, and have been testing it from twistd directly on the command line. The server does work as expected, however it no longer responds to SIGINT (^C). Removing the callLater is all that's needed to let the server respond to SIGINT.
If the warmup function is called directly (not by callLater, i.e., the arrangement which makes systemd give up while waiting for warm up to complete), the resulting server also continues to respond to SIGINT.
Is there a better / good way to handle this sort of long-running warmup code?
Why would twistd / the reactor not respond to SIGINT? Am I missing something here?
Twisted is a single-threaded thing. It sounds like your "cache warmup" code is blocking the reactor for those 300 seconds. One easy way to fix this would be using deferToThread to let it run without blocking the reactor.
New to Elixir and OTP. Trying some hello-world examples with GenServer. I'm writing tests where each test spawns a GenServer process. They are not explicitly terminated. Will the process die after the test finishes?
Follow up: If it's not terminated, do they turn into "zombie" process? Also, is there a pattern for terminating them after the test completes?
No, processes are not garbage collected. Yes, they turn into "zombie" processes.
The pattern to prevent that is to link the processes. When one linked process terminates - all linked processes will terminate too. So if you start your server using GenServer.start_link from the test process - this will link the server process to the test process and when the test exists, the server will exit too.
I am running a rewrite map with an external rewrite program (prg) in apache2 that may produce an error and die.
When the rewrite map is not running any more the system obviously doesn't function properly.
So I wanted to start a simple wrapper shell script that itself executes map program (which is written in php) and restarts it if it dies:
#!/bin/bash
until /usr/bin/php /somepath/mymap.php; do
echo "map died but i will restart it right away!"
done
If I try that in the shell by hand, it works fine, however it does not work when started by the webserver.
...and then communicates with the rewriting engine via its stdin and
stdout file-handles. For each map-function lookup it will receive the
key to lookup as a newline-terminated string on stdin. It then has to
give back the looked-up value as a newline-terminated string on stdout
or the four-character string ``NULL'' if it fails...
The reason seems pretty clear to me. The first script takes stdin but doesn't redirect it to the sub script.
I guess I somehow need to define a descriptor using exec and redirect stdin/stdout of the scripts properly. But how do I do that?
It's a common problem that some script works executed "by hand" and do not work when executed indirectly (via cron or from apache).
Usually the root cause is one of:
your script needs some extra environment varaible (PATH, LD_LIBRARY_PATH, etc.)
your script requires terminal
First thing to do is to grab some debug info, so add to your script:
env > /tmp/$0.env # get environment
and get stderr:
... /usr/bin/php /somepath/mymap.php 2>/tmp/$0.stderr ...
This may lead you to the solution.
If your script requires terminal and you cannot fix it you can run your script via gnu screen.
Good luck.
Michał Šrajer has given one very common cause of trouble (environment). You should certainly be sure that the environment is sufficiently well set up, because Apache sets its own environment rigorously and does not pass on any inherited junk values; it only passes what it is configured to pass (SetEnv and PassEnv directives, IIRC).
Another problem is that you think your mapping process will fail; that's worrying. Further, it is symptomatic of yet another problem, which I think is the main one.
The first time the map process is run, it will read the request from the web server, but if the mapping fails, you rerun it - but the web server is still waiting for the output from the original request, and the map process is waiting for the input, so there's an impasse.
Note that a child process automatically inherits the standard input and standard output of its parent unless you do something to change it.
If you think things might fail, you'll need to capture the standard input so that when you rerun the program, you can resupply the input (though why it would work on the second time when it failed on the first is a separate mystery).
Maybe:
if tee /tmp/xx.$$ | /usr/bin/php /somepath/mymap.php
then : OK
else
until /usr/bin/php /somepath/mymap.php < /tmp/xx.$$
do echo "map died but I will restart it right away!"
done
fi
rm -f /tmp/xx.$$
Unresolved issues include:
You should add traps to ensure that the temporary file is removed;
You should probably not use /tmp as the directory;
The messages probably do not get sent to the web server and from thence to the client browser until the overall script terminates, so the echoed messages simply mess up the start of the response;
There is no limit on the number of failures;
There is no logging of the failures;
There is no attempt to fix the problem that caused the failure;
And there are probably others I've not thought of yet.
until is not a valid keyword, you probably have it aliased and aliases do not work on scripts. my mistake, it seems it is.
regardless, this is what you want to do:
while true; do
/usr/bin/php /somepath/mymap.php
done
if this also fails then yes, either your program expects a terminal or you have something missing in your env.
Preface
I'm writing an web-server that gives to users an access to some program written on C (I'm using an Python wrapper over this C-program, it is PyCLIPS). To serve a lot of users, the web-server has to start a lot of copies of this C-program, because one copy can serve very few users in the same time, about 1-3 users. In addition, each user should work only with his own copy, therefore should be a lot of copies of the C-program.
This C-program is a CLIPS engine if it'll help to understand.
So, to solve this design problem, I want write a Twisted TCP server that will be like a pool of long-running-processes. Each of long-running-processes is a small Twisted TCP server that gives an access to one copy of the C-program.
In example, a user ask the pool-server to reserve a long-running-process for him, then the pool-server create and run a long-running-process that starts listening on some port, then the pool-server return the host and port of this long-running-process to user. Now, user can communicate with this long-running-process directly.
Questions
How start these long-running-process from the pool-server? The pool-server and each of long-running-processes are should be separate Twisted servers.
Is Twisted a good choice for these aims?
Maybe there are another ways how solve this design problem?
Thanks a lot.
Using Twisted for this sounds like it makes sense. The low-level API that supports running processes in Twisted is reactor.spawnProcess. Some examples of its usage are given in the process howto document.
This API is well suited for dealing with many long running processes, and as with all I/O APIs in Twisted, it works well when combined with another event source, such as a web server users can use to request new processes be started up.
reactor.spawnProcess is more like the standard library subprocess module than like the multiprocessing package. It gives you a way to launch a child process running a particular executable, with particular arguments, etc. It does not provide a high-level API for running a particular Python function in another process. However, it's not too hard to build such a thing (at least for a particular case). Consider this approach:
from sys import executable
from os import environ
from twisted.internet import reactor
implementation = """\
from yourapp import somefunction
somefunction()
"""
reactor.spawnProcess(executable, [executable, "-c", implementation], env=environ)
reactor.run()
This just launches a new Python interpreter (whichever one you happen to be running) and uses the -c option to specify a program on the command line for it to run.
This is sort of related to a previous, yet so far unsuccessful question of mine. I have a daemon that is placed in the LaunchAgents folder (on Mac) and it should run perpetually in the background, but after a couple of days it just stops for no apparent reason. I have no idea why and thus my question:
What are the reasons that a daemon might randomly stop?
Thanks for the help!
A Daemon is just a long lasting (forked) process. The reason a Daemon crashes is the same any other program crashes:
attempting to read or write memory
that is not allocated for reading or
writing by that application
(segmentation fault) or x86 specific
(general protection fault)
attempting to execute privileged or
invalid instructions
attempting to perform I/O operations
on hardware devices to which it does
not have permission to access
passing invalid arguments to system
calls
attempting to access other system
resources to which the application
does not have permission to access
(bus error)
attempting to execute machine
instructions with bad arguments
(depending on CPU architecture):
divide by zero, operations on denorms
or NaN values, memory access to
unaligned addresses, etc.
Since it's a LaunchAgent, it runs as part of your login session, and hence will be killed if you log out.
On the other hand, if it's dying before you log out, and you can't find/fix whatever is causing it to crash/exit, or you can tell launchd to automatically restart it by adding
<key>KeepAlive</key>
</true>
to its .plist