We use PHP gearman workers to run various tasks in parallel. Everything works just fine, and I have silly little shell script to spin them up when I want them. Being a programmer (and therefore lazy), I wanted to see if I could spin these up via an upstart script.
I figured out how to use the instance stanza, so I could start them with an instance number:
description "Async insert workers"
author "Mike Grunder"
env SCRIPT_PATH="/path/to/my/script"
instance $N
script
php $SCRIPT_PATH/worker.php
end script
And this works great, to start them like so:
sudo start async-worker N=1
sudo start async-worker N=2
The way I want to use these workers is to spin up some number of them (maybe one per core, etc), and I would like to do this on startup. To be clear, I don't need the upstart script to detect the number of cores. I'm happy to just say "do 8 instances", but that's why I want multiple running. Is there a way for me to use the "start on" clause in an upstart script to do this automatically?
For example, start instance 1, 2, 3, 4? Then have them exit on shutdown properly?
I suppose I could hook this into an init.d script, but I was wondering if upstart can handle something like this, or if anyone has figured out this issue.
Cheers guys!
What you need is a bootstrap task that runs on startup and iterates over all your worker jobs, starting each one.
#/etc/init/async-workers-all.conf
start on runlevel [2345]
task
env NUM_WORKERS=8
script
for i in `seq 1 $NUM_WORKERS`
do
start async-worker N=$i
done
end script
The key is to make this a task, which tells upstart to let the task run to completion before emitting any events for it. See http://upstart.ubuntu.com/cookbook/#task and http://upstart.ubuntu.com/cookbook/#instance
Related
Is it possible to execute the Unidata process from the Unix Command line??
If it's possible, can anyone please let me know how to??
I just want to add some Unidata Processes into the shell script and run it from the Unix
Cron job.
Unidata Process
Unix Command line
Yes! There are several approaches, depending on how your application is setup.
Just pipe the input to the udt process and let 'er rip
$cd /path/to/account
$echo "COUNT VOC" | udt
This will run synchronously, and you may have to also respond to any prompts your application puts up, unless it is checking to see if the session is connected to a tty. Check the LOGIN paragraph in VOC to see what runs at startup.
Same, but run async as a phantom
$cd /path/to/account
$udt PHANTOM COUNT VOC
This will return immediately, the commands will run in the background. Have to check the COMO/PH file for the output from the command. It's common for applications to skip or have a cut down startup process when run as a phantom (check for #USERTYPE)
If none of the above work because of the way your application is written, use something like expect to force the issue.
spawn udt
expect "ogin:"
send "rubbleb\r"
etc.
https://en.wikipedia.org/wiki/Expect for more info on expect
So I am trying to keep my Node server on a embedded computer running when it is out in the field. This lead me to leveraging inittab's respawn action. Here is the file I added to inittab:
node:5:respawn:node /path/to/node/files &
I know for a fact that when I startup this node application from command line, it does not get to the bottom of the main body and console.log "done" until a good 2-3 seconds after I issue the command.
So I feel like in that 2-3 second window the OS just keeps firing off respawns of the node app. I see in the error logs too in fact that the kernel ends up killing off a bunch of node processes because its running out of memory and stuff... plus I do get the 'node' process respawning too fast will suspend for 5 minutes message too.
I tried wrapping this in a script, dint work. I know I can use crontab but thats every minute... am I doing something wrong? or should I have a different approach all together?
Any and all advice is welcome!
TIA
Surely too late for you, but in case someone else finds such a problem: try removing the & from the command invocation.
What happens is that when the command goes to the background (thanks to the &), the parent (init) sees that it exited, and respawns it. Result: a storm of new instantations of your command.
Worse, you mention embedded, so I guess you are using busybox, whose init won't rate-limit the respawning - as would other implementations. So the respawning will only end when the system is out of memory.
inittab is overkill for this. I found out what I need is a process monitor. I found one that is lightweight and effective; it has some good reports of working great out in the field. http://en.wikipedia.org/wiki/Process_control_daemon
Using this would entail configuring this daemon to start and monitor your Node.js application for you.
That is a solution that works from the OS side.
Another way to do it is as follows. So if you are trying to keep Node.js running like I was, there are several modules written meant to keep other Node.js apps running. To mention a couple there are forever and respawn. I chose to use respawn.
This method entails starting one app written in Node.js that uses the respawn module to start and monitor the actual Node.js app you were interested in keeping running anyway.
Of course the downside of this is that if the Node.js engine (V8) goes down altogether then both your monitoring and monitored process will go down with it :-(. But its better than nothing!
PCD would be the ideal option. It would go down probably only if the OS goes down, and if the OS goes down then hope fully one has a watchdog in place to reboot the device/hardware.
Niko
Is there any way to call mprirun inside FORTRAN program? I'm working on public linux cluster via ssh and the main idea is to automatically enqueue program after its execution is over.
I tried to write something like this at the end of the program:
CALL system('mpirun -np 16 -maxtime 100 TestNP')
But recieved this error:
sh: mpirun: command not found
Any ideas ?
The problem is the missing path prefix, so specifying an absolute path for mpirun should help. However there are several problems with your approach:
If every MPI process executes it, you would have too many instances running, so only one of the nodes (e.g. the master node) should execute it.
The original program won't be finished, until the one called via the system() call did not finish. So, if your queue is wall-clock limited, you don't gain anything at all.
Typically, tasks like this are done via shell-scripts. E.g. in Bash you would write something like:
while true; do
mpirun your_program
done
This would re-invoke mpirun continuously until not killed by you or the queuing system. (So be careful with it!)
I have a complex screen-scraping script that I've put together that uses Selenium2, the Selenium web driver and PHP binding script, so at the end of it all, I have a PHP script that drives Selenium, which in turn fetches a URL, parses some Javascript, fills out a form, blah blah blah, and then returns the HTML that is ultimately what I'm after. It all works great on my local computer (as a development and proof-of-concept environment).
So.
For production, I need this script to run automatically three times every day. I am trying to figure out if it would be better for me to set up everything on my server (meaning: figure out how to get Firefox for Linux going, then Java, then Selenium2, etc, etc... not trivial for me; Damn it Jim, I'm a coder, not a sysadmin!), or if I can use a 3rd-party Selenium testing service like Sauce Labs' OnDemand, or any of these other cloud-based Selenium services.
Those 3rd party solutions seem like they're all set up for "unit testing," which is totally not what I'm doing. I don't know about that stuff, or using PHPUnit, or doing tests with builds, or whatever. I just want to run my straightforward PHP script 3x/day and have it talk to Selenium to drive a browser and do my screen scraping.
Are one of those 3rd party solutions a good idea for what I'm trying to accomplish, or are they overkill/too far away from my (relatively simple) goal?
First, I want to let you know that I use Selenium with Ruby so I am assuming that running your php script will start up the selenium webdriver and run your tests... I will just explain how easily run your script 3 times a day without needing to be a sysadmin master.
Linux has an extremely stable and robust command called cron which is what you will need to use. It allows you to schedule actions to happen daily/hourly/whatever.
The first thing you want to do is to go to the directory with your script. I will refer to your script as script.php.
First thing is to make sure that the top line of your script is:
#!/usr/bin/php
In the directory you will execute the following command to make your file accessible by the system:
chmod +x script.php
Now set up your cron job with the following command:
crontab -e
Then put in your job:
00 4,12,20 * * * /home/sean/script.php
00 - Means at 00 minutes.
4,12,20 - Are the hours (it is a 24 hour clock.)
The first: * - Every day
The second: * - Every month
The third: * - Every Day of the week
So this script would run every day, every week, every month at 4,noon and 8pm.
Obviously change the directory to the script on your system and set the times to whenever you want the scraping to occur.
I hope this helps!
-Appended stuff for the java/firefox-
First off, take this all with a grain of salt since I am using Ruby :)
Okay to get java/firefox running you will probably want to grab the selenium standalone. You can grab it here.
Then to run the selenium server you just:
java -jar selenium-server-standalone-2.5.0.jar
You can run put the standalone server starting in the cron job and then close it in your script file.
I´m writing a Powershell script to do a bunch of things and when finished it will be run as a scheduled task. For that reason I want to be able to check whether an older instance is still alive when I start running the script and kill the older one if it exists.
I was thinking I would use something like this:
$process = Get_Process | $name
$process.kill
But how to get the $name variable in a simple way?
Does anyone have a better suggestion?
Best regards,
Gísli
You can do this in windows scheduled task configuration. The settings depends on the OS you are using though.
EDIT: that is you can configure the task to be killed after a certain period of time (i.e. when your next one starts).
Why do you need to get the name? Get-Process returns high fidelity Process objects and you can operate on it directly.
To get a process of a particular name use $n = Get-Process notepad, say, and then do $n.kill() to kill it. If you do need to check the name again, do $n.Name. To see what properties and methods you can use, try $n | get-member
And make sure you read the manual: http://technet.microsoft.com/en-us/library/dd347630.aspx