My initializers/task_scheduler.rb
require 'rubygems'
require 'rufus/scheduler'
scheduler = Rufus::Scheduler.start_new
scheduler.cron '*/5 * * * *' do
Project.cron_for_report
end
My Issue is I am getting 12 mails of report every 5 minutes. I should get only 1 mail every 5 minute. Should I use something like mutex?? Please give some example code.
Please help.
Thanks in advance.
It does not sound like a rufus-scheduler issue, it sounds like you are running 12 Rails processes, but you don't tell what you're running Rails on (Unicorn, Passenger, ... ?).
Since you're running on Ubuntu, you are free to use crond. Why don't you have a look at https://github.com/javan/whenever ? It'll let you leverage crond's strength from your Rails application.
Rufus-scheduler is fun, but if you simply start it each time Rails starts, you'll end up with a rufus-scheduler instance for each Rails process, unless you place some logic in the initializer to avoid such a situation (lock files, etc). And your logic has to be smart enough to avoid locks pointing on dead schedulers preventing starting new, needed, schedulers, etc.
Since you deploy on Linode/Ubuntu, you're free to consider other alternatives like Whenever.
Sorry, no example code, I don't want you to become a copy-paste programmer.
Related
So I am trying to keep my Node server on a embedded computer running when it is out in the field. This lead me to leveraging inittab's respawn action. Here is the file I added to inittab:
node:5:respawn:node /path/to/node/files &
I know for a fact that when I startup this node application from command line, it does not get to the bottom of the main body and console.log "done" until a good 2-3 seconds after I issue the command.
So I feel like in that 2-3 second window the OS just keeps firing off respawns of the node app. I see in the error logs too in fact that the kernel ends up killing off a bunch of node processes because its running out of memory and stuff... plus I do get the 'node' process respawning too fast will suspend for 5 minutes message too.
I tried wrapping this in a script, dint work. I know I can use crontab but thats every minute... am I doing something wrong? or should I have a different approach all together?
Any and all advice is welcome!
TIA
Surely too late for you, but in case someone else finds such a problem: try removing the & from the command invocation.
What happens is that when the command goes to the background (thanks to the &), the parent (init) sees that it exited, and respawns it. Result: a storm of new instantations of your command.
Worse, you mention embedded, so I guess you are using busybox, whose init won't rate-limit the respawning - as would other implementations. So the respawning will only end when the system is out of memory.
inittab is overkill for this. I found out what I need is a process monitor. I found one that is lightweight and effective; it has some good reports of working great out in the field. http://en.wikipedia.org/wiki/Process_control_daemon
Using this would entail configuring this daemon to start and monitor your Node.js application for you.
That is a solution that works from the OS side.
Another way to do it is as follows. So if you are trying to keep Node.js running like I was, there are several modules written meant to keep other Node.js apps running. To mention a couple there are forever and respawn. I chose to use respawn.
This method entails starting one app written in Node.js that uses the respawn module to start and monitor the actual Node.js app you were interested in keeping running anyway.
Of course the downside of this is that if the Node.js engine (V8) goes down altogether then both your monitoring and monitored process will go down with it :-(. But its better than nothing!
PCD would be the ideal option. It would go down probably only if the OS goes down, and if the OS goes down then hope fully one has a watchdog in place to reboot the device/hardware.
Niko
It appears like it would be nearly impossible to read through the logs of your production application every morning searching for errors and the lines in which those errors are occurring. Is there something I can use which will automatically generate a daily report for me of any errors that occur in my application and the location of those errors. Something I can review each morning without taking up too much time?
I'd suggest picking a service that errors can be reported to and aggregated, etc. There are quite a few options. I personally use errbit (free, self hosted) in conjunction with the airbrake gem (which can be configured to use it). Or you can signup for Airbrake itself. Or use Exceptional.io, or one of the other services.
https://github.com/errbit/errbit
https://airbrakeapp.com/pages/home
http://www.exceptional.io/
http://newrelic.com/
https://www.ruby-toolbox.com/categories/exception_notification
Is there a plugin/package to display status information for a PBS queue? I am currently running an apache webserver on the login-node of my PBS cluster. I would like to display status info and have the ability to perform minimal queries without writing it from scratch (or modifying an age old python script, ala jobmonarch). Note, the accepted/bountied solution must work with Ubuntu.
Update: In addition to ganglia as noted below, I also looked that the Rocks Cluster Toolkit, but I firmly want to stay with Ubuntu. So I've updated the question to reflect that.
Update 2: I've also looked at PBSWeb as well as MyPBS neither one appears to suit my needs. The first is too out-of-date with the current system and the second is more focused on cost estimation and project budgeting. They're both nice, but I'm more interested in resource availability, job completion, and general status updates. So I'm probably just going to write my own from scratch -- starting Aug 15th.
Have you tried Ganglia?
I have no personal experience but few sysadmin I know are using it.
Following pages may help,
http://taos.groups.wuyasea.com/articles/how-to-setup-ganglia-to-monitor-server-stats/3
http://coe04.ucalgary.ca/rocks-documentation/2.3.2/monitoring-pbs.html
my two cents
Have you tried using nagios: http://www.nagios.org/ ?
I have a PHP script that seemed to stop running after about 20 minutes.
To try to figure out why, I made a very simple script to see how long it would run without any complex code to confuse me.
I found that the same thing was happening with this simple infinite loop. At some point between 15 and 25 minutes of running, it stops without any message or error. The browser says "Done".
I've been over every single possible thing I could think of:
set_time_limit ( session.gc_maxlifetime in the php.ini)
memory_limit
max_execution_time
The point that the script is stopped is not consistent. Sometimes it will stop at 15 minutes, sometimes 22 minutes.
Please, any help would be greatly appreciated.
It is hosted on a 1and1 server. I contacted them and they don't provide support for bugs caused by developers.
At some point your browser times out and stops loading the page. If you want to test, open up the command line and run the code in there. The script should run indefinitely.
Have you considered just running the script from the command line, eg:
php script.php
and have the script flush out a message every so often that its still running:
<?php
while (true) {
doWork();
echo "still alive...";
flush();
}
in such cases, i turn on all the development settings in php.ini, of course on a development server. This display many more messages, including deprecation warnings.
In my experience of debugging long running php scripts, the most common cause was memory allocation failure (Fatal error: Allowed memory size of xxxx bytes exhausted...)
I think what you need to find out is the exact time that it stops (you can set an initial time and keep dumping out the current time minus initial). There is something on the server side that is stopping the file. Also, consider doing an ini_get to check to make sure the execution time is actually 0. If you want, set the time limit to 30 and then EVERY loop you make, continue setting at 30. Every time you call set_time_limit, the counter resets and this might allow you to bypass the actual limits. If this still isn't working, there is something on 1and1's servers that might kill the script.
Also, did you try the ignore_user_abort?
I appreciate everyone's comments. Especially James Hartig's, you were very helpful and sent me on the right path.
I still don't know what the problem was. I got it to run on the server with using SSH, just by using the exec() command as well as the ignore_user_abort(). But it would still time out.
So, I just had to break it into small pieces that will run for only about 2 minutes each, and use session variables/arrays to store where I left off.
I'm glad to be done with this fairly simple project now, and am supremely pissed at 1and1. Oh well...
I think this is caused by some process monitor killing off "zombie processes" in order to allow resources for other users.
Run the exec using "2>&1" to log anything including stderr.
In my output I managed to catch this:
...
script.sh: line 4: 15932 Killed php5-cli -d max_execution_time=0 -d memory_limit=128M myscript.php
So something (an external force, not PHP itself) is killing my process!
I use IdWebSpace which is excellent BTW but I think most shared hosting providers impose this resource/process control mechanism just to be sane.
Every morning when I get into work I launch about a dozen apps and whatnot (FF, TB, VSx2-3, Eclipse, SSH, SVN update x2-3). Needles to say this does a good job of warming up my HDD for the day. I rather suspect that it would run a lot faster if they were launched sequentially (not to mention that I wouldn't need to click in 17 different places).
Is there a preexisting product that can kick off a sequence of tasks/apps/etc. where each task is only started after the last app is done hammering the HDD?
It would nerd to be able to kick apps like VS and firefox and also be able to trigger explorer context menu items like SVN update in TortoiseSVN.
Try SlickRun, it's free, I've used it for years, I use it constantly and I'd be lost without it.
Think of it like a configurable Start->Run command, it'll do what you want (you can configure n second pauses between multiple commands), and if you install it you'll use it for a thousand different things before the first week is out.
P.S. I have no stake in SlickRun, I just like it :)
Unfortunately, I don't know of any software that can do this for you automatically.
However, can't you trigger the updates through a console SVN task? If so, can't this be done by creating a batch file? It's low tech, and you might want to add a few pauses between each task, but it should do what you want.
As you mention TortoiseSVN, I'll assume your O/S is windows.
You could launch an Autohotkey script at startup. I don't think it can easily detect HDD activity, but you can at least wait until each window appears with the WinWaitActive command.
If each application has an average time they take to complete, you could simply use Windows' Scheduled Tasks application. Obviously you'll need to be running Windows but Scheduled Tasks can be found in the Control Panel.
Execute "Add Schedules Task", select the program, the frequency and then the specific time.