I want to start redis and redis-scheduler from a rake task so I'm doing the following:
namespace :raketask do
task :start do
system("QUEUE=* rake resque:work &")
system("rake redis:start")
system("rake resque:scheduler")
end
end
The problem is the redis starts in the foreground and then this never kicks off the scheduler. If It won't start in the background (using &). Scheduler must be started AFTER redis is up and running.
similar to nirvdrum. The resque workers are going to fail/quit if redis isn't already running and accepting connections.
check out this gist for an example of how to get things started with monit (linux stuff).
Monit allows one service to be dependent on another, and makes sure they stay alive by monitoring a .pid file.
That strikes me as not a great idea. You should have your redis server started via an init script or something. But, if you really want to go this way, you probably need to modify your redis:start task to use nohup and background the process so you can disconnect from the TTY and keep the process running.
Related
On a celery service on CENTOS which runs a single task at a time, the termination of a task is simple:
revoke(id, terminate=True, signal='SIGINT')
However while the interrupt signal is being processed, the running task gets revoked. Then a new task - from the queue - starts on the node. This is troublesome. Two task are running at the same time on the node. The signal handling could take up to a minute.
The question is how a signal could be sent to a running task, without actually terminating the task in celery?
Or let's say is there any way to send a signal to a running task?
The assumption is user should be able to send a signal from a remote node. In other words user does not have access to list the running processes of the node.
Any other solution is welcome.
I don't understand your goal.
Are you trying to kill the worker? if so, I guess you are talking about t "Warm shutdown", so you can send the SIGTEERM to the worker's process. The running task will get a chance to finish but no new task will be added.
If you're just interested in revoking a specific task and keep using the same worker, can you share your celery configuration and the worker command? are you sure you're running with concurrency 1 ?
Recently, I started to have some trouble with one of me Redis cluster. used_memroy and used_memory_rss increasing constantly.
According to some Googling, I found following discussion:
https://github.com/antirez/redis/issues/4570
Now I am wandering if it is safe to run SCRIPT FLUSH command on my production Redis cluster?
Yes - you can run the SCRIPT FLUSH command safely in a production cluster. The only potential side effect is blocking the server while it executes. Note, however, that you'll want to call it in each of your nodes.
I have a Jenkins instance to which I am sending build request programmatically through API. My server gets restarted once in a day.
I have observed that when Jenkins server gets restarted, Jenkins is not keeping any track of queued Jobs and running jobs. We looses those jobs to be triggered.
Also I wanted to monitor programmatically if the queued build was actually executed or not. But when we restart the Jenkins, queue ids gets started from one.
Is there any way [any plugin] available that persistent the queued build and continue executing after restart in the same order as they were queued ?
Also want to continue queue numbed at which it was there before restart.
According to this and this, /safeRestart should be enough for what you need.
Or you can use Naginator plugin to restart failed (due to Jenkins going down) builds
I have a Rails 3.2.20 app which I'm introducing Resque into the environment to background job mail and sms alerts. I have everything setup in development properly and now I'm to the point of preparing to merge my branch and push to staging and then to production for testing. But I have a few questions.
1.) How much memory will I need to run Resque in production (approximately). I understand that by starting up a Resque worker it loads the full environment. I'm a bit tight on memory and don't want to have any issues. I'll be using a single Resque worker as our email/sms traffic is very light and we don't mind queues being backed up for a few seconds to a minute. I know this question is very vague, but I'd like to get a feel for the memory footprint that Resque requires.
2.) I will have Redis running but need to figure out how to start a Resque worker on deployment as well as kill the existing Resque worker. I've come up with the following which I would add as a Cap after deploy action.
task :resque_restart do
run "kill $(ps aux | grep 'resque' | awk '{print $2}')"
run "cd #{current_path}; bundle exec rake resque:work QUEUE=*"
end
I haven't actually tested this with Capistrano 2 (which I'm using), but have tested the commands manually and the first command does kill all the resque rake tasks and the second command starts up the worker with all Queues enabled.
I'm not sure if this is the best way to go or not, so I'd really like to hear some feedback on this simple Capistrano task I wrote.
3.) What is the best way to monitor my Resque rake task. So for instance if it crashes or catches a signal to terminate, how can I have it restarted so the app doesn't crash and to assure the worker rake task is always running?
Thanks in advance for any advice or guidance you can provide.
It really depends on the size of your app. From my experience, generally a single Resque worker isn't much larger than your app's footprint. However, if your Resque worker will instantiate a lot of large objects, the size of the Resque instance could grow very quickly.
Check out the capistrano-resque gem. It provides all this functionality for you, plus more. :)
There are several options for this. A lot of people have followed something similar to this post about running Resque in Production and using the God gem. Personally, I've used a process similar to what is described in this post using Monit. Monit can be a bit of a pain to set up, so I'd strongly recommend checking out the God gem.
On rails code deployment, sidekiq is re-started and we would like to remove the sidekiq specific redis cache from the instance before it re-starts.
This is what we want to achieve
1. sidekiq:stop
2. connect to remote redis pointed to by sideiq
3. select database (say select 1).
3. remove cache (say flushall)
How should I automate this via capistrano.
You can flush Sidekiq queues by calling them directly, or in their own Rake Task in your Step #3
Sidekiq::ScheduledSet.new.clear #clear the scheduled queue..
Sidekiq::RetrySet.new.clear #clear any quequed retries.