I am using workless gem to scale delayed job related queue workers. In this process, I have to scale two different type of workers using workless. My Procfile is as follows.
worker: QUEUE=monthly_reports rake jobs:work
worker2: QUEUES=Dashboard_Reports,pulling rake jobs:work
Workless automatically scales the first worker but not my worker2. Can anyone suggest how to scale both workers using workless
What you want to do is not possible with the way the workless gem is currently written. If you take a look at the code you'll see that the scaling code has the process type of 'worker' hard coded. You could modify it to support a different process type via an environment variable but it still wouldn't work with multiple process types.
Related
So, I've been given a code base which uses daemons, daemons-rails and delayed jobs to trigger a number of *_ctl files in /lib/daemons/ but here's the problem:
If two people do an action which starts the daemons doing some heavy lifting then whichever one clicks second will have to wait for the first to complete. Not good. We need to start multiple daemons on each queue.
Ideally what I want to do is read a config file like this:
default:
queues: default ordering
num_workers: 10
hours:
queues: slow_admin_tasks
num_workers: 2
minutes:
queues: minute
num_workers: 2
This would mean that 10 daemon processes are started to listen to the default and ordering queues, 2 for slow_admin tasks etc.
How would I go about defining multiple daemons like this, it looks like it might be in one of these places:
/lib/daemons/*_ctl
/lib/daemons/*.rb
/lib/daemons/daemons
I thought it might be a change to the daemons-rails rake tasks, but they just hit the daemons file.
Has anyone looked in to scaling daemons-rails in this way? Where can I get more information?
I suggest you to try Foreman.
Take a look at this discution.
Foreman can help manage multiple processes that your Rails app depends upon when running in development. You can find a tutorial regarding Foreman on RailsCasts. There's a video tutorial + some source code examples.
I also suggest you to take a look at Sidekiq.
Sidekiq allows you to move jobs into the background for asynchronous processing. It uses threads instead of forks so it is much more efficient with memory compared to Resque. You can find a tutorial here.
I also suggest you to take a look at Resque
Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. You can find a tutorial here.
I have a Rails 3 app deployed heroku. I have a Sidekiq worker at app/workers/task_worker.rb:
class TaskWorker
include Sidekiq::Worker
def perform
...
end
end
How to schedule execution of TaskWorker.perform_async daily at 12:01 a.m?
You might want to have a look at sidetiq too. https://github.com/tobiassvn/sidetiq The gem supports complex timing expressions via the ice_cube gem.
I personally found comfortable to have a gem that would integrate seemlessly with sidekiq.
Something like that should work:
class TaskWorker
include Sidekiq::Worker
include Sidetiq::Schedulable
recurrence do
daily.hour_of_day(0).minute_of_hour(1)
end
def perform
# do magic
end
end
Careful though when using this gem since there are some performance related issues with some time expressions. https://github.com/tobiassvn/sidetiq/wiki/Known-Issues. The expression I gave you should circumvent this issue though.
I don't like the overhead Sidetiq adds to Sidekiq so I sought out a different solution.
Apparently Heroku has a little-known, but free scheduler addon that allows you to run rake tasks every 10 minutes, hourly or daily. This is Heroku's answer to cron jobs and it's nice that it's a free add-on. It should work for most non-critical scheduling.
Heroku states in their docs that the scheduler is a "Best Effort" service which may occasionally (but rarely) miss a scheduled event. If it is critical that this job is run, you'll probably want to use a custom clock process. Custom clock processes are more reliable but they count toward your dyno hours. (And as such, incur fees just like any other process.)
Currently it looks like clockwork is the recommended clock process on Heroku.
I'm stating the obvious, but what's wrong with having a Cron Job that invokes the Sidekiq job every night at that time?
I've seen several solutions for doing this:
Redis / Resque
Delayed Job
Heroku Scheduler
Clockwork
Heroku scheduler won't work because it runs at random times and only once per 10 minutes at its most frequent.
Running on Cedar. Running multiple web dynos.
EDIT: Here's what I want to do:
Call an arbitrary method with params at an arbitrary point in the future. Something like Schedule.set(Notification.send_update_to_user(574), Time.now + 1.days)
I would choose Sidekiq, though there are several other options suitable for your example. Sidekiq lets you schedule jobs to run at arbitrary times in the future:
NotificationUpdateWorker.perform_at(Time.now + 1.day, 574)
The delayed extensions would let you write instead:
Notification.delay_for(1.day).send_update_to_user(574)
Try with rufus/scheduler.
require 'rubygems'
require 'rufus/scheduler'
scheduler = Rufus::Scheduler.start_new
scheduler.every '1m' do
Checkin.check_checkin()
end
After looking at different options, I chose Delayed Job, which is well documented on Heroku.
For jobs that need to run at a certain time each day or once an hour, Heroku scheduler works well, but sometimes it doesn't run.
I am developing a Rails3 app which will run on Heroku Cedar stack and needs to constantly check for new tweets under a certain hashtag. I have the logic to do this in place but I would like to run this task in the background so as not to interfere with the main app performance. I also need to write any new tweets found to a database so I will need access to Active Record. I am looking for advise on what might be the best way to achieve this.
I do something similar, it doesn't matter for me if tweets are slightly out of date - we use the scheduler for 10 minute executions of a rake task which is watching a hashtag. We can change the frequency of the executing to hourly/daily should we feel 10 mins is too frequent.
You could use the Heroku scheduler to regularly execute a Rake task (or some other script).
Alternatively, if you're checking for Tweets in response to a certain user action or some other event, you could use a task queue like Delayed Job.
I have a app and i am currently using delayed_job. I was wondering if there are any recommended gems that do scheduling of repetitive tasks.
I want to schedule task that happen on a certain frequency to clean the database/sending emails/run other methods.
I mite want to run some tasks every day or every hour.
Is there any good ones out there that are fairly easy to setup and config which do not use CRON.
You can convert that repetitive work to rakes and call those rakes via cron.
For setting cron over server, whenever is nice gem
checkout here - http://railscasts.com/episodes/164-cron-in-ruby
there are some very good gems about schedule repetitive tasks
i tried delayed job it is simple and easy to use you can watch rails cast to details of using it
You can try rufus-scheduler https://github.com/jmettraux/rufus-scheduler It is an app based scheduler (unless you use it's cron feature).