I am migrating my existing rails app to heroku. I have memory and time intensive delayed jobs that run almost 20 hours a day and I have a clockwork to handle the time specific jobs. the clockwork jobs are not so heavy and run a very few times in one day.
Is is possible to run both the delayed job process and the clockwork process using a single heroku process using bluepill ?
I do not want to pay for one more worker just for the sake of clockwork processes.
Try using heroku scheduler, it will allow you to schedule jobs on a daily, hourly or 10 minute interval.
Related
I'm trying to execute a process to update my database, but the problem is that I set different RecurringJobs for it at different hours.
Today when I checked hangfire status, since yesterday that I instanced hangfire, I found the job should execute yesterday and the one task for today, both executed 30 minutes ago at the same time, and this has created duplicates in the database.
Can you help me with this?
If your problem is one of concurrency, you can solve it by running hangfire single threaded. Simply configure the number of hangfire worker threads on startup:
var server = new BackgroundJobServer(new BackgroundJobServerOptions
{
WorkerCount = 1
});
This will force hangfire to process queued jobs sequentially.
Alternatively, if you have the Pro version of hangfire you can control order using batch chaining.
I don't know if a worker can be considered as a thread.
Within a hangfire worker, single threaded code will be run by exactly one thread
This doesn't look like a concurrency issue as has been suggested. It's not completely clear what you are trying to do but I'm assuming you want the job to run at 7, 12:45, and 17:30 and had issues because both the 7am and 17:30 job ran at the same time (7am).
Based on the created time it looks like you created these around 14:30. That means the 17:30 job should have ran but didn't until the next morning around 7am. My best guess is this was hosted in IIS and the site app pool was recycled.
This would cause any recurring jobs that were supposed to run to be delayed until the app pool / site was started again (which I assume was around 7am).
Check out these documents on how to ensure your site is always running: http://docs.hangfire.io/en/latest/deployment-to-production/making-aspnet-app-always-running.html
If it's not an IIS issue something must have caused the BackgroundJobServer to stop monitoring the database for jobs until ~7:00am (server shutdown, error, etc).
I'm working on a workflow that has both Hive and Java actions. Very often we have been noticing that there is a few minutes delay between Java action start time and the job submission time. We don't see that with Hive jobs, meaning Hive jobs seem to be submitted almost immediately after they are started. The Java jobs do not do much and so they finish successfully in seconds after they are submitted but the time between start and submission seem to be very night ( 4 -5 minutes). We are using fair scheduler and the there are enough mapper/reducer slots available. But still even if it's a resource problem the Hive jobs should also show delay between start and submission but they don't ! Java jobs are very simple jobs and they don't process any files etc and basically used to call a web service and they spawn only single mapper and no reducers where are the Hive jobs creates hundreds of mapper/reducer tasks but still there is not delay between start and submission. We are not able to figure out why oozie is not submitting the Java job immediately. Any ideas?
I've seen several solutions for doing this:
Redis / Resque
Delayed Job
Heroku Scheduler
Clockwork
Heroku scheduler won't work because it runs at random times and only once per 10 minutes at its most frequent.
Running on Cedar. Running multiple web dynos.
EDIT: Here's what I want to do:
Call an arbitrary method with params at an arbitrary point in the future. Something like Schedule.set(Notification.send_update_to_user(574), Time.now + 1.days)
I would choose Sidekiq, though there are several other options suitable for your example. Sidekiq lets you schedule jobs to run at arbitrary times in the future:
NotificationUpdateWorker.perform_at(Time.now + 1.day, 574)
The delayed extensions would let you write instead:
Notification.delay_for(1.day).send_update_to_user(574)
Try with rufus/scheduler.
require 'rubygems'
require 'rufus/scheduler'
scheduler = Rufus::Scheduler.start_new
scheduler.every '1m' do
Checkin.check_checkin()
end
After looking at different options, I chose Delayed Job, which is well documented on Heroku.
For jobs that need to run at a certain time each day or once an hour, Heroku scheduler works well, but sometimes it doesn't run.
I've been reading through Heroku's documentation but just found it plain confusing. I have an app up that has both a web-based front-end (with web process) and a task that's set to run every day at midnight by Heroku Scheduler (shows up on heroku ps as run.1).
So, my heroku ps looks like this:
Process State Command
------- ---------- ------------------------------------
run.1 up for 21h python webpage/listings.py
web.1 up for 8m python ./manage.py runserver 0.0.0..
What I'm trying to figure out is, is this considered two dynos? Is the run task considered a background task?
Main question: Will this cost money?
Yes, a Heroku Scheduler will accrue usage and will cost money if you go over your 750 free dyno-hours you are given per app each month. As long as you keep within that limit, you won't be charged.
Scheduler runs one-off dynos, which accrue usage just like regular dynos. They will appear with a “scheduler” dyno type in your Heroku invoice.
There is 750 of free hours.
In the billing, dynos are divided in four groups: worker (background dynos), web dynos, rake and one-off-process (when executing "heroku run", for example used by the scheduler).
More at https://devcenter.heroku.com/articles/usage-and-billing
I use hirefire-gem with Delayed-Job 3 on heroku cedar-stack and it is working pretty good in terms of hiring/firing but performance of the job execution is terrible. firing up the background job and seeing the results in the UI takes about 5-8 seconds locally and about 25-30 seconds (!) on heroku.
Processing time of the jobs is about the same locally/deployed but hiring workers (scaling, up, starting, ...) seems to take a lot of time(?).
is that a common issue? is there a solution (rake tasks, etc.)?
Thanks a lot.
Best, Phil
It's down to the fact that your worker isn't running all the time but spinning up for each individual job. The lag is the code start-up time.
If you have a full time dyno the jobs should process almost instantaneously.