I'm currently using the delayed_job gem to queue and run background tasks in my application. In the local system, I can just use rake jobs:work to run the queued tasks. However, when I deploy my app onto Heroku, I do not want to continue using the rake command. Instead, I want the rake command to be called automatically. Is there a way to do so, without paying for a worker in Heroku?
I use cron with out problems (with django). All do you need is to configure as task the same command that you can execute after heroku run command.
Remember that cron time compute as worker time, be sure that command ends.
No, you can't do it without a worker.
The earlier point saying you need a worker is right, however you do have free worker hours. There 750 free hours per month http://www.heroku.com/pricing#1-0. Given a 31 day month is 744 hours, you have at least 6 free worker hours to use each month.
If you use the workless gem https://github.com/lostboy/workless this will spin up the worker only when needed (i.e. jobs waiting in delayed_job), then close it down again. Works perfectly for my app, and 6 hours of background processing time a month is more than enough for my requirements.
Related
Spinnaker on Gke :
Pipeline is getting failed after starting manually
the first issue is the .spin/config didnt get created, I created that manually as missing in the steps https://cloud.google.com/solutions/continuous-delivery-spinnaker-kubernetes-engine
then when i started pipeline manually , it is giving me an error on production stage
Exception ( Wait For Manifest To Stabilize )
WaitForManifestStableTask of stage Deploy Production Backend timed out after 30 minutes 4 seconds. pausedDuration: 0 seconds, elapsedTime: 30 minutes 4 seconds,timeoutValue: 30 minutes
Many times this is caused by Kubernetes being unable to schedule the pods. Just had the same problem this morning and it was due to lack of resources. That would be the first thing I'd check. You can find the pod in the "Server Group" under the "Infrastructure" tab and view the details of why it's in the state it is.
I find this Spinnaker GitHub page insightful. I wanted to add it as a comment because I still don't have enough reputation.
I'm using a open-source chef server managing about 150 nodes.
Analytics/Reporting module is not activated in the chef server due to resource constraints.
"chef-client" is running on all the nodes every 30 minutes
How can I find, how much time each chef-client run is taking to complete?
I'm trying to find the nodes that are slowest in completing their chef-client runs
Chef Server doesn't store this information. You'll need to manage it yourself, possibly using a handler as linked above in the comments. A simple option would be to make a handler which stores the duration of the last run as a node attribute, but the sky is the limit. If you want something to help debug long runs once you find them, check out my poise-profiler cookbook.
I'm developing a Rails 3.2.16 app and deploying to a Heroku dev account with one free web dyno and no worker dynos. I'm trying to determine if a (paid) worker dyno is really needed.
The app sends various emails. I use delayed_job_active_record to queue those and send them out.
I also need to check a notification count every minute. For that I'm using rufus-scheduler.
rufus-scheduler seems able to run a background task/thread within a Heroku web dyno.
On the other hand, everything I can find on delayed_job indicates that it requires a separate worker process. Why? If rufus-scheduler can run a daemon within a web dyno, why can't delayed_job do the same?
I've tested the following for running my every-minute task and working off delayed_jobs, and it seems to work within the single Heroku web dyno:
config/initializers/rufus-scheduler.rb
require 'rufus-scheduler'
require 'delayed/command'
s = Rufus::Scheduler.singleton
s.every '1m', :overlap => false do # Every minute
Rails.logger.info ">> #{Time.now}: rufus-scheduler task started"
# Check for pending notifications and queue to delayed_job
User.send_pending_notifications
# work off delayed_jobs without a separate worker process
Delayed::Worker.new.work_off
end
This seems so obvious that I'm wondering if I'm missing something? Is this an acceptable way to handle the delayed_job queue without the added complexity and expense of a separate worker process?
Update
As #jmettraux points out, Heroku will idle an inactive web dyno after an hour. I haven't set it up yet, but let's assume I'm using one of the various keep-alive methods to keep it from sleeping: Easy way to prevent Heroku idling?.
According to this
https://blog.heroku.com/archives/2013/6/20/app_sleeping_on_heroku
your dyno will go to sleep if he hasn't serviced requests for an hour. No dyno, no scheduling.
This could help as well: https://devcenter.heroku.com/articles/clock-processes-ruby
I use the delayed job gem to handle my email deliveries. It is working fine in the development and I am very happy with it. However after I deployed to the server, when I use command:
RAILS_ENV=production script/delayed_job start
it will be working. I've checked the log file and database, everything is fine and I can receive the mails just as I expected. However, when I exit from the server, nothing is going to happen.
I've checked my database by using sequel pro and seen that the delayed job has created a row in the DB and after the time in the run_at column, the row would disappear, but no mails can be received. When I log in again, the delayed job process is still running, and the log is nothing strange, but I just cannot receive and email that I suppose to. I can't keep my self log in all the time. Without the delayed job, I can use the traditional way and it's working properly but slow. Why the delayed job failed after I log out of the server?
This is my delayed job setting in the config/initializers/delay_job.rb
require "bcrypt"
Delayed::Worker.max_attempts = 5
Delayed::Worker.delay_jobs = !Rails.env.test?
Delayed::Worker.destroy_failed_jobs = false
P.S. I am not sure is it anything to do with the standalone passenger as I have to use different version of rails so I have to use a standalone passenger with port 3002.
I think I've found the solution.
After reading through this https://github.com/collectiveidea/delayed_job/wiki/Common-problems#wiki-jobs_are_silently_removed_from_the_database
I soon realized I might miss the "require bcrypt" in the configuration file.
I use RVM and have many gemsets, but just this particular gemset has the gem bcrypt-ruby. The delayed job might use the global or default gemset after I log out the system, so I install bcrypt-ruby in all the gemsets and restart the standalone passenger and it works!.
But still, I dont really know the connection between bcrypt and the delayed job.
I have a worker doing some processing 24/7. However, sometimes the code crashes and it needs to be restarted (even if I catch the exception, I have to restart the worker in order for it to work).
What do you do when this happens or am I doing something wrong and this shouldn't happen at all? Does your dynos/workers crash or it is just me?
thanks
Heroku is supposed to restart a worker every time it crashes. As far as I know, you don't have to select or configure anything. Whatever is in your jobs:work task will be executed as soon as it fails.
In the event that you are heavily dependent on background jobs in your web app. You could create a rake task that finds the last record to be updated and execute a background job to update it. Or perhaps automate the rake task to find the rest of the records that need updating, since the last crash.
Alternatively, you force worker restart manually as indicated in this article (using delayed_job):
heroku workers 0;
heroku workers 1;
Or perhaps you can restart a specific worker by doing (mentioned in this article):
heroku restart worker.1
By the way, try the 1.9 stack. Make sure your app is 1.9.2 compatible, before doing so. Hopefully crashes are less frequent there:
heroku stack:migrate bamboo-mri-1.9.2
In the event, that such issues still arise. Best to contact Heroku support. They are very responsive at what they do.
Latest command to restart a specific heroku web worker (2014):
heroku ps:restart web.1
(tested on Cedar stack)
At times, for instance in case of DB crashes, the worker may not restart automatically. you would need to do this.
heroku restart web.1
It worked for me.