Is there a way to show or use the queue length of one of the gems delayed job or resque in one of my views in rails 3?
I'd like to implement something like "you are number X in line, please wait"
assume I have many many jobs (like sending a lot of emails) and the queue does not empty as soon as there is something in it.
Not sure of delayed_job but you for resque can get the information about
the length of the message in resque queue provide you know the queue name
Here How
require "rubygems"
require "redis"
require 'redis/namespace'
redis = Redis.new
namespace ||= :resque
#redis = Redis::Namespace.new(namespace, :redis => redis)
puts #redis.llen "queue:[my queue name]"
llen is redis command to et the length of a lists omething like .length and .count for array in ruby
Hope this help
Related
I have installed Celery with RabbitMQ.
Problem is that for every result that is returned, Celery will create in the Rabbit, queue with the task's ID in the exchange celeryresults.
I still want to have results, but on ONE queue.
my celeryconfig:
from datetime import timedelta
OKER_URL = 'amqp://'
CELERY_RESULT_BACKEND = 'amqp'
#CELERY_IGNORE_RESULT = True
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT=['json', 'application/json']
CELERY_TIMEZONE = 'Europe/Oslo'
CELERY_ENABLE_UTC = True
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
'every-minute': {
'task': 'tasks.remote',
'schedule': timedelta(seconds=30),
'args': (),
},
}
Is that possible? How?
Thanks!
amqp backend creates a new queue for each task. Alternatively, there is a new rpc backend which keeps results in a single queue.
http://docs.celeryproject.org/en/master/whatsnew-3.1.html#new-rpc-result-backend
Nothing unusual.
That is how celery works when we use amqp as result backend. It will create a new temporary queue for every result corresponding to each tasks that worker consumes.
If you are not interested in the result, you can try CELERY_IGNORE_RESULT = True setting
If you do want to store the result, then i would recommend using a different result backend like Redis.
You say you want Celery to keep the result on one queue. Now, to answer your question, let me ask you one:
How do you expect each producer to check for it's relevant result without reading every single message off the queue to find the one it needs/wants?
In essence, what you want is a database of key-value pairs so that the lookup is O(1). The only way to do that with a queue broker is to create one queue for each "pair".
I understand that having many GUID queues is not neat or pretty, but it's conceptually the only way to do it on a messaging broker.
This solution won't keep all the results to ONE queue, but it will at least clean up the extra queues right when you're done with them.
If you use Redis as your backend, when you're done with a result that has created an errant queue, run result.forget(). This will cause both the result and the queue for the result to disappear. This can help you manage the number of queues you have, and prevent OOM issues.
I am using celery with a rabbitmq backend. It is producing thousands of queues with 0 or 1 items in them in rabbitmq like this:
$ sudo rabbitmqctl list_queues
Listing queues ...
c2e9b4beefc7468ea7c9005009a57e1d 1
1162a89dd72840b19fbe9151c63a4eaa 0
07638a97896744a190f8131c3ba063de 0
b34f8d6d7402408c92c77ff93cdd7cf8 1
f388839917ff4afa9338ef81c28aad75 0
8b898d0c7c7e4be4aa8007b38ccc00ea 1
3fb4be51aaaa4ac097af535301084b01 1
This seems to be inefficient, but further I have observed that these queues persist long after processing is finished.
I have found the task that appears to be doing this:
#celery.task(ignore_result=True)
def write_pages(page_generator):
g = group(render_page.s(page) for page in page_generator)
res = g.apply_async()
for rendered_page in res:
print rendered_page # TODO: print to file
It seems that because these tasks are being called in a group, they are being thrown into the queue but never being released. However, I am clearly consuming the results (as I can view them being printed when I iterate through res. So, I do not understand why those tasks are persisting in the queue.
Additionally, I am wondering if the large number queues that are being created is some indication that I am doing something wrong.
Thanks for any help with this!
Celery with the AMQP backend will store task tombstones (results) in an AMQP queue named with the task ID that produced the result. These queues will persist even after the results are drained.
A couple recommendations:
Apply ignore_result=True to every task you can. Don't depend on results from other tasks.
Switch to a different backend (perhaps Redis -- it's more efficient anyway): http://docs.celeryproject.org/en/latest/userguide/tasks.html
Use CELERY_TASK_RESULT_EXPIRES (or on 4.1 CELERY_RESULT_EXPIRES) to have a periodic cleanup task remove old data from rabbitmq.
http://docs.celeryproject.org/en/master/userguide/configuration.html#std:setting-result_expires
I would like to change the logging levels of a running Rails 3.2.x application without restarting the application. My intent is to use it to do short-time debugging and information gathering before reverting it to the usual logging level.
I also understand that the levels in ascending order are debug, info, warn, error, and fatal, and that production servers log info and higher, while development logs debug and higher.
I understand that if I run
Rails.logger.level=:debug #or :info, :warn, :error, :fatal
Will this change the logging level immediately?
If so, can I do this by writing a Rake task to adjust the logging level, or do I need to support this by adding a route? For example in config/routes.rb:
match "/set_logging_level/:level/:secret" => "logcontroller#setlevel"
and then setting the levels in the logcontroller. (:level is the logging level, and :secret which is shared between client and server, is something to prevent random users from tweaking the log levels)
Which is more appropriate, rake task or /set_logging_level?
Why don't you use operating system signals for that? For example on UNIX user1 and user2 signals are free to use for your application:
config/initializers/signals.rb:
trap('USR1') do
Rails.logger.level = Logger::DEBUG
end
trap('USR2') do
Rails.logger.level = Logger::WARN
end
Then just do this:
kill -SIGUSR1 pid
kill -SIGUSR2 pid
Just make sure you dont override signals of your server - each server leverages various signals for things like log rotation, child process killing and terminating and so on.
In Rails console, you can simply do:
Rails.logger.level = :debug
Now all executed code will run with this log level
As you have to change the level in the running rails instance, a simple rake task will not work.
I would go with the dedicated route.
instead of a shared secret I would use the app's standard user authentication (if your app has users) and restrict access to admin/super user.
In your controller LogController try this
def setlevel
begin
Rails.logger.level = Logger.const_get(params[:level].upcase)
rescue
logger.info("Logging level #{params[:level]} not supported")
end
end
You can also use gdb to attach to the running process, set the Rails.logger to debug level and then detach. I have created the following 1 liner to do this for my puma process:
gdb attach $(pidof puma) -ex 'call(rb_eval_string("Rails.logger.level = Logger::DEBUG"))' -ex detach -ex quit
NOTE: pidof will return multiple pids, in descending order. So if you have multiple processes with the same name this will only run on the first one returned by pidof. The others will be discarded by the "gdb attach" command with the message: "Excess command line arguments ignored. (26762)". However you can safely ignore it if you only care about the first process returned by pidof.
Using rufus-scheduler, I created this schedule:
scheduler.every 1.second do
file_path = "#{Rails.root}/tmp/change_log_level.#{Process.pid}"
if File.exists? file_path
log_level = File.open(file_path).read.strip
case log_level
when "INFO"
Rails.logger.level = Logger::INFO
Rails.logger.info "Changed log_level to INFO"
when "DEBUG"
Rails.logger.level = Logger::DEBUG
Rails.logger.info "Changed log_level to DEBUG"
end
File.delete file_path
end
end
Then, log level can be changed by creating a file under tmp/change_log_level.PID, where pid is the process id of the rails process. You can create a rake/capistrano task to detect and create these files, allowing you to quickly switch log level of your running production server.
Just remember to start rufus in the worker threads, if you are using unicorn or similar.
I'm using Resque on a rails-3 project to handle jobs that are scheduled to run every 5 minutes. I recently did something that snowballed the creation of these jobs and the stack has hit over 1000 jobs. I fixed the issue that caused that many jobs to be queued and now the problem I have is that the jobs created by the bug are still there and therefore It becomes difficult to test something since a job is added to a queue with 1000+ jobs.
I can't seem to stop these jobs. I have tried removing the queue from the redis-cli using the flushall command but it didn't work. Am I missing something? coz I can't seem to find a way of getting rid of these jobs.
Playing off of the above answers, if you need to clear all of your queues, you could use the following:
Resque.queues.each{|q| Resque.redis.del "queue:#{q}" }
If you pop open a rails console, you can run this code to clear out your queue(s):
queue_name = "my_queue"
Resque.redis.del "queue:#{queue_name}"
Resque already has a method for doing this - try Resque.remove_queue(queue_name) (see the documentation here). Internally it performs Resque.redis.del(), but it also does other cleanup, and by using an api method (rather than making assumptions about how resque works) you'll be more future-proof.
Updated rake task for clearing (according to latest redis commands changes): https://gist.github.com/1228863
This is what works now:
Resque.remove_queue("...")
Enter redis console:
redis-cli
List databases:
127.0.0.1:6379> KEYS *
1) "resque:schedules_changed"
2) "resque:workers"
3) "resque:queue:your_overloaded_queue"
"resque:queue:your_overloaded_queue" - db which you need.
Then run:
DEL resque:queue:your_overloaded_queue
Or if you want to delete specified jobs in queue then list few values from db with LRANGE command:
127.0.0.1:6379> LRANGE resque:queue:your_overloaded_queue 0 2
1) "{\"class\":\"AppClass\",\"args\":[]}"
2) "{\"class\":\"AppClass\",\"args\":[]}"
3) "{\"class\":\"AppClass\",\"args\":[]}"
Then copy/paste one value to LREM command:
127.0.0.1:6379> LREM resque:queue:your_overloaded_queue 5 "{\"class\":\"AppClass\",\"args\":[]}"
(integer) 5
Where 5 - number of elements to remove.
It's safer and bulletproof to use the Resque API rather than deleting everything on the Resque's Redis. Resque does some cleaning in the inside.
If you want to remove all queues and associated enqueued jobs:
Resque.queues.each {|queue| Resque.remove_queue(queue)}
The queues will be re-created the next time a job is enqueued.
Documentation
Let's say I have delayed_job running in the background. Tasks can be scheduled or run immediately(some are long tasks some are not)
If a task is too long, a user should be able to cancel it. Is it possible in delayed job? I checked the docs and can't seem to find a terminate method or something. They only provide a catch to cancel delayed job itself(thus cancelling all tasks...I need to just cancel a certain running task)
UPDATE
My boss(who's a great programmer btw) suggested to use Ruby Threading for this feature of ours. Is this possible? Like creating new threads per task and killing that thread while it's running?
something like:
t1 = Thread.new(task.run)
self.delay.t1.join (?) -- still reading on threads so correct me if im wrong
then to stop it i'll just use t1.stop (?) again don't know yet
Is this possible? Thanks!
It seems that my boss hit the spot so here's what we did(please tell us if there's some possibility this is bad practice so I can bring it up):
First, we have a Job model that has def execute! (which runs what it's supposed to do).
Next, we have delayed_job worker in the background, listening for new jobs. Now when you create a job, you can schedule it to run immediately or run every certain day (we use rufus for this one)
When a job is created, it checks if its supposed to run immediately. If it is, it adds itself to the delayed job queue. The execute function creates a Thread, so each job has its own thread.
User in the ui can see if a job is running(if there's a started_at and no finished_at). If it IS running, there's a button to cancel it. Canceling it just sets the job's canceled_at to Time.now.
While the job is running it also checks itself if it has a canceled_at or if Time.now is > finished_at. If so, kill the thread.
Voila! We've tested it for one job and it seems to work. Now the only problem is scaling...
If you see any problems with this please do so in the comments or give more suggestions if ever :) I hope this helps some one too!
Delayed::Job is an < ActiveRecord::Base model, so you can query it just like you normally would like Delayed::Job.all(:conditions => {:last_error => nil}).
Delayed::Job objects have a payload field which contain a serialized version of the method or job that you're attempting to run. This object is accessed by their '#payload_object' method, which loads the object in question.
You can combine these two capabilities to make queriable job workers, for instance, if you have a User model, and the user has a paperclip'ed :avatar, then you can make a method to delete unprocessed jobs like so:
class User < ActiveRecord::Base
has_attached_file :avatar, PaperclipOptions.new(:avatar)
before_create :'process_avatar_later'
def process_avatar_later
filename = Rails.root.join('tmp/avatars_for_processing/',self.id)
open(filename, 'w') do |file| file <<self.avatar.to_file end
Delayed::Job.enqueue(WorkAvatar.new(self.id, filename))
self.avatar = nil
end
def cancel_future_avatar_processing
WorkAvatar.future_jobs_for_user(self.id).each(&:destroy)
#ummm... tell them to reupload their avatar, I guess?
end
class WorkAvatar < Struct.new(:user_id, :path)
def user
#user ||= User.find(self.user_id)
end
def self.all_jobs
Delayed::Job.scoped(:conditions => 'payload like "%WorkAvatar%"')
end
def self.future_jobs_for_user(user_id)
all_jobs.scoped(:conditions => {:locked_at => nil}).select do |job|
job.payload_object.user_id == user_id
end
end
def perform
#user.avatar = File.open(path, 'rb')
#user.save()
end
end
end
It's possible someone has made a plugin make queryable objects like this. Perhaps searching on GitHub would be fruitful.
Note also that you'd have to work with any process monitoring tools you might have to cancel any running job worker processes that are being executed if you want to cancel a job that has locked_at and locked_by set.
You can wrap the task into a Timeout statement.
require 'timeout'
class TaskWithTimeout < Struct.new(:parameter)
def perform
Timeout.timeout(10) do
# ...
end
rescue Timeout::Error => e
# the task took longer than 10 seconds
end
end
No, there's no way to do this. If you're concerned about a runaway job you should definitely wrap it in a timeout as Simone suggests. However, it sounds like you're in search of something more but I'm unclear on your end goal.
There will never be a way for a user to have a "cancel" button since this would involve finding a method to directly communicate with the worker running process running the job. It would be possible to add a signal handler to the worker so that you could do something like kill -USR1 pid to have it abort the job it's currently working and move on. Would this accomplish you goal?