Im using resque to have some jobs running in the background, a client ( user ) initiates these by doing an action on the web app in there browser.
The problem is it takes several seconds for the action to be triggered. How could one speed it up ? I need resque to respond more instant.
IM using all default setup and config nothing modified.
Are there any guidelines for configuration or suggestions out of the field to make resque response faster?
Im running with 1 worker and low queues like 1,2 at a time.
Resque workers check the queue every 5 seconds by default, taken from the Resque page on Github:
start
loop do
if job = reserve
job.process
else
sleep 5 # Polling frequency = 5
end
end
shutdown
Under "Polling frequency" it then says:
You can pass an INTERVAL option which is a float representing the polling frequency.
The default is 5 seconds, but for a semi-active app you may want to use a smaller value.
$ INTERVAL=0.1 QUEUE=file_serve rake environment resque:work
Also you could have a look at something like beanstalkd instead, you can watch this railscast about it.
Related
I am struggling to find proper setting to delay timeout for workers in RabbitMQ.
By default prefetchCount since the version 2.0 are set to 250 and exactly this amount of messages are being received and processed.
I would like to keep workers busy, until they clear up an entire queue (lets say 10k messages).
I can manipulate this number manually, such as changing default limit or assigning more threads resulting in multiplying default number.
Results are always the same. Once the number is reached, workers stop their job and application finish its execution
o.s.a.r.l.SimpleMessageListenerContainer : Successfully waited for workers to finish.
I would like them to finish when the queue is empty. Any ideas?
The logger.info("Successfully waited for workers to finish."); happens only in one place - doShutdown(). And this one is called from the shutdown(), which is called from the destroy() or stop().
I somehow think that you exit from your application by some reason. You just don't block the main() to work permanently.
Please, share a simple project we can play with.
I would like to launch the worker after an action and this worker must be launch automatically after every 5 seconds to check a status and when the status is checked, it must be stops. How can I do that? Thanks for your help
I am building an app that is using Twilio and we need to call Twilio's server a certain period after the call starts.
I'm using RoR and right now its on Heroku, but we can move it elsewhere if need be.
I've looked at Delayed Jobs and Cron jobs but I don't know enough about it to know which route I should take.
Seems like Delayed jobs and cron jobs are usually reoccurring (not very accurate with timing) and are started with rake not by a user action, right?
What tool will give me minute accuracy for this use case?
Twilio Evangelist here.
I like to approach this with Redis backed worker queues. Redis is a key-value store, which we can use with Resque (pronounced 'rescue'), and Resque-Scheduler to provide a queueing mechanism. For example, we can have the application respond to user interaction by creating a call using the Twilio REST API. Then we can queue a worker task that will do something with the call a specified amount of time later. Here, I'm just going to get the status of the call after a few seconds (instead of waiting for the status callback when the call completes).
Heroku have a really good walkthrough on using Resque, there is also an excellent Rails Casts episode on Resque with Rails. A data add-on is available from Heroku called Reddis Cloud to provide a Redis server. Although for development I run a local server on my computer.
I've created a simple app based on the Heroku tutorial and the Rails Casts episode. It has a single controller and a single model. I'm using a callback on the model to create an outbound Twilio call. Such that when a new Call record is created, we initiate the outbound Twilio call:
class Call < ActiveRecord::Base
# Twilio functionality is in a concern...
include TwilioDialable
after_create do |call|
call.call_sid = initiate_outbound call.to,
call.from,
"http://example.com/controller/action"
call.save
call
end
end
I'm using a concern to initiate the call, as I may want to do that from many different places in my app:
module TwilioDialable
extend ActiveSupport::Concern
include Twilio::REST
def initiate_outbound to, from, url
client = Twilio::REST::Client.new ENV['TW_SID'], ENV['TW_TOKEN']
client.account.calls.create(to: to, from: from, url: url).sid
end
end
This simply creates the outbound Twilio call, and returns the call SID so I can update my model.
Now that the call has been placed, so I want to check on its status. But as we want the call to ring a little first, we'll check it in 15 seconds. In my controller I use the Resque.enqueue_in method from Resque-Scheduler to manage the timing for me:
Resque.enqueue_in(15.seconds, MessageWorker, :call => #call.id)
This is instructing Resque to wait 15 seconds before performing the task. It is expecting to call a Resque worker class called MessageWorker, and will pass a hash with the parameters: {:call => #call.id}. To make sure this happens on time, I set the RESQUE_SCHEDULER_INTERVAL to 0.1 seconds, so it will be very precise. The MessageWorker itself is fairly simple, it finds the call record and uses the Call SID to get the updated status and save this to the database:
class MessageWorker
#queue = :message_queue
def self.perform params
# Get the call ID to do some work with...
call = Call.find(params["call"])
# Setup a Twilio Client Helper with credentials form the Environment...
client = Twilio::REST::Client.new ENV['TW_SID'], ENV['TW_TOKEN']
# We could get some
status = client.account.calls.get(call.call_sid).status
# Update my call object.
call.status = status
end
end
When I ran this locally, I had to start the Redis server, the Redis Worker rake task, the Redis Schedule rake task, and I am using Foreman and Unicorn to run the Rails app. You can configure all of this into your Procfile for running on Heroku. The tutorial on Heroku is a great walkthrough for setting this up.
But now I can create a call and have it dial out to a phone. In the mean time I can refresh the calls/show/:id page and see the value magically updated when the worker process is run 15 seconds later.
I also find that resque-web is really useful for debugging Resque, as it makes it really easy to see what is happening with your tasks.
Hope this helps!
I am using Heroku to host a small app. It's running a screenscraper using Mechanize and Nokogiri for every search request, and this takes about 3 seconds to complete.
Does the screenscraper block anyone else who wants to access the app at that moment? In other words, is it in holding mode for only the current user or for everyone?
If you have only one heroku dyno, then yes it is the case that other users would have to wait in line.
Background jobs are for cases like yours where there is some heavy processing to be done. The process running rails doesn't do the hard work up-front, instead it triggers a background job to do it and responds quickly, freeing itself up to respond to other requests.
The data processed by the background job is viewed later - perhaps in a few requests time, or whenever the job is done, loaded in by javascript.
Definitely, because a dyno is single threaded if it's busy scrapping then other requests will be queued until the dyno is free or ultimately timed out if they hang for more than 30 seconds.
Anything that relies on an external service would be best run through a worker via DJ - even sending an email, so your controller puts the message into the queue and returns the user to a screen and then the mail is picked up by DJ and delivered so the user doesn't have to wait for the mailing process to complete.
Install the NewRelic gem to see what your queue length is doing
John.
If you're interested in a full service worker system we're looking for beta testers of our Heroku appstore app SimpleWorker.com. It's integrated tightly into Heroku and offers a number of advantages over DJ.
It's as simple as a few line of code to send your work off to our elastic worker cloud and you can take advantage of massive parallel processing because we do not limit the number of workers.
Shoot me a message if you're interested.
Chad
We have a memory intensive processing for certain functionality and we would like to limit the number of parallel requests to this processing. We are able to configure by using "Work Managers" in WebLogic and putting a limit on the number of threads for that servlet.
For example, if we put maximim thread limit as 3, then if there are 10 parallel requests; 7 requests are in queue. There could be situations where these the requests waiting in queue could take up to 30-40 minutes to be processed. We did simple testing and the received page cannot be displayed due to timeout after 15 mins and received the message after 1 hour.
Does any one know if there is a setting in WebLogic to increase/decrease timeout and avoid page cannot be displayed?
Appreciate if any one has any thoughts around this.
Does any one know if there is a setting in WebLogic to increase/decrease timeout and avoid page cannot be displayed?
There might be something but I actually didn't check as it would be a bad advice anyway. By looking for this, you are trying to solve the wrong problem here. A browser is just not made for long-running process like the one you are describing (>30mn) even if you don't mind the user waiting (not mentioning that he could refresh the page and queue more and more jobs).
So, the right answer here is in my opinion: use asynchronism, this is the perfect use case. When the user clicks on the button, send a JMS message to a queue (or create a Quartz job) and send the user a page with a request ID telling him to come back later. When the processing is done, update the status somewhere and make the status/result available to the user. Really, the user experience will be better doing this and you'll face less problems than with a browser.
1) Use some other tool (not browser) like WGET where you can control timeout parameter (--timeout).
2) Why do you use HTTP? Use message driven beans and send message JMS to that and don't care about time outs.
Perhaps quartz can do what you need? Start a job and check in on it as you need to?