I'm taking message from queue, processing it and for some messages at some point I understand that I have to wait for several seconds before trying to process it.
Options are:
I can call sleep() and wait, but sometimes there could hundrends of messages, and luanching that many workers is not an option.
I can simply requeue message, but is it a good idea? I will be processing it again and again with the same result. (Processing includes DB queuring, etc)
I can send myself a new message, including timestamp to know if it's time to process a message or not. So on arriaval I'm just going to check timestamp and if it's good procceed to usual processing.
Obviously, third way it better, but should I send this messages to the same exchange/queues, or is it a good idea to create different exchange "processing_pass2"? Or may be there are some built-in functionality #rabbitMQ to perform such a task? Requeue message to appear again after given number of seconds?
Related
I'm needing to execute a process in the future, let's say 20min, based on some event happening, but I may need to cancel that scheduled process depending on different factors. Or , i may need to restart the timer on the job, depending on another event....etc. You get the idea. All different permutations of this. Does anyone know of a good technology for this need? Maybe quartz(does quartz suck? does it do all these things?), maybe activemq, maybe some other job scheduling technology?
Thanks!
-Ron
ActiveMQ's scheduler is a good fit for this. The pattern can go something like:
Kick off a process (get some identifier)
Send a message to the ActiveMQ scheduler to fire in x time period
Message Consumer receives the timer message, pulls the identifier to check on the status
If process is done.. continue and finish up
If process needs more wait time, send another timer message to ActiveMQ
Everything is asynchronous, and code required is very minimal. The big advantage of using ActiveMQ is you can have multiple consumers listening for the scheduled message to provide for high availability.
I need to know if it's possible to know or get a message from RabbitMQ (consumers) which is processing (maybe if it's taking long time). But, I don't want to stop the service. I hope my question is clear enough.
I guess you mean if RabbitMQ provides any build-in notification or tracking about whether a specific message is being processed or not. The answer is No. But you should easily be able to implement one by your own. For instance, when begin to process a message in the consumer, the consumer could send a notification message to a notification queue.
I have part of my application that receives string messages from remote clients and decodes these into a _Message class. I then want to pass these messages into a queue for immediate processing. The FIFO method is exactly what I require as I would particularly prefer to process these messages in order of receiving.
These messages come in fairly constant (24 hours a day, maybe 1 every couple of seconds or so...) so I need to ensure that I capture them all and no messages get lost or rejected.
Each _Message will then run through a Routine which will decode and action various parts of the message content.
Therefore, what would be the best way of handling a constant message pool? I started to go down the path of Queues with Queue.Enqueue and Queue.Dequeue but I'm not sure how to constantly poll for items within the Queue without affecting performance or resources.
I then came across ThreadPooling (something very new to me) which sounds like it could be down the right path, but I'm not 100% sure on how it works or how to set it up correctly.
Or....can I use ThreadPooling in conjunction with a Queue? And simply add items into my Queue and have the ThreadPool automatically detect new items?
Any help or guidance would be appreciated. Thanks
I was wondering if this is possible. I want to pull a task from a queue and have some work that could potentially take anywhere from 3 seconds or longer (possibly) minutes before an ack is sent back to RabbitMQ notifying that the work has been completed. The work is done by a user, hence this is why the time it takes to process the job varies.
I don't want to ack the message immediately after I pop off the queue because I want the message to be requeued if no ack is received. Can anyone give me any insights into how to solve my problem?
Having a long timeout should be fine, and certainly as you say you want redelivery if something goes wrong, so you want to only ack after you finish.
The best way to achieve that, IMO, would be to have multiple consumers on the queue (i.e. multiple threads/processes consuming from the same queue). That should be fine as long as there's no particular ordering constraint on your queue contents (i.e. the way there might be if the queue were to contain contents representing Postgres data that involves FK constraints).
This tutorial on the RabbitMQ website provides more info (Python linked, but there should be similar tutorials for other languages): https://www.rabbitmq.com/tutorials/tutorial-two-python.html
Edit in response to comment from OP:
What's your heartbeat set to? If your worker doesn't acknowledge the heartbeat within the set period of time, the server will consider the connection to be dead.
Not sure which language you're using, but for Java you would use the setRequestedHeartbeat method to specify the heartbeat.
The way you implement your workers, it's vital that the heartbeat can still be sent back to the RabbitMQ server. If something blocks the client from sending the heartbeat, the server will kill the connection after the time interval expires.
We need to be able to specify a delay in retrying failed messages. NServiceBus retries more or less instantly up to n times (as configured) before moving the message to error queue.
What I need to be able to do is for a given message type specify that its not to be retried for an arbitrary period of time
I've read the post here:
NServiceBus Retry Delay
but this doesn't give what I'm looking for.
Kind regards
Ben
This isn't supported as of right now. What you can do is let the messages go to the error queue and setup and endpoint to monitor that queue. Your code could then determine the rules for replaying messages. You could use a Saga to achieve this in combination with the Timeout manager.
Typically you'll have some rules around when to replay messages. In NSB 3.0 we have a better way to do this using the FaultManager. This gives you options on where to put failed messages and includes the exception. One of the options is a DB which you could then set up a job to inspect the exception and determine what to do with it.
Lastly a low tech way of getting this is to schedule a job that runs the ReturnToSourceQueue tool periodically to "clean up". We are doing this and including an alert so we don't endlessly cycle messages around.