I have a service which tasks worker processes via RabbitMQ. The messages are sent with a TTL, and the worker will not ack the message until it successfully completes the task sent in the message.
The tasking process will monitor workers for timeouts, and if a worker exceeds the timeout it will be terminated. Since the message isn't ack'd, the message is re-queued immediately and the next worker will pick up the message (this is useful in my scenario, as workers are unreliable and may fail but subsequent attempts typically succeed.
However, I would also like the ability to cancel a message. Terminating and re-creating the worker process is the normal procedure (it's single threaded, so I can't send a separate 'cancel' message to the worker). However, doing so leads to the message immediately re-queueing if the TTL has not been exceeded.
The only suggested solution I've found is here, which suggested a separate data source which checks if a message is still valid. However, that answer is both a) old and b) inconvenient.
Does RabbitMQ offer a means to cancel a message once it's been placed into the queue?
Unfortunately rabbitmq does not have a way to cancel a message.
Without the ability to send a "cancel" message to your consumer, you may have to do something like what that other post suggests.
Another option to consider: message processing should be idempotent. That is, processing the same message more than once should only cause the desired result to occur once (the first time it is processed).
Idempotence is often achieved through the use of a correlationid in messaging. You can attach a correlationid to your message, then check a database or other service to see if that message should still be processed. If you want to "cancel" the message, you would update the other database/service with that specific correlationid to say "this one has been processed already" or "has been canceled" or something like that.
Related
I am doing a POC to work with RabbitMQ and have a questions about how to listen to queues conditionally!
We are consuming messaging from a queue and once consumed, the message will be involved in an upload process that takes longer times based on the file size. And as the file sizes are larger, sometimes the external service we invoke running out of memory if multiple messages are consumed and upload process is continuing for the previous messages.
That said, we would like to only consume the next message from the queue once the current/previous message is processed completely. I am new to JMS and wondering how to do it.
My current thought is, the code flow will manually pull the next message from the queue when it completes the process of previous message as the flow knows that it has completed the processing but if that listener is only used in code flow to manually call, how it will pull the very first message!
The JMS spec says that message consumers work sequentially:
The session used to create the message consumer serializes the
execution of all message listeners registered with the session
If you create a MessageListener and use that with your consumer, the JMS spec states the listener's onMessage will be called sequentially, i.e. once per message after each message has been processed by the listener. So in effect each message waits until the previous has completed.
I'm trying to use RabbitMQ in a more unconventional way (though at this point i can pick any other message queue implementation if needed)
I have one queue (I can have more if needed) that where customers are fetching N messages asynchronous. After they do their work I send the results from the client to the db.
I have two problems: first I don't want that they will work on the same message, second I want to grantee that I wont lose messages in case that my customer will close the browser or just stop working.
I looked at the documentation and saw the TTL which was perfect for me if I could alter that message that got timeout isn't going to be deleted but to move to another queue. can't find a way to alter this.
Moreover I looked at the confirmation option which in the first glance looked what I wanted,that mechanism is working like this: when the consumer gets a message he send confirmation to queue, I thought I can delay this confirm and send it when the work is done on the client side.
my problem was that I can't program the queue that if any message didn't get confirm then return it to the queue (or to another).
I also find how to do a scheduled message but it didn't help either because I don't want that the message will be inserted to the queue in five min,I want that when a customer will receive a message it will be locked in the queue for 5 min until confirm to delete is set otherwise return it to the queue.
Can I do temporary queue that enables my mechanism?
If someone can help with one of the problems or suggest another architecture or option to do it in another MQ it would be great.
Resources:
confirmation:
http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
post about locks but his problem was a batcher component:
Locks and batch fetch messages with RabbitMq
TTL:
https://www.rabbitmq.com/ttl.html
Schedule a message:
https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/
my problem was that I can't program the queue that if any message
didnt get confirm then return it to the queue (or to another).
RabbitMQ does this anyhow, so all you have to do is switch off the auto-ack flag, you figured this out
I thought I can delay this confirm and send it when the work is done
on the client side.
so just send the ACK once you've finished with processing the message.
All the unacknowledged messages remain in the queue and are re-delivered to next consumer (or the same one when it's up again, depending on your setup)
I have this issue, I want to know my rabbit is working great.
I am not gonna send the message, so, Im not 100% sure is being sent correctly. But the problem is this.
After all is configured and all....
I see at the RabbitMQ web manager
And when I supposedly send a message the I see activity on the "message rates" chart but nothing at the "queued messages" .
I frankly dont know whats going on, is it too fast that doesnt need to queue the messages? Or something is misconfigured?
Any idea of the difference?
Thanks.
In case RabbitMQ receive non-routable message it drop it. So while message was received, it was not queued.
You may configure Alternate Exchanges to catch such messages.
In my case,
Situation1:
when my Exchange in rabbitTemplate.convertAndSend was not set properly -- the message was not sent to the correct queue -- the Queued messages was empty all time.
however, Message rates is not zero, it does show there are message get sent.
Which correspond to what the other answer is saying:
In case RabbitMQ receive non-routable message it drop it.
Situation2:
when my Exchange in rabbitTemplate.convertAndSend was indeed set properly -- the message was sent to the correct queue -- the Queued messages was queuing up the message.
Everything seems fine.
Situation3:
(continue from Situation2)
And now, I turn on the receiver service which has the #RabbitListener.
The Queued messages immediately drops down to 0, and never goes up again.
But the transporting of messages is still working fine.
Situation4:
(continue from Situation2)
And now, I change the receiver service to use the rabbitTemplate.receiveAndConvert.
Which I manually receive the message from the queue every 2s by using a loop.
(message is also sent from sender service every 2s by using a loop, same as the situations before.)
Now, the Queued messages stays at constant -- a straight line
(depends on how many message you have queued up, in my case 1, before the receiver service is up, then it stays at 1).
Conclusion:
I suspect that, when the message is consumed too fast, the Queued messages will just show 0.
Which correspond to what the OP is saying:
is it too fast that doesnt need to queue the messages?
(or, I could screw up some setting in RabbitMQ and led to wrong conclusion. I dont think so, but idk, I am not familiar with RabbitMQ.)
I am using the alerts feature of IronMQ service provided by IronIO to start workers.
I have things setup so that a message is pushed onto the push queue. The push queue sends an alert that starts a worker. The worker pulls off the message on the push queue, reserving it. Sometimes for whatever reason the job fails, the reservation for a message expires, and the message becomes available again. However, from what I can tell, no alert is sent when the reservation expires on a message. So the message sits in the queue until another message is added to the queue firing an alert and starting a worker. But the new message is not processed.
Are alerts created for messages that have a reservation expire in IronMQ? Is there any documentaion that I missed describing what can happen?
I am working on having workers pull off multiple messages but I am running into issues unrelated to iron io when processing multiple messages in the same worker.
Also is there a way to pull off the top of the queue. To avoid pulling off messages that may be causing errors? Should I just modify my workers to delete messages causing errors?
Currently there are no alerts for when a message times out and goes back on the queue, but that does seem like it would be a good idea. I assume this is a pretty inactive queue? I made a feature request for this here: https://trello.com/c/XcHi0NdN/35-fire-alert-when-a-message-times-out-goes-back-on-queue
And regarding messages that are causing issues, your best bet would be to add them to a different queue (an error queue) and delete them off the original queue. Then you can go through the error queue to figure out why certain messages are causing you problems. This is known as a "dead letter queue" btw and we have a feature request for it here, please give it a vote! https://trello.com/c/bGnJcNa9/26-dead-letter-queue
We are having MVC application which reads data from MSMQ.
We are trying to find out a way to read message from queue and remove it from queue only if user has done a successful operation on the queue.
The message should remain in the queue until user completes the processing, the message should not be available to anyone else until the user who is processing the message object has finished the operation.
Is there a property for a Message object to be set as Peeked which will not allow reading of this message again until ether it is put back into the queue or removed from the queue?
We are not sure if using MSMQ is a good idea in this case?
It sounds like you need to use your queue(s) in transactional mode. Then, your client can receive a message, process it, and then commit the transaction, at which point the message will be finally dequeued. While the transaction is active, however, other clients will not see the message -- it will be held in reserve until the transaction completes or is aborted.
This MSDN article has a decent overview of usage patterns for reliable messaging with MSMQ:
http://msdn.microsoft.com/en-us/library/ms978430.aspx
The Queue is the right idea. Your approach of "leave it in the queue, locked, but still kind-of-available" is wrong.
You may need multiple queues.
Process A enqueues something in Queue 1
Process B dequeues from Queue 1 and starts work.
If B is successful, that's it.
Otherwise, it gets queued somewhere else (perhaps the same queue, or perhaps Queue 2) for follow-up work.
If it went back into Queue 1, B will find it again, eventually. If it went to another Queue, then another process does cleanup, logging, error fixup or whatever, possibly putting something back in Queue 1.
A Queue isn't a database -- there's nothing stateful (no "don't look at me, I'm being processed").
A Queue is transient storage. Someone writes, someone else reads, and that's it.
If you want reliability, read this: http://msdn.microsoft.com/en-us/library/ms978430.aspx
And this: http://blogs.msdn.com/shycohen/archive/2006/02/20/535717.aspx
And this: http://www.request-response.com/blog/PermaLink,guid,03fb0e40-b446-42b5-ad90-3be9b0260cb5.aspx
Reliability is a feature of the queue, not your application. You can do a "recoverable read". It's a transaction that's part of the queue API.