We are having MVC application which reads data from MSMQ.
We are trying to find out a way to read message from queue and remove it from queue only if user has done a successful operation on the queue.
The message should remain in the queue until user completes the processing, the message should not be available to anyone else until the user who is processing the message object has finished the operation.
Is there a property for a Message object to be set as Peeked which will not allow reading of this message again until ether it is put back into the queue or removed from the queue?
We are not sure if using MSMQ is a good idea in this case?
It sounds like you need to use your queue(s) in transactional mode. Then, your client can receive a message, process it, and then commit the transaction, at which point the message will be finally dequeued. While the transaction is active, however, other clients will not see the message -- it will be held in reserve until the transaction completes or is aborted.
This MSDN article has a decent overview of usage patterns for reliable messaging with MSMQ:
http://msdn.microsoft.com/en-us/library/ms978430.aspx
The Queue is the right idea. Your approach of "leave it in the queue, locked, but still kind-of-available" is wrong.
You may need multiple queues.
Process A enqueues something in Queue 1
Process B dequeues from Queue 1 and starts work.
If B is successful, that's it.
Otherwise, it gets queued somewhere else (perhaps the same queue, or perhaps Queue 2) for follow-up work.
If it went back into Queue 1, B will find it again, eventually. If it went to another Queue, then another process does cleanup, logging, error fixup or whatever, possibly putting something back in Queue 1.
A Queue isn't a database -- there's nothing stateful (no "don't look at me, I'm being processed").
A Queue is transient storage. Someone writes, someone else reads, and that's it.
If you want reliability, read this: http://msdn.microsoft.com/en-us/library/ms978430.aspx
And this: http://blogs.msdn.com/shycohen/archive/2006/02/20/535717.aspx
And this: http://www.request-response.com/blog/PermaLink,guid,03fb0e40-b446-42b5-ad90-3be9b0260cb5.aspx
Reliability is a feature of the queue, not your application. You can do a "recoverable read". It's a transaction that's part of the queue API.
Related
We are working in a microservice architecture and we are using RabbitMQ as a message broker. We want to avoid the scenarios where the following happens:
An entity begins its creation but it takes a while for it to finish.
The system decides that the creation time has taken too long and that the entity should be deleted due to a timeout, so it sends out a message to delete the entity which is currently still being created
Delete message gets consumed and the system checks whether the entity exists and does not find it due to the entity still being in the process of being created.
Delete entity message consumer returns an error due to not finding the entity.
How can we ensure that the delete message is consumed after the create message is finished in such a way that we do not block the consumption of other messages?
How can we ensure that the delete message is consumed after the create message is finished in such a way that we do not block the consumption of other messages?
Let's say your entity creation timeout is N. The worker(s) responsible for creating entities should know about this timeout, and should be able to cancel entity creation should N be reached. This isn't strictly necessary but it sounds like your entity creation may be resource intensive so cancellation should be a feature you have.
If your workers know to cancel entity creation when timeout N is reached, then perhaps you don't even need the deletion message?
If you keep the delete message, the workers processing that could do the following:
First, ensure your queue has a Dead Letter Exchange configured
Consume the message, and try to delete the entity
If deletion succeeds, great, ack the message with RabbitMQ and you're done
If deletion fails, nack (reject) the message with RabbitMQ and do set requeue to be false. This will cause the message to be routed to the dead-letter exchange
A worker should consume from a queue bound to this dead-letter exchange. You could have a queue dedicated to re-trying entity deletions. When a worker consumes a message from this queue, it can re-try the deletion. If it fails, you can reject it again (after a delay, of course) and, if this queue has the same dead-letter settings, the same process will happen
Finally, ensure that your deletion workers respect the count property and only try a certain number of times to delete an entity. If a limit is exceeded, this should create an exception in your system
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I have the following use case that I'm trying to setup in rabbit MQ:
Normally process A should handle all messages sent to queue A.
However if process A goes down (is no longer consuming from queue A) Then process B should handle the messages until process A comes back up.
At first it looks like consumer priorities might be the solution. https://www.rabbitmq.com/consumer-priority.html. However that will send messages to process B when process A is just blocked working on other messages. I only want them sent to process B when process A is down.
A 2nd option might be dead lettering. https://www.rabbitmq.com/dlx.html. If process A is not reading from queue A the messages will eventually time out and then move to an exchange that forwards them to a queue that process B reads. However that options requires waiting for the message to timeout which is not ideal. Also the message could timeout even while process A is still working which is not ideal.
Any ideas how rabbit MQ could be configured for the use case described above? Thanks
According to your answers to my questions, I would probably use a priority on consumer so that process A handles a maximum of messages, along with a high prefetch count (if possible, and you must ensure your process can handle such a high number).
Then, process B would handle the messages that process A cannot handle due to the high load, or all the messages when process A is not available. It is probably acceptable that in the case of high load some messages are handled with a higher delay. Do not forget to set a low prefetch count for process B.
Hope this helps.
I'm trying to use RabbitMQ in a more unconventional way (though at this point i can pick any other message queue implementation if needed)
I have one queue (I can have more if needed) that where customers are fetching N messages asynchronous. After they do their work I send the results from the client to the db.
I have two problems: first I don't want that they will work on the same message, second I want to grantee that I wont lose messages in case that my customer will close the browser or just stop working.
I looked at the documentation and saw the TTL which was perfect for me if I could alter that message that got timeout isn't going to be deleted but to move to another queue. can't find a way to alter this.
Moreover I looked at the confirmation option which in the first glance looked what I wanted,that mechanism is working like this: when the consumer gets a message he send confirmation to queue, I thought I can delay this confirm and send it when the work is done on the client side.
my problem was that I can't program the queue that if any message didn't get confirm then return it to the queue (or to another).
I also find how to do a scheduled message but it didn't help either because I don't want that the message will be inserted to the queue in five min,I want that when a customer will receive a message it will be locked in the queue for 5 min until confirm to delete is set otherwise return it to the queue.
Can I do temporary queue that enables my mechanism?
If someone can help with one of the problems or suggest another architecture or option to do it in another MQ it would be great.
Resources:
confirmation:
http://www.rabbitmq.com/blog/2011/02/10/introducing-publisher-confirms/
post about locks but his problem was a batcher component:
Locks and batch fetch messages with RabbitMq
TTL:
https://www.rabbitmq.com/ttl.html
Schedule a message:
https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages-with-rabbitmq/
my problem was that I can't program the queue that if any message
didnt get confirm then return it to the queue (or to another).
RabbitMQ does this anyhow, so all you have to do is switch off the auto-ack flag, you figured this out
I thought I can delay this confirm and send it when the work is done
on the client side.
so just send the ACK once you've finished with processing the message.
All the unacknowledged messages remain in the queue and are re-delivered to next consumer (or the same one when it's up again, depending on your setup)
I have a service which tasks worker processes via RabbitMQ. The messages are sent with a TTL, and the worker will not ack the message until it successfully completes the task sent in the message.
The tasking process will monitor workers for timeouts, and if a worker exceeds the timeout it will be terminated. Since the message isn't ack'd, the message is re-queued immediately and the next worker will pick up the message (this is useful in my scenario, as workers are unreliable and may fail but subsequent attempts typically succeed.
However, I would also like the ability to cancel a message. Terminating and re-creating the worker process is the normal procedure (it's single threaded, so I can't send a separate 'cancel' message to the worker). However, doing so leads to the message immediately re-queueing if the TTL has not been exceeded.
The only suggested solution I've found is here, which suggested a separate data source which checks if a message is still valid. However, that answer is both a) old and b) inconvenient.
Does RabbitMQ offer a means to cancel a message once it's been placed into the queue?
Unfortunately rabbitmq does not have a way to cancel a message.
Without the ability to send a "cancel" message to your consumer, you may have to do something like what that other post suggests.
Another option to consider: message processing should be idempotent. That is, processing the same message more than once should only cause the desired result to occur once (the first time it is processed).
Idempotence is often achieved through the use of a correlationid in messaging. You can attach a correlationid to your message, then check a database or other service to see if that message should still be processed. If you want to "cancel" the message, you would update the other database/service with that specific correlationid to say "this one has been processed already" or "has been canceled" or something like that.
I have thousands of unacked messages in my dev environment which I can't restart.
Is there a way to remove (purge) all messages even if they are unacknowledged?
Close the channel that the unacked messages reside on, which will nack them back into the queue, then call purge.
You have to make consumer ack them (or nack) and only after that they will be removed. Alternatively you can shutdown consumers and purge the queue completely.
If you are looking for some way to purge all unacked messages - there are no such feature nor in AMQP protocol neither in RabbitMQ.
It looks like your consumer is the cause of the problem, so you have to adjust it (rewrite) to release message immediately after it processed or failed.
Once there are no "ready" messages in the queue, delete it and recreate.
YOU WILL LOSE THE QUEUE CONTENTS with this method.
You need to put messages back into the queue before you can purge them:
close the channel
close the connection (the script doesn't work for me)
As an alternative, this doesn't require to wait:
delete and recreate the queue
restart the server
You need to call basic.recover to force all unacked messages to be re-enqueued to a channel that failed. Be aware of the errata concerning this function specifying that only the requeue mode is supported by RabbitMQ.
For software developer use below code.
channel.purgeQueue(queue-name);
if we use this code the Queue will be clear and same queue will exist.
One way this can happen is if the consumer is stuck recycling the same messages due to a processing error. In this case, the RabbitMQ queue management interface may show the messages as Unacked, but really they are being read from the queue and processed (to the point of the failure) then requeued (to enable a retry) at a rapid pace -- maybe thousands of times per second.
During this loop, the messages exist briefly in the Ready state, but are immediately removed again by you application -- and the cycle begins again. As an example, this auto-requeue behavior is the default for Spring AMQP.
Since the messages are never left in the Ready state, the Management Interface's Get Message(s) button is unlikely work. What can work, if you have queue access, is to run a separate custom consumer instance, perhaps locally, but with the specific intent of removing and not requeuing the messages in question.
By RabbitMQ's Fair Dispatch mechanism, your additional consumer will likely receive the messages in question and have the opportunity to perform your custom handling.
You might even write a custom utility to do this, with logic to filter, analyze, or deadletter the messages of interest.
If you want to clear the contents of the queue, then you can use the AMQP method queue.purge: There is queue purge in AMQP: http://www.rabbitmq.com/amqp-0-9-1-reference.html#queue.purge
You could do similar using the management plugin.