I'm using ActiveMQ and I would like to know how to solve this specific case.
When the consumer is down, the producer sends a message to the queue. The message will remain in the queue until the consumer is running to consume it.
Now imagine I shutdown the producer, the message will STILL remain in the queue. Now i run the consumer and it will try to consume that message, but won't be able to reply back to the producer since its down.
I would like to solve this problem by cleaning the messages if the producer is out.
The ActiveMQ Broker cleans the Queue after stopping. I would like to do the same for the messages of a respective producer.
Thanks.
Based on what I understand now from your question and additional comments I propose to add a Message Property to your messages to identify the Producer, and write a small utility that uses a Message Selector to read all messages matching the Producer from the queue. You can run that utility straight after the Producer is stopped (or crashes), and that should quite accurately do what you want to achieve.
EDIT: although primarily focused on EE, the Sun/Oracle JavaEE Tutorial contains a very good chapter on general JMS programming that starts off with standalone producers and consumers. The accompanying source code bundle can be downloaded here, the ready to comoile samples in that bundle should get you started very quickly.
You can solve it a couple of ways. One is to set a TTL on the message so it goes away. The other is to connect via JMX and purge the Queue or remove the specific message using a selector statement or with the Message's specific MessageId value.
See this article for some hints.
Related
I have a RabbitMQ broker and a Camel Java application that consumes the messages. The consumer process one message in ca. 1 second. I produce 10'000 messages; this is almost instanteneous. Immediately the RabbitMQ console reports that the queue contains no more that ca. 7'000 messages. This is an issue for me, because if the consumer fails, 3'000 messages or so are lost.
I tried several options:
stream caching: camelContext.setStreamCaching(false); // or true
throttling: from(queue).throttle(1)
prefetch: from(rabbitmq:exchange&queue=q&prefetchEnabled=true&prefetchCount=1&prefetchGlobal=false&prefetchSize=0
all to no avail. I never observe a regular decrese (10'000, then 9'999, 9'998, etc) in the number of messages in the queue. It is quite the opposite: I can always see that messages get out of the queue in large chunks (typically 3'000 messages), although the consumer actually processes them slowly.
How can I consume the messages one by one?
I know this might be a bit too late for you but I have also been battling the same issue and I finally found a solution to make the Camel RabbitMq component consume messages one by one.
You just set the autoAck property to false.
Below I have a sample of the whole route definition:
rabbitmq:your-exchange?routingKey=rtkey&queue=qname&autoDelete=false&durable=true&guaranteedDeliveries=true&threadPoolSize=1&autoAck=false
Then the important part is just:
guaranteedDeliveries=true&threadPoolSize=1&autoAck=false
The threadPoolSize and guaranteedDeliveries properties are not necessary to achieve what you want, I am just putting them here for completeness' sake.
Here is a link to the documentation
Run this route in a transaction. That way a failed consumer will end up with rollback of the messages onto the queue.
I have got several messages on error queue which has name TestQueue_errors.
One of the messages on error queue is important and should be moved back to service queue TestQueue so it can be processed again. The other messages on error queue are broken and should stay on error queue.
I have tried to do that with shovel plugin but it seems it is able only to move all messages from one queue to another. Is there a way I could achieve that, to move single message from one queue to another?
As far as I know Rabbit Management does not allow to do it. The only thing you can do is to publish this message again.
Maybe there are some tools which give possibility to achieve it but it is not a standard behaviour.
Here are actions which you are able to perform on the queue (from RabbitMQ Management page):
Move all messages from one queue to another
Get all messages without requeue option (they would not be in the queue anymore)
Get first N messages without requeue option and then move the rest of messages to another queue
I have the following problem.
My program sends messages directly to the Queue (without exchange). I need to monitor incoming of new messages and send them to other Queue without removing them from source queue.
I don't have access to program code, so I'm not able to publish messages to exchange first.
Is it possible to solve this problem using the management web interface of RabbitMQ?
I tried to use shovel plugin, but it removes all messages from source queue after ack.
First to clear up few things:
My program sends messages directly to the Queue (without exchange) This is not true, at the very least (and most likely in this case) nameless exchange is used.
removes all messages from source queue after ack
this is by design and therefore perfectly fine.
You should never keep messages in the queue, queue is made to be consumed. As Derick Bailey says here
RabbitMQ is not a database. RabbitMQ is a message broker and queueing system.
on the same link you will find your answer. I cannot give a concrete one since you didn't provide motivation, but whatever it is keeping messages in the queue is never good!
Maybe you want to log/store your message first and then process it with the consequence of processing being some 3rd action or whatever...
I'm going through a few examples using NServiceBus and I've stumbled across a feature I'm hoping ships with MassTransit (As it is a free service).
The feature is based around 'poisoned' messages.
If, due to a bug in your system, these messages cant ever be handled, and end up permanently in the error queue.
NServiceBus has a cool feature whereby, once you have corrected the bugs in your code, allows those messages in the error queue to be 'redirected' to the original working queue, to be redelivered.
This is done by using a NServiceBus specific tool :- ReturnToSourceQueue.exe.
Does MassTransit have a similar tool for this kind of issue?
Or is there another workaround availble, preferbly to work with RabbitMQ.
With RabbitMQ, it's easy to move messages between queues. You can use the management console to do it manually, by installing the shovel plug-in.
You can also create shovels in RabbitMQ that are scheduled, and perform the message movement in response to that schedule. The visibility of having the shovels configured in RabbitMQ has been invaluable to our operations staff, since they rarely think that a Windows Scheduled Task (or other random scheduler) is going to be doing something as risky as moving previously failed messages back into the production queues.
I would suggest reading this blog post on how MassTransit deals with poison messages: Error Handling in MassTransit with RabbitMQ
The tooling around RabbitMQ is so much better than anything MSMQ provides, which is one of the reasons we have completely abandoned MSMQ for production queuing.
This functionality is easily recreated with nothing more than RabbitMQ and a bit of code. While it's nice that NServicebus includes it, building it with MassTransit should be easy enough.
(note: i haven't used .NET in a few years, so my knowledge of NSB and MT are a bit rusty... this will be high level answer only, no code)
The thing to start with, is a proper configuration of a dead letter exchange and a poison message queue. https://www.rabbitmq.com/dlx.html
Once you have knowledge that a message is causing errors and is a bad message, you can reject or nack (with no requeue) the message in order to send it through the dead letter exchange (DLX).
Once a message has gone through the DLX, you will have some additional properties on the message, including:
queue - the name of the queue the message was in before it was dead-lettered,
exchange - the exchange the message was published to (note that this will be a dead letter exchange if the message is dead lettered multiple times),
routing-keys - the routing keys (including CC keys but excluding BCC ones) the message was published with,
there will be more, but these are the things you want to pay attention to. by examining these properties on the message, you can re-send the original message back through the original exchange, with the original routing-keys. alternatively, you can re-send straight to the original destination queue... i think sending through the exchange would be better, personally, as the original queue might not exist anymore (depending on system configuration, consumers creating exclusive queues, etc).
with this information, recreating the feature set should not be too difficult. rabbitmq provides all of the features that you need, you just have to write a bit of code to take advantage of it.
I am using RabbitMQ version 3.0.2 & I see close to 1000 message in Error queue. I want to know
At what point messages are moved to the error queues?
Is there a way to know why a certain message is being moved to an error queue?
Is there any way to move message from error queue to normal queue?
Thank you
a) they fail to deserialize or b) the consumer throws an exception processing that message five times
Not really... If you peek at the message in the queue, the payload headers might contain a note but I don't think we did that. If you turn logging on (NLog, log4net, etc) you should be able to see the exceptions in your log. You'll have to correlate message ids at that point to figure out exactly why.
There is no built in way via MassTransit. Mostly because there doesn't seem to be a great, generic way to handle this. Everyone wants some process around this. Dru did create a BusDriver app (in the main MT source repo) that could be used to move messages back to the exchange in question. This default behaviour is there so you at least know things have been failing if you don't put in the infrastructure to handle it.
To add to Travis' answer, During my development I found some other reasons for messages going onto the error queue:
The published message type has no consumer
A SAGA and a consumer are expecting the same concrete message type. Even if you try and differentiate using "Accepts" and ".Selected", both a SAGA and a Consumer should not be programmed to receive the same message type.