I'm getting an strange behavior in my DEV environment. When I send a message to my queue it's dispatched correctly, but the next message (can be same content or different) always fails and is directly sent to my deadletter queue. And then this pattern is repeated, one OK, one sent to deadletter.
In my local setup, everything is working OK, but not in my DEV env so this makes a little difficult to debug/troubleshoot. Not sure what could be wrong or different. I'm new on RabbitMq so maybe I need to include more information (if so please let me know).
Does anyone have an idea of what could be causing it? Or does anyone have experienced something like this before?
RabbitMq Version is: 3.8.2
My rabbitmq.config file is:
[{rabbitmq_management,[{tcp_config,[{port,15672}]}]}, {rabbit,[{total_memory_available_override_value,3999997952}, {tcp_listeners,[5672]}, {loopback_users,[]}]}].
My two queues are configured this way:
**my-queue.dev**
Type: Classic
Features: D, DLX
**my-queue.dev.deadletter**
Type: Classic
Features: D
Kind regards!
Related
TL;DR: I need to "replay" dead letter messages back into their original queues once I've fixed the consumer code that was originally causing the messages to be rejected.
I have configured the Dead Letter Exchange (DLX) for RabbitMQ and am successfully routing rejected messages to a dead letter queue. But now I want to look at the messages in the dead letter queue and try to decide what to do with each of them. Some (many?) of these messages should be replayed (requeued) to their original queues (available in the "x-death" headers) once the offending consumer code has been fixed. But how do I actually go about doing this? Should I write a one-off program that reads messages from the dead letter queue and allows me to specify a target queue to send them to? And what about searching the dead letter queue? What if I know that a message (let's say which is encoded in JSON) has a certain attribute that I want to search for and replay? For example, I fix a defect which I know will allow message with PacketId: 1234 to successfully process now. I could also write a one-off program for this I suppose.
I certainly can't be the first one to encounter these problems and I'm wondering if anyone else has already solved them. It seems like there should be some sort of Swiss Army Knife for this sort of thing. I did a pretty extensive search on Google and Stack Overflow but didn't really come up with much. The closest thing I could find were shovels but that doesn't really seem like the right tool for the job.
Should I write a one-off program that reads messages from the dead letter queue and allows me to specify a target queue to send them to?
generally speaking, yes.
you could set up a delayed re-try to resend the message back to the original queue, using a combination of the delay message exchange plugin.
but this would only automate the retries on an interval, and you may not have fixed the problem before the retries happen.
in some circumstances this is ok - like when the error is caused by an external resource being temporarily unavailable.
in your case, though, i believe your thoughts on creating an app to handle the dead letters is the best way to go, for several reasons:
you need to search through the messages, which isn't possible RMQ
this means you'll need a database to store the messages from the DLX/queue
because you're pulling the messages out of the DLX/queue, you'll need to ensure you get all the header info from the message so that you can re-publish to the correct queue when the time comes.
I certainly can't be the first one to encounter these problems and I'm wondering if anyone else has already solved them.
and you're not!
there are many solutions to this problem that all come down to the solution you've suggested.
some larger "service bus" implementations have this type of feature built in to them. i believe NServiceBus (or the SaaS version of it) has this built in, for example - though I'm not 100% sure of it.
if you want to look into this further, do some search for the term "poison message" - this is generally the term used for this situation. I've found a few things on google with a quick search, that may help you down the path:
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-January/025019.html
https://web.archive.org/web/20170809194056/http://tafakari.co.ke/2014/07/rabbitmq-poison-messages/
https://web.archive.org/web/20170809170555/http://kjnilsson.github.io/blog/2014/01/30/spread-the-poison/
hope that helps!
I am using RabbitMQ version 3.0.2 & I see close to 1000 message in Error queue. I want to know
At what point messages are moved to the error queues?
Is there a way to know why a certain message is being moved to an error queue?
Is there any way to move message from error queue to normal queue?
Thank you
a) they fail to deserialize or b) the consumer throws an exception processing that message five times
Not really... If you peek at the message in the queue, the payload headers might contain a note but I don't think we did that. If you turn logging on (NLog, log4net, etc) you should be able to see the exceptions in your log. You'll have to correlate message ids at that point to figure out exactly why.
There is no built in way via MassTransit. Mostly because there doesn't seem to be a great, generic way to handle this. Everyone wants some process around this. Dru did create a BusDriver app (in the main MT source repo) that could be used to move messages back to the exchange in question. This default behaviour is there so you at least know things have been failing if you don't put in the infrastructure to handle it.
To add to Travis' answer, During my development I found some other reasons for messages going onto the error queue:
The published message type has no consumer
A SAGA and a consumer are expecting the same concrete message type. Even if you try and differentiate using "Accepts" and ".Selected", both a SAGA and a Consumer should not be programmed to receive the same message type.
Well I've been doing this NServiceBus project for a while and once I got it working for PubSub I then spent the rest of the time on the actual workflow logic. However, I can see a serious issue which I want to get around (or rather learn how to handle correctly).
A publisher publishes a message to the storage queues of any subscribers as far as I understand. Great. But what happens when the subscriber isn't running (I've read other posts about this and they don't seem to be asking the same question).
Scenario - I get the publisher to Publish a message when no subscribers are running (attached/requested messages to be relayed to them).. I then find that.. the message is "gone" just simply isn't there! where did it go? Did the publisher say "hey, no one's subscribing to this, so I wont bother publishing it?", shouldn't it NOT do that and require at least one subscriber?
Can anyone shed any light on this? (nservicenewbie)
You should publish an event that has happened - a statement of fact, that other handler may or may not be interested in. It's perfectly valid to have zero subscribers! If this is not the case then maybe you should be Send()ing a command instead of Publish()ing an event.
If you are using a persistent subscription storage, start the subscriber up once and it will always be subscribed. If the subscriber is offline, messages for it will pile up in its Input Queue, ready to be processed when the subscriber comes back online.
If you're just testing with NServiceBus, the NServiceBus.Host.exe is running in the Lite profile, which uses in-memory (non-persistant) subscription storage, which would result in what you are seeing.
Ah ha! Well though it's not always an error to have no subscriber for a message type, there is a way to handle it.
In your publisher simply modify the:
IBus Bus
To use (you will need NServiceBus.Core.dll and the NS NServiceBus.Unicast):
IUnicastBus Bus
Then you can attach an handler to:
Bus.NoSubscribersForMessage += .......
This can then put the message in an error queue.. or perhaps retry forever.. or publish something else etc.. etc.. what ever you want. Thus ensuring there is nothing lost where your particular system (from a business perspective) requires an outcome
I'm trying to consume messages on an ActiveMQ topic from a C# application. I'm using the 1.3 .net release, and I don't receive any messages.
I have existing code that uses older libraries (and libraries built on top of libraries that I don't want to use or upgrade) that work fine, so I know messages are travelling along the topic, but my simple program with newer libraries just doesn't see them.
I'm using code copied and pasted from http://remark.wordpress.com/articles/publish-subscribe-with-activemq-and-nms/ as my test. The SimpleTopicSubscriber (with a simple change to make it a non-durable consumer) just doesn't receive anything. The SimpleTopicPublisher works just fine - I can send a simple message and it gets through, no problem.
Looking at the JMX console, I can see my subscriber connect, see that it's on the right topic, but it just doesn't get any messages.
Any ideas? I've even tried using the 1.2 libraries, but that didn't make any difference.
Thanks
Matt
Turns out there were 2 problems. I edited the example code incorrectly and passed in a durable consumer id instead of a selector. Not terribly clever.
Once I'd fixed this, I could receive messages, but only with the 1.1 release of the NMS/ActiveMq dlls. I couldn't receive messages with version 1.2 or 1.3.
But if I pass wireformat.version=2 as a parameter when connecting, everything works ok. The broker is 5.0.0 and I strongly suspect this would work ok with a later version of the broker.
Thanks
Matt
There are a couple reasons why your client might not be receiving messages, one could be that you didn't call Connection.Start(), without that the message pump won't start dispatching messages to your client. The other reason might be that your now non-durable Topic subscriber was started after the publisher in which case there'd be no messages delivered since you are using a Topic and Topics are like queue's in that once a message is sent its forgotten about so a subscriber that joins up later doesn't receive any messages that were sent before it subscribed.
Regards
Tim.
Open Source Integration: http://fusesource.com
I have a bunch of message on an outgoing MSMQ that have been addressed incorrectly due to a typo. They are stuck on the outgoing queue since the box they have been directed at does not exist.
The name of the queue is of the form
DIRECT=OS:foo\private$\MyQueue and is in state Waiting to connect
and I want to send to
DIRECT=OS:bar\private$\MyQueue
Will an entry in the hosts file suffice, or if not is there any other way that I can go this? I don't want to have to create a server called and a process that recieves the messages and forwards them on to the correct server if at all possible
I tried the idea I suggested above of adding the incorrect entry into the Hosts file with the correct IP address. This caused the messages to get dispatched to the remote computer, but I think WCF dropped them as they didn't have the correct endpoint adress.
Not recommended if you don't want to lose the messages! Fortunately for me these were not critical, so it was worth a try.
Oh well
Modan,
I experienced something similar years ago and the only way I could find was to setup the box and Q (in a VM) and forward them to correct queue. Maybe you can delete and resend the messages with the fix? Is that possible? Unfortunately don't think there's a good way around this.
-Bryan