Using RabbitMQ 3.8.14 and 0.5.0 of the plugin
https://github.com/noxdafox/rabbitmq-message-deduplication
Along with the MassTransit library in C#.
If I set up a queue to use deduplication it seems that I can only get it to work if I turn off the publisher confirms.
With publisher confirms set to true (the default and desired setting) then when a duplicate is sent I get an exception like the following
MassTransit.RabbitMqTransport.MessageNotAcknowledgedException
If I set it to false then duplicates are filtered out and things work as expected.
I would like to have the deduplication and the message confirmations working if possible.
MassTransit with publisher confirm is not supported by the plugin.
This is due to the fact the library is asking to confirm the message was published into the queue. As the message is a duplicate, it will not be published within the queue. The plugin informs the Broker of this fact and the Broker sends a negative acknowledgement back to the library.
As the Broker does not support complex notifications semantics, there is no simple way to sort this out. In other words, the Broker can only state whether the messages was published in the queue. It cannot provide any extra information in regards.
You can find a lengthy explanation of the above in this comment.
Related
I am using MassTransit with RabbitMQ at transport layer, and faced the need of messages deduplication.
Adding new massage to the queue should be skipped if duplicated message already queued (even if that message is processing by consumer). Duplicates could be identified by content of message for example.
Sending DoWork1, DoWork2, DoWork3 could be processed in parallel, but sending DoWork1, DoWork2, DoWork2 - duplicate should be skipped, and as far as DoWork1, DoWork2 processed same messages could be enqueued and should not be supposed as duplicates.
Solution 1: use "RabbitMQ Message Deduplication Plugin" at the exchange layer, ideal as for me, but not sure that solves described problem.
Solution 2: implement custom middleware with third party data storage.
Is there any better solution for described problem?
Thanks for help in advance!
The RabbitMQ deduplication plugin was designed for that purpose.
You can either de-duplicate at the exchange or at the queue. The main difference is the exchange de-duplicates a message if it has seen it previously while the queue de-duplicates it if already contains a copy of it.
When publishing a message, just set the x-deduplication-header header with a string which uniquely identifies a message (for example the MD5 hash of its body).
Using custom middleware will allow you more freedom of action at the cost of your own development.
I'm going through a few examples using NServiceBus and I've stumbled across a feature I'm hoping ships with MassTransit (As it is a free service).
The feature is based around 'poisoned' messages.
If, due to a bug in your system, these messages cant ever be handled, and end up permanently in the error queue.
NServiceBus has a cool feature whereby, once you have corrected the bugs in your code, allows those messages in the error queue to be 'redirected' to the original working queue, to be redelivered.
This is done by using a NServiceBus specific tool :- ReturnToSourceQueue.exe.
Does MassTransit have a similar tool for this kind of issue?
Or is there another workaround availble, preferbly to work with RabbitMQ.
With RabbitMQ, it's easy to move messages between queues. You can use the management console to do it manually, by installing the shovel plug-in.
You can also create shovels in RabbitMQ that are scheduled, and perform the message movement in response to that schedule. The visibility of having the shovels configured in RabbitMQ has been invaluable to our operations staff, since they rarely think that a Windows Scheduled Task (or other random scheduler) is going to be doing something as risky as moving previously failed messages back into the production queues.
I would suggest reading this blog post on how MassTransit deals with poison messages: Error Handling in MassTransit with RabbitMQ
The tooling around RabbitMQ is so much better than anything MSMQ provides, which is one of the reasons we have completely abandoned MSMQ for production queuing.
This functionality is easily recreated with nothing more than RabbitMQ and a bit of code. While it's nice that NServicebus includes it, building it with MassTransit should be easy enough.
(note: i haven't used .NET in a few years, so my knowledge of NSB and MT are a bit rusty... this will be high level answer only, no code)
The thing to start with, is a proper configuration of a dead letter exchange and a poison message queue. https://www.rabbitmq.com/dlx.html
Once you have knowledge that a message is causing errors and is a bad message, you can reject or nack (with no requeue) the message in order to send it through the dead letter exchange (DLX).
Once a message has gone through the DLX, you will have some additional properties on the message, including:
queue - the name of the queue the message was in before it was dead-lettered,
exchange - the exchange the message was published to (note that this will be a dead letter exchange if the message is dead lettered multiple times),
routing-keys - the routing keys (including CC keys but excluding BCC ones) the message was published with,
there will be more, but these are the things you want to pay attention to. by examining these properties on the message, you can re-send the original message back through the original exchange, with the original routing-keys. alternatively, you can re-send straight to the original destination queue... i think sending through the exchange would be better, personally, as the original queue might not exist anymore (depending on system configuration, consumers creating exclusive queues, etc).
with this information, recreating the feature set should not be too difficult. rabbitmq provides all of the features that you need, you just have to write a bit of code to take advantage of it.
I'm using ActiveMQ and I would like to know how to solve this specific case.
When the consumer is down, the producer sends a message to the queue. The message will remain in the queue until the consumer is running to consume it.
Now imagine I shutdown the producer, the message will STILL remain in the queue. Now i run the consumer and it will try to consume that message, but won't be able to reply back to the producer since its down.
I would like to solve this problem by cleaning the messages if the producer is out.
The ActiveMQ Broker cleans the Queue after stopping. I would like to do the same for the messages of a respective producer.
Thanks.
Based on what I understand now from your question and additional comments I propose to add a Message Property to your messages to identify the Producer, and write a small utility that uses a Message Selector to read all messages matching the Producer from the queue. You can run that utility straight after the Producer is stopped (or crashes), and that should quite accurately do what you want to achieve.
EDIT: although primarily focused on EE, the Sun/Oracle JavaEE Tutorial contains a very good chapter on general JMS programming that starts off with standalone producers and consumers. The accompanying source code bundle can be downloaded here, the ready to comoile samples in that bundle should get you started very quickly.
You can solve it a couple of ways. One is to set a TTL on the message so it goes away. The other is to connect via JMX and purge the Queue or remove the specific message using a selector statement or with the Message's specific MessageId value.
See this article for some hints.
I'm currently using ActiveMQ for my queueing system, and I'm wanting to make the transition to RabbitMQ. One feature I've been using that belongs to ActiveMQ is a redelivery policy, as sometimes our consumer rejects a message because it cannot handle it at this time, but may want to try again later, so it requeues it.
Right now in AMQP, when I reject a message, it's instantly pulled off the queue again immediately and tried again.
Is there a way, in RabbitMQ, to specify a redelivery policy for a queue, consumer, or message?
I also had problems with that behaviour. According to documentation (as far as I remember, maybe in newer version something changed) after requeue it is not stated where a message will be placed (it was described as undetermined). In my testcases (with version 2.8.2) some of messages were put to the end of a queue and one message (precisely first from clients prefetch) land on beggining (and being consumed immediately). In our application this caused livelock.
You could walkaround this by publishing copy of message to a queue and acking already delivered one in one transaction (but I recommend to carefully read section about transactions in docs) or use deadlettering to deal with temporaly unprocessable messages.
I'm trying to consume messages on an ActiveMQ topic from a C# application. I'm using the 1.3 .net release, and I don't receive any messages.
I have existing code that uses older libraries (and libraries built on top of libraries that I don't want to use or upgrade) that work fine, so I know messages are travelling along the topic, but my simple program with newer libraries just doesn't see them.
I'm using code copied and pasted from http://remark.wordpress.com/articles/publish-subscribe-with-activemq-and-nms/ as my test. The SimpleTopicSubscriber (with a simple change to make it a non-durable consumer) just doesn't receive anything. The SimpleTopicPublisher works just fine - I can send a simple message and it gets through, no problem.
Looking at the JMX console, I can see my subscriber connect, see that it's on the right topic, but it just doesn't get any messages.
Any ideas? I've even tried using the 1.2 libraries, but that didn't make any difference.
Thanks
Matt
Turns out there were 2 problems. I edited the example code incorrectly and passed in a durable consumer id instead of a selector. Not terribly clever.
Once I'd fixed this, I could receive messages, but only with the 1.1 release of the NMS/ActiveMq dlls. I couldn't receive messages with version 1.2 or 1.3.
But if I pass wireformat.version=2 as a parameter when connecting, everything works ok. The broker is 5.0.0 and I strongly suspect this would work ok with a later version of the broker.
Thanks
Matt
There are a couple reasons why your client might not be receiving messages, one could be that you didn't call Connection.Start(), without that the message pump won't start dispatching messages to your client. The other reason might be that your now non-durable Topic subscriber was started after the publisher in which case there'd be no messages delivered since you are using a Topic and Topics are like queue's in that once a message is sent its forgotten about so a subscriber that joins up later doesn't receive any messages that were sent before it subscribed.
Regards
Tim.
Open Source Integration: http://fusesource.com