I'm trying to consume messages on an ActiveMQ topic from a C# application. I'm using the 1.3 .net release, and I don't receive any messages.
I have existing code that uses older libraries (and libraries built on top of libraries that I don't want to use or upgrade) that work fine, so I know messages are travelling along the topic, but my simple program with newer libraries just doesn't see them.
I'm using code copied and pasted from http://remark.wordpress.com/articles/publish-subscribe-with-activemq-and-nms/ as my test. The SimpleTopicSubscriber (with a simple change to make it a non-durable consumer) just doesn't receive anything. The SimpleTopicPublisher works just fine - I can send a simple message and it gets through, no problem.
Looking at the JMX console, I can see my subscriber connect, see that it's on the right topic, but it just doesn't get any messages.
Any ideas? I've even tried using the 1.2 libraries, but that didn't make any difference.
Thanks
Matt
Turns out there were 2 problems. I edited the example code incorrectly and passed in a durable consumer id instead of a selector. Not terribly clever.
Once I'd fixed this, I could receive messages, but only with the 1.1 release of the NMS/ActiveMq dlls. I couldn't receive messages with version 1.2 or 1.3.
But if I pass wireformat.version=2 as a parameter when connecting, everything works ok. The broker is 5.0.0 and I strongly suspect this would work ok with a later version of the broker.
Thanks
Matt
There are a couple reasons why your client might not be receiving messages, one could be that you didn't call Connection.Start(), without that the message pump won't start dispatching messages to your client. The other reason might be that your now non-durable Topic subscriber was started after the publisher in which case there'd be no messages delivered since you are using a Topic and Topics are like queue's in that once a message is sent its forgotten about so a subscriber that joins up later doesn't receive any messages that were sent before it subscribed.
Regards
Tim.
Open Source Integration: http://fusesource.com
Related
Using RabbitMQ 3.8.14 and 0.5.0 of the plugin
https://github.com/noxdafox/rabbitmq-message-deduplication
Along with the MassTransit library in C#.
If I set up a queue to use deduplication it seems that I can only get it to work if I turn off the publisher confirms.
With publisher confirms set to true (the default and desired setting) then when a duplicate is sent I get an exception like the following
MassTransit.RabbitMqTransport.MessageNotAcknowledgedException
If I set it to false then duplicates are filtered out and things work as expected.
I would like to have the deduplication and the message confirmations working if possible.
MassTransit with publisher confirm is not supported by the plugin.
This is due to the fact the library is asking to confirm the message was published into the queue. As the message is a duplicate, it will not be published within the queue. The plugin informs the Broker of this fact and the Broker sends a negative acknowledgement back to the library.
As the Broker does not support complex notifications semantics, there is no simple way to sort this out. In other words, the Broker can only state whether the messages was published in the queue. It cannot provide any extra information in regards.
You can find a lengthy explanation of the above in this comment.
I'm going through a few examples using NServiceBus and I've stumbled across a feature I'm hoping ships with MassTransit (As it is a free service).
The feature is based around 'poisoned' messages.
If, due to a bug in your system, these messages cant ever be handled, and end up permanently in the error queue.
NServiceBus has a cool feature whereby, once you have corrected the bugs in your code, allows those messages in the error queue to be 'redirected' to the original working queue, to be redelivered.
This is done by using a NServiceBus specific tool :- ReturnToSourceQueue.exe.
Does MassTransit have a similar tool for this kind of issue?
Or is there another workaround availble, preferbly to work with RabbitMQ.
With RabbitMQ, it's easy to move messages between queues. You can use the management console to do it manually, by installing the shovel plug-in.
You can also create shovels in RabbitMQ that are scheduled, and perform the message movement in response to that schedule. The visibility of having the shovels configured in RabbitMQ has been invaluable to our operations staff, since they rarely think that a Windows Scheduled Task (or other random scheduler) is going to be doing something as risky as moving previously failed messages back into the production queues.
I would suggest reading this blog post on how MassTransit deals with poison messages: Error Handling in MassTransit with RabbitMQ
The tooling around RabbitMQ is so much better than anything MSMQ provides, which is one of the reasons we have completely abandoned MSMQ for production queuing.
This functionality is easily recreated with nothing more than RabbitMQ and a bit of code. While it's nice that NServicebus includes it, building it with MassTransit should be easy enough.
(note: i haven't used .NET in a few years, so my knowledge of NSB and MT are a bit rusty... this will be high level answer only, no code)
The thing to start with, is a proper configuration of a dead letter exchange and a poison message queue. https://www.rabbitmq.com/dlx.html
Once you have knowledge that a message is causing errors and is a bad message, you can reject or nack (with no requeue) the message in order to send it through the dead letter exchange (DLX).
Once a message has gone through the DLX, you will have some additional properties on the message, including:
queue - the name of the queue the message was in before it was dead-lettered,
exchange - the exchange the message was published to (note that this will be a dead letter exchange if the message is dead lettered multiple times),
routing-keys - the routing keys (including CC keys but excluding BCC ones) the message was published with,
there will be more, but these are the things you want to pay attention to. by examining these properties on the message, you can re-send the original message back through the original exchange, with the original routing-keys. alternatively, you can re-send straight to the original destination queue... i think sending through the exchange would be better, personally, as the original queue might not exist anymore (depending on system configuration, consumers creating exclusive queues, etc).
with this information, recreating the feature set should not be too difficult. rabbitmq provides all of the features that you need, you just have to write a bit of code to take advantage of it.
I am using RabbitMQ version 3.0.2 & I see close to 1000 message in Error queue. I want to know
At what point messages are moved to the error queues?
Is there a way to know why a certain message is being moved to an error queue?
Is there any way to move message from error queue to normal queue?
Thank you
a) they fail to deserialize or b) the consumer throws an exception processing that message five times
Not really... If you peek at the message in the queue, the payload headers might contain a note but I don't think we did that. If you turn logging on (NLog, log4net, etc) you should be able to see the exceptions in your log. You'll have to correlate message ids at that point to figure out exactly why.
There is no built in way via MassTransit. Mostly because there doesn't seem to be a great, generic way to handle this. Everyone wants some process around this. Dru did create a BusDriver app (in the main MT source repo) that could be used to move messages back to the exchange in question. This default behaviour is there so you at least know things have been failing if you don't put in the infrastructure to handle it.
To add to Travis' answer, During my development I found some other reasons for messages going onto the error queue:
The published message type has no consumer
A SAGA and a consumer are expecting the same concrete message type. Even if you try and differentiate using "Accepts" and ".Selected", both a SAGA and a Consumer should not be programmed to receive the same message type.
Using NserviceBus 3.3 with Raven for subscription persistence.
I'm creating a prototype application that will consume messages from a publisher in our test environment. The application will only be used for a few weeks, at which point it may be (essentially) thrown away in its current form.
I don't want the publisher to continue to send messages to the outbound queue for this subscriber. In effect, I want its existence to be completely removed from the system.
How would I go about removing all knowledge of this subscriber from the system?
To do this you need to manage subscriptions manually.
Have a look at the PubSub sample, specifically this file and you also need to tell the bus not to autosubscribe, this sample should provide you with all the code required to do this.
The link to the PubSub article is broken. Here is the new link: https://github.com/Particular/NServiceBus.Msmq.Samples/tree/master/PubSub
Well I've been doing this NServiceBus project for a while and once I got it working for PubSub I then spent the rest of the time on the actual workflow logic. However, I can see a serious issue which I want to get around (or rather learn how to handle correctly).
A publisher publishes a message to the storage queues of any subscribers as far as I understand. Great. But what happens when the subscriber isn't running (I've read other posts about this and they don't seem to be asking the same question).
Scenario - I get the publisher to Publish a message when no subscribers are running (attached/requested messages to be relayed to them).. I then find that.. the message is "gone" just simply isn't there! where did it go? Did the publisher say "hey, no one's subscribing to this, so I wont bother publishing it?", shouldn't it NOT do that and require at least one subscriber?
Can anyone shed any light on this? (nservicenewbie)
You should publish an event that has happened - a statement of fact, that other handler may or may not be interested in. It's perfectly valid to have zero subscribers! If this is not the case then maybe you should be Send()ing a command instead of Publish()ing an event.
If you are using a persistent subscription storage, start the subscriber up once and it will always be subscribed. If the subscriber is offline, messages for it will pile up in its Input Queue, ready to be processed when the subscriber comes back online.
If you're just testing with NServiceBus, the NServiceBus.Host.exe is running in the Lite profile, which uses in-memory (non-persistant) subscription storage, which would result in what you are seeing.
Ah ha! Well though it's not always an error to have no subscriber for a message type, there is a way to handle it.
In your publisher simply modify the:
IBus Bus
To use (you will need NServiceBus.Core.dll and the NS NServiceBus.Unicast):
IUnicastBus Bus
Then you can attach an handler to:
Bus.NoSubscribersForMessage += .......
This can then put the message in an error queue.. or perhaps retry forever.. or publish something else etc.. etc.. what ever you want. Thus ensuring there is nothing lost where your particular system (from a business perspective) requires an outcome