Hi guys,
I'm looking for a method to count a number of matching messages in ActiveMQ. Here is the code for adding a message:
ObjectMessage myBeanMessage = session.createObjectMessage();
myBeanMessage.setObject(myBean);
myBeanMessage.setStringProperty("myProperty", myBean.getProperty());
producer.send(myBeanMessage);
Now I want to count the number of message in myQueue whose myProperty = "a property String", for example, but do not found any QueueMBean api:
http://activemq.apache.org/maven/5.7.0/activemq-core/apidocs/org/apache/activemq/broker/jmx/QueueViewMBean.html
The only api that maybe satisfy my requirement is
int copyMatchingMessagesTo(String selector, String destinationName) throws Exception
Copies the messages matching the given selector
Returns: the number of messages copied
Throws: Exception
which mean that I have to copy all matching messages to another queue and count the number of messages copied.
However, I feel that to copy all matching message just to find how many messages are copied is somehow "unneccessary waste of resource".
So, is there any way that I can count the number of matching messages directly?
Thanks
well, you can always browse the queue using that selector, and count the messages browsed. That means you will read the messages to a client, but not really copying them to another queue. If that is more performant/resource friendly in your scenario or worse, I cannot tell.
The use case seems to tread into the realm of ActiveMQ (or JMS in general) anti-patterns.
You need to remember that a Queue is not a database and treating it as such will, most likely end in tears. In JMS you have the option of using a Queue Browser with selector to browse the Queue contents. You must tread carefully there however as the spec only states that a browser will give you a snapshot of the Queue with no guarantee that you will browse all message. In ActiveMQ when you browse a Queue you can only browse what will fit into memory and then the browse stops so your count could be way off for really deep Queues.
Related
I am using ActiveMQ to store messages to be used later. It is working as expected, but there is a specific scenario I need to fit which I cannot figure out.
The short question is this.
Is there a way to run a query on the queue to find out all messages with certain header values?
The problem in detail is this :
So there is a set of data that is coming in multiple messages and the requirement is to use that data only after all messages for that has come in.
So if the dataset has lets say 50 messages i need to wait for those 50 messages and then read them in.
I am adding headers to each message to denote they belong to a certain set.
Like "TotalSets"=50 , "SetId"=39 .
I would like to write a thread that keeps tracing if all sets for a particular batch has arrived.
NMS is a .NET equivalent to the JMS messaging API so the means of filtering messages is the same as in JMS, your subscription applies a JMS Message Selector when created to tell the broker what messages it is interested in. The session methods to create MessageConsumer instances have variants that accept the selector using the JMS defined syntax which is your means of filtering messages.
Before a consumer nacks a message, is there any way the consumer can modify the message's state so that when the consumer consumes it upon redelivery, it sees that changed state. I'd rather not reject + reenqueue new message, but please let me know if that's the only way to accomplish this.
My goal is to determine how many times specific messages are being redelivered. I see two ways of doing this:
(1) On the message itself as described above. The message would be a container of basic stats and the application payload message.
(2) In some external storage. We would uniquely identify the message by the message id that we set.
I know 2 is possible, but my question is if 1 is possible.
There is no way to do (1) like you want. You would need to change the message, thus the message would become another message. If you want to do something like that (and it's possible that you meant this with I'd rather not reject + reenqueue new message) - you should ACK the message, increment one field in it and publish it again (again, maybe this is what you meant when you said reenqueue it). So your message payload would have some ID, counter, and again (obviously different) payload that is the content.
Definitvly much better way is (2) for multiple reasons:
it does not interfere with business logic, that is this diagnostic part is isolated
you are leaving re-queueing to rabbitmq (as you are supposed to do), meaning that you are not worrying about losing messages and handling some message meta info which has no use for you business logic
it's actually supposed to be used - the ACKing and NACKing, that's why it's in the AMQP specification
since you do need the number of how many times specific messages have been redelivered, you have it somewhere externally, meaning that it's independent of (rabbitmq's) message persistence, lifetime, potentially queue durability mirroring etc
Even if this question was marked as solved some time ago, I want to mention that there is a way at least for the redelivery. It might be integrated after the original answer. There is a different type of queues in RabbitMQ called Quorum queues.
Quorum queues offer the option to set redelivery limit:
Quorum queues support poison message handling via a redelivery limit. This feature is currently unique to Quorum queues.
In order to archive this, RabbitMQ is counting the numbers of deliveries in the header. The header attribute is called: x-delivery-count
In out product we need to count occasionally the number of messages in a queue that hold a specific property value.
The straight forward way is to use a queue browser with a selector. But:
Is there a way to get this statistics without browsing all relevant messages? We don't need the data, only the count.
If we do go for the queue browser approach, will it also browse messages that were consumed by some consumer, but not acknowledged yet?
There is not guaranteed way to count messages in a queue based on a selector. There are a number of reasons for this one of which is that the JMS Spec does not require that a QueueBrowser even return every message on a Queue. ActiveMQ won't return more than the number of messages that it can fit into memory so if you have a deep queue you won't have an accurate count. None of the management tools will do this for you either since it's not really ActiveMQ's job to be a database.
Here's some articles on messaging anti-patterns that you should read so you are better prepared for adventures in messaging.
Article 1 and Article 2
I am using RabbitMQ version 3.0.2 & I see close to 1000 message in Error queue. I want to know
At what point messages are moved to the error queues?
Is there a way to know why a certain message is being moved to an error queue?
Is there any way to move message from error queue to normal queue?
Thank you
a) they fail to deserialize or b) the consumer throws an exception processing that message five times
Not really... If you peek at the message in the queue, the payload headers might contain a note but I don't think we did that. If you turn logging on (NLog, log4net, etc) you should be able to see the exceptions in your log. You'll have to correlate message ids at that point to figure out exactly why.
There is no built in way via MassTransit. Mostly because there doesn't seem to be a great, generic way to handle this. Everyone wants some process around this. Dru did create a BusDriver app (in the main MT source repo) that could be used to move messages back to the exchange in question. This default behaviour is there so you at least know things have been failing if you don't put in the infrastructure to handle it.
To add to Travis' answer, During my development I found some other reasons for messages going onto the error queue:
The published message type has no consumer
A SAGA and a consumer are expecting the same concrete message type. Even if you try and differentiate using "Accepts" and ".Selected", both a SAGA and a Consumer should not be programmed to receive the same message type.
In our scenario I'm thinking of using the pub sub technique. However I don't know which is the better option.
1 ########
A web service of ours will publish a message that something has happened when it is called externally, ExternalPersonCreatedMessage!
This message will contain a field that represents the destinations to process the message into (multiple allowed).
Various subscribers will subscribe. These subscribers will filter the message to see if any action is required by checking the destination field.
2 ########
A web service of ours will parse the incoming call and publish specific types of messages depending on the destinations supplied in the field. i.e. many Destination[n]PersonCreatedMessage messages would be created.
Subscribers will subscribe to only the specific message they care for. i.e. not having to filter any messages
QUESTIONS
Which of the above is the better option and why? And how do I stop myself from making RequestMessages. From what I've read/seen I should be trying to structure this in a way of PersonCreated, PersonDeleted i.e. SOMETHING HAS HAPPENED and NOT in the REQUEST SOMETHING TO HAPPEN form such as CreatePerson or DeletePerson
Are my thoughts correct? I've been looking for guidance on how to structure messages and making sure I don't go down a wrong path but have found no guidance out there on do's and dont's. Can any one help and guide? I want to try and get this correct from the off :)
Based on the integration scenario in the referenced article, it appears to me that you may need a Saga to complete the workflow of accept message -> operate on message -> send confirmation. In the case that the confirmation is sent immediately after the operation, you could use NSBs message handler pipeline feature which allows you to chain handlers in a specified sequence such as...
First<FilterHandler>.Then<DoWorkHandler>().AndThen<SendConfirmationHandler>();
In terms of the content filtering, you can do this although you incur some transport overhead, meaning the queue will have to accept the message and the process will always call the first handler on every message(you can short-circuit the above pipeline at any point). It may be the case that what you really want is a Distributor/Worker setup where all Workers are the same and you can handle some load.
If you truly have different endpoints with completely different logic, then I would have the Publisher process(only accepts and Publishes message) do the work of translating the inbound message to something else a Subscriber can then be interested in. If then you find that a given Published message only ever has 1 Subscriber, then you don't need to Publish at all, you need to just Bus.Send() to the correct endpoint.
The way NServiceBus handles pub-sub is more like your option two.
A publisher service has an input queue and a subscription store.
A subscriber service has an input queue
The subscriber, on start-up will send a subscription message to the input queue of the publisher
The subscription message contains the type of message subscriber is interested in and the subscribers queue address
The publisher records the subscription in the subscription store.
The publisher receives a message.
The publisher evaluates the message type against the list of subscriptions
For each match found the publisher sends the message to the queue address.
In my opinion, you should stop thinking about destinations. Messages are messages. They should not have any inherent destination information in them. The subscription mechanism defines the addressing/routing requirements for the solution.