Content-based consumer priority in ActiveMQ - activemq

Question

I'm working with consumers that need to load heavy resources to handle messages' content. Messages in the same family require the same resources.
I tried using message grouping to make sure messages in the same family are given to a consumer where the resources are already loaded. However I want the consumer have priority on messages in the same family, not exclusivity (loading resources takes time, treating messages takes even longer). For example, if consumers 1 and 2 are available then messages with ID group1 should always be given to consumer 1 and messages with ID group2 should always be given to consumer 2. If only consumer 2 is available, it should consume any message. That way, consumer 1 has priority on messages group1, but not exclusivity. Likewise consumer 2 has priority on messages group2, but not exclusivity.
Is there a way for message grouping in AMQ to not give exclusivity to consumer ? Or a way for consumer priority to be set dynamically based on content?
I'm using ActiveMQ 5.16.2, but I can switch to Artemis if that is preferable.

Answer 1

In this use case, I suggest filtering over message groups. I would contend this you are seeking to partition the message flow, where the competing consumer pattern provided by message groups is not meeting your need. If you require these flows to be handled by a specific consumer, it is practical that you would eventually want to have that consumer have a HA-standby or parallel consumer-- which would be an additional driver to partition or filter the traffic.
There are two approaches-- you can do server-driven config using Virtual Destinations (via CompositeQueue w/ filtering, etc) or client-driven config using Virtual Topics and selectors.
Here is the client-driven approach.
Publish to a Virtual Topic (ie. topic://VT.ORDER.EVENT)
Consume from Virtual Consumer queue(s) (ie. queue://VQ.APP1.VT.ORDER.EVENT, queue://VQ.APP2.VT.ORDER.EVENT)
Have consumers connect with selector
Each group of messages will arrive to their own queue, and you can have 1 .. n consumers on each queue.
Bonus: You can still have consumers monitor the full traffic flow by registering a wildcard selector or subscribing directly to the topic.
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="VT.>" prefix="VQ.*." selectorAware="true"/>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
Persist selectors:
<plugins>
<virtualSelectorCacheBrokerPlugin persistFile="<some path>${activemq.data}/virtual-topic-selectorcache.data" />
</plugins>
ref: Virtual Topics

Answer 2

The whole point of message grouping is to serialize the consumption of messages. The simplest way to do that is to that is to ensure all the messages in the group are dispatched to the same consumer. Therefore, there is no way for message grouping to not give exclusivity to consumer as that would defeat the purpose of grouping in the first place. That said, a single consumer can receive messages from multiple groups concurrently. This is happens when there are more groups than consumers.
It's also worth noting that you can't control which consumer consumes which group. The point of grouping is just that one consumer gets the messages from the group, but not which one.
Consumer priority is set by the consumer itself as the documentation describes. It is not based on the content of the message being dispatched.
You might try setting a property on the messages which reflect what "family" they are in and then your consumers can use selectors to consume the messages they care about.
Keep in mind that you can always have multiple consumers in your client application. Each client could potentially have two consumers - a high priority consumer with a selector and then a lower priority consumer without a selector. The broker would prioritize the consumers with selectors but if there was no match it would dispatch messages to the consumers without selectors.