I understand the topic is for broadcasting messages to the clients. But in the ActiveMQ document, it says about load balancing:
Another way of explaining Message Groups is that it provides sticky load balancing of messages across consumers; where the JMSXGroupID is kinda like a HTTP session ID or cookie value and the message broker is acting like a HTTP load balancer.
Here comes what I don't understand.
Is it not broadcasting?
What I want to do is below:
If producer send messages to group A, somehow the broker control the traffic and the messages "physically" should not go to group B.
Is it even possible with Virtual Topic?
VirtualTopics create queues for each topic subscriber and replicates messages to each one, so it is "broadcasting" all messages to all subscribers.
ActiveMQ message groups provide you with an exclusive consumer thread for a given message group (distinct JMSXGroupID). You can't control which consumer picks up each group and the same consumer thread can process multiple groups.
As far as I know, if you set the JMSXGroupID on a message sent to a VirtualTopic, it will arrive in each subscriber's virtual queue. However, if you have multiple threads processing from those queues (maxConcurrentConsumers > 1), then they will be single-threaded per message group, etc.
Overall, if you want certain subscribers to only process a subset of messages sent to a topic, then use message selectors to down select this subset.
Related
I need a queue/message broker that would allow me to receive by 1 message for each user.
Previously I've been using FIFO SQS and each user has its own message group id, which allows me to have 1 message inflight for each user.
But I've ended up with a problem when I have more than 20.000 messages in the queue, because A FIFO queue looks through the first 20k messages to determine available message groups, which makes all messages above the 20.000 limit to be unavailable for processing.
Do you guys have any advice for a message broker/queue that would allow me to achieve the same behavior without having a limit of 20.000 messages?
I was thinking about using Redis Lists (where each user has its own list and I do an RPOP for each existent list)
A couple options to overcome the consumer window problem you describe in ActiveMQ:
Use Virtual Topics and have consumers register a selector. The broker will filter messages to each consumer into their own queue. This does not require any broker changes and is completely dynamic via the client setting the selector.
Use Composite Destinations to filter messages to separate queues. This is static broker routing and requires server-side config change for any new routing rules. The clients do not need to be changed
Both approaches documented here:
ref: https://activemq.apache.org/virtual-destinations
I'm working with consumers that need to load heavy resources to handle messages' content. Messages in the same family require the same resources.
I tried using message grouping to make sure messages in the same family are given to a consumer where the resources are already loaded. However I want the consumer have priority on messages in the same family, not exclusivity (loading resources takes time, treating messages takes even longer). For example, if consumers 1 and 2 are available then messages with ID group1 should always be given to consumer 1 and messages with ID group2 should always be given to consumer 2. If only consumer 2 is available, it should consume any message. That way, consumer 1 has priority on messages group1, but not exclusivity. Likewise consumer 2 has priority on messages group2, but not exclusivity.
Is there a way for message grouping in AMQ to not give exclusivity to consumer ? Or a way for consumer priority to be set dynamically based on content?
I'm using ActiveMQ 5.16.2, but I can switch to Artemis if that is preferable.
In this use case, I suggest filtering over message groups. I would contend this you are seeking to partition the message flow, where the competing consumer pattern provided by message groups is not meeting your need. If you require these flows to be handled by a specific consumer, it is practical that you would eventually want to have that consumer have a HA-standby or parallel consumer-- which would be an additional driver to partition or filter the traffic.
There are two approaches-- you can do server-driven config using Virtual Destinations (via CompositeQueue w/ filtering, etc) or client-driven config using Virtual Topics and selectors.
Here is the client-driven approach.
Publish to a Virtual Topic (ie. topic://VT.ORDER.EVENT)
Consume from Virtual Consumer queue(s) (ie. queue://VQ.APP1.VT.ORDER.EVENT, queue://VQ.APP2.VT.ORDER.EVENT)
Have consumers connect with selector
Each group of messages will arrive to their own queue, and you can have 1 .. n consumers on each queue.
Bonus: You can still have consumers monitor the full traffic flow by registering a wildcard selector or subscribing directly to the topic.
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="VT.>" prefix="VQ.*." selectorAware="true"/>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
Persist selectors:
<plugins>
<virtualSelectorCacheBrokerPlugin persistFile="<some path>${activemq.data}/virtual-topic-selectorcache.data" />
</plugins>
ref: Virtual Topics
The whole point of message grouping is to serialize the consumption of messages. The simplest way to do that is to that is to ensure all the messages in the group are dispatched to the same consumer. Therefore, there is no way for message grouping to not give exclusivity to consumer as that would defeat the purpose of grouping in the first place. That said, a single consumer can receive messages from multiple groups concurrently. This is happens when there are more groups than consumers.
It's also worth noting that you can't control which consumer consumes which group. The point of grouping is just that one consumer gets the messages from the group, but not which one.
Consumer priority is set by the consumer itself as the documentation describes. It is not based on the content of the message being dispatched.
You might try setting a property on the messages which reflect what "family" they are in and then your consumers can use selectors to consume the messages they care about.
Keep in mind that you can always have multiple consumers in your client application. Each client could potentially have two consumers - a high priority consumer with a selector and then a lower priority consumer without a selector. The broker would prioritize the consumers with selectors but if there was no match it would dispatch messages to the consumers without selectors.
At the moment we have number of publishers (micro-services) which publish their messages to exchange. Each message has a serviceId attribute. The queue is connected to a single subscriber (micro-service) which processes the queue messages, processing of a single message is a costly operation (takes about 20-30 secs).
Currently we have the following situation: service A publishes ~200 messages, after some seconds service B publishes 2 messages. So the subscriber will process these 2 messages only after the first 200 will be processed.
We want to process the messages in the order they came to the queue, but with respect to the source serviceId.
Obvious solution is to split the queue to a separate queues (one per publisher) and subscribe to each queue separately, but the number of publishers can change, we need to request them dynamically and subscribe (unsubscribe) to them.
Another approach is to replicate our subscriber app to have one to one relationship between publisher and subscriber, but this will require more system resources.
What would be the best approach to handle this situation?
Thanks!
/!\ Be careful, publishers publish to an exchange, not to a queue.
We want to process the messages in the order they came to the queue,
but with respect to the source serviceId.
If I understand well, you want to load balance your messages according to a serviceId, and serviceIds are not known in advance.
The solution I would suggest here is to have a direct exchange, with routing keys such as xxxxx.<serviceId>. Then, you can bind one queue by serviceId (that is: one queue for service A, one for service B, ...), each consumer consuming on all queues.
Then you have to handle the publisher subscription: I would make a publisher publish a "hello" message, this message being consumed by each consumer, which in turn bind a new queue for that service (using xxxxx.<newServiceId>), and finally publish a response back (so that the publisher can start sending messages).
Note: each service queue is the same for all consumers, resulting in the worker configuration (see this tutorial)
Hope this helps.
I am attempting to use ActiveMQ 5.8.0 message groups in my application, and am not getting the results that I expected after reading the documentation.
I start two (or more) consumers for a particular queue, and then I send messages to the queue. In the producer's createMessage method, I am using:
message.setStringProperty("JMSXGroupID", "foo");
to set the GroupID. Note that for testing purposes, I am hard-coding "foo". It will eventually be a string set by the producer.
Since I only have a single message group being set in my messages, I expected to see that one consumer would become active and consume all of the messages in the queue, while the other one would remain quiescent.
Instead, I see the first message get processed by the first consumer, and then the second message get processed by the second consumer after the first consumer is finished. The consumers continue to take turns in this manner until all of the messages are consumed.
Is this the expected behavior, or is there some additional configuration that I need to do on either ActiveMQ or my producer or consumer to make sure that each GroupID gets associated to a single consumer.
AMQ message groups just guarantees that a single consumer will be active for a given group ("foo", etc) at a time...it doesn't bind that group to a specific consumer, so AMQ's internal consumer load balancing is likely just alternating between consumers.
regardless, this shouldn't be an issue when you are using dynamic JMSXGroupID values with multiple consumers, etc.
I have configured two brokers A and B using networkconnectors.
Is the message order preserved if I am using exclusive consumer (single consumer) or message groups(JMXgroupID)?
In the network of broker documentation I found this:
Total message ordering is not preserved with networks of brokers. Total ordering works with a single consumer but a networkBridge introduces a second consumer. In addition, network bridge consumers forward messages via producer.send(..), so they go from the head of the queue on the forwarding broker to the tail of the queue on the target. If single consumer moves between networked brokers, total order may be preserved if all messages always follow the consumer but this can be difficult to guarantee with large message backlogs.
In single broker message ordering is possible through exclusive consumer. What will happen if I am using network of brokers with exclusive consumers?
Total message ordering in a network of brokers doesn't work - even if you have a single consumer or multiple consumers using the "exclusive consumer" feature.
Consider the following scenario with 2 brokers (let's call these broker-A & broker-B) in a network, 1 consumer (consumer-A), 2 producers (producer-A & producer-B), and 1 queue (queue-A).
producer-A connects to broker-A, sends a message (message-1) to queue-A, and disconnects.
producer-B connects to broker-B, sends a message (message-2) to queue-A, and disconnects.
producer-A reconnects to broker-A, sends a message (message-3) to queue-A, and disconnects.
consumer-A connects to broker-A and receives the messages from queue-A. Even though the messages were sent in order message-1, message-2, message-3 the consumer will actually receive the messages in the order message-1, message-3, message-2 because message-1 and message-3 were sent to broker-A and message-2 was sent to broker-B and had to be moved across the network of brokers to broker-A based on consumer demand.
It's worth noting that one of the main goals of a network of brokers is scalability. However, in order to guarantee messages are consumed in order the messages have to be consumed serially which can drastically reduce performance and would almost certainly nullify any scalability gains provided by the network of brokers. Total message ordering and a network of brokers are fundamentally opposed ideas. If you really want total message ordering you shouldn't use a network of brokers.