I have configured two brokers A and B using networkconnectors.
Is the message order preserved if I am using exclusive consumer (single consumer) or message groups(JMXgroupID)?
In the network of broker documentation I found this:
Total message ordering is not preserved with networks of brokers. Total ordering works with a single consumer but a networkBridge introduces a second consumer. In addition, network bridge consumers forward messages via producer.send(..), so they go from the head of the queue on the forwarding broker to the tail of the queue on the target. If single consumer moves between networked brokers, total order may be preserved if all messages always follow the consumer but this can be difficult to guarantee with large message backlogs.
In single broker message ordering is possible through exclusive consumer. What will happen if I am using network of brokers with exclusive consumers?
Total message ordering in a network of brokers doesn't work - even if you have a single consumer or multiple consumers using the "exclusive consumer" feature.
Consider the following scenario with 2 brokers (let's call these broker-A & broker-B) in a network, 1 consumer (consumer-A), 2 producers (producer-A & producer-B), and 1 queue (queue-A).
producer-A connects to broker-A, sends a message (message-1) to queue-A, and disconnects.
producer-B connects to broker-B, sends a message (message-2) to queue-A, and disconnects.
producer-A reconnects to broker-A, sends a message (message-3) to queue-A, and disconnects.
consumer-A connects to broker-A and receives the messages from queue-A. Even though the messages were sent in order message-1, message-2, message-3 the consumer will actually receive the messages in the order message-1, message-3, message-2 because message-1 and message-3 were sent to broker-A and message-2 was sent to broker-B and had to be moved across the network of brokers to broker-A based on consumer demand.
It's worth noting that one of the main goals of a network of brokers is scalability. However, in order to guarantee messages are consumed in order the messages have to be consumed serially which can drastically reduce performance and would almost certainly nullify any scalability gains provided by the network of brokers. Total message ordering and a network of brokers are fundamentally opposed ideas. If you really want total message ordering you shouldn't use a network of brokers.
Related
So I'm testing RabbitMQ in one node. Plain and simple,
One producer sends messages to the queue,
Multiple consumers take tasks from that queue.
Currently consumers execute thousands of messages per second, they are too fast so I need them to slow down. Managing consumer-side throttling is not possible due to network unreliable nature.
Collectively consumers must not take more than 10 messages per second altogether from that queue.
Is there a way to configure RabbitMQ so as the queue dispatches a maximum of 10 messages per second?
If I remember correctly, once Rabbit MQ has delivered a message to the queue, it's up to consumers to consume a message. There are various consumers in different languages, you haven't mentioned anything specific, so I'm giving a generic answer.
In my understanding, you shouldn't try to impose any restrictions on Rabbit MQ itself, instead, consider implementing connection pool of message consumers that will be able to handle not more than X messages simultaneously on the client side. Alternatively, you can provide some kind of semaphore at the handler itself, but not on the Rabbit MQ server itself.
As in kafka offset is used to manage the sequence of data that needed to be transfer to consumer, how this type of management is done in RabbitMQ to manage the data sequence to prevent data loss.
RabbitMQ consumers do not maintain a client side offset like Kafka consumers must. Kafka stores all messages for a configured time period and clients manage their own offsets meaning that different clients can consume messages from different offsets within the queue. This means that different Kafka consumers are not competing consumers unless they coordinate and share their offset.
RabbitMQ is very different. Messages are stored until they get sent to a consumer and then acknowledged or they expire. If there are multiple consumers for one queue then they are competing consumers and one message can only be consumed by one of the many consumers.
RabbitMQ has the concept of a delivery tag which is a monotonically incrementing number that increments per message delivered over a channel. It has no global meaning, it has the scope of a single channel and so is not shared between consumers on different servers, processes. Consumers only need to track this number for message acknowledgement purposes, it is RabbitMQ that is responsible for choosing which message is delivered to which consumer.
Is it possible in RabbitMQ to set one consumer (i.e. Android App) as primary, but to send the message to a sencond consumer (i.e. server) when the first consumer is not acknowledging the message?
When one consumer is not ACKing the messages they get re-queued (after some timeout or when connection terminates etc) and then they are distributed to other consumer(s) in round robin manner.
Also what you may be interested in are consumer priorities.
Consumer priorities allow you to ensure that high priority consumers
receive messages while they are active, with messages only going to
lower priority consumers when the high priority consumers block.
I had two ActiveMQ brokers (A and B) that were configured as store-forward network. They work perfectly to forward messages from A to B when there is a consumer connected on broker B and producer sends messages to A. The problem is that when the consumer is killed and reconnected to A, the queued messages on B (they were forwarded from A) won't forward back to A where the consumer connected to. Even I send new messages to B, all messages were stuck on B until I restart brokers. I have tried to set networkTTL="4" and duplex="true" on the broker network connector, but it doesn't work.
Late answer, but hopefully this will help someone else in the future.
Messages are getting stuck in B because by default AMQ doesn't allow messages to be sent back to a broker to which they have previously been delivered. In the normal case, this prevents messages from going in cycles around mesh-like network topologies without getting delivered, but in the failover case it results in messages stuck on one broker and unable to get to the broker where all the consumers are.
To allow messages to go back to a broker if the current broker is a dead-end because there are no consumers connected to it, you should use replayWhenNoConsumers=true to allow forwarding messages that got stuck on B back to A.
That configuration option, some settings you might want to use in conjunction with it, and some considerations when using it, are described in the "Stuck Messages (version 5.6)" section of http://activemq.apache.org/networks-of-brokers.html, http://tmielke.blogspot.de/2012/03/i-have-messages-on-queue-but-they-dont.html, and https://issues.apache.org/jira/browse/AMQ-4465. Be sure that you can live with the side effects of these changes (e.g. the potential for duplicate message delivery of other messages across your broker-to-broker network connections).
Can you give more information on the configuration of broker A and B, as well as what you are trying to achieve?
It seems to me you could achieve what you want by setting a network of brokers (with A and B), with the producer only connecting to one, the consumer to the other.
The messages will automatically be transmitted to the other broker as long as the other broker has an active subscription to the destination the message was sent to.
I would not recommend changing the networkTTL if you are not sure of the consequences it produces (it tends to lead to unwanted messages loops).
I understand the topic is for broadcasting messages to the clients. But in the ActiveMQ document, it says about load balancing:
Another way of explaining Message Groups is that it provides sticky load balancing of messages across consumers; where the JMSXGroupID is kinda like a HTTP session ID or cookie value and the message broker is acting like a HTTP load balancer.
Here comes what I don't understand.
Is it not broadcasting?
What I want to do is below:
If producer send messages to group A, somehow the broker control the traffic and the messages "physically" should not go to group B.
Is it even possible with Virtual Topic?
VirtualTopics create queues for each topic subscriber and replicates messages to each one, so it is "broadcasting" all messages to all subscribers.
ActiveMQ message groups provide you with an exclusive consumer thread for a given message group (distinct JMSXGroupID). You can't control which consumer picks up each group and the same consumer thread can process multiple groups.
As far as I know, if you set the JMSXGroupID on a message sent to a VirtualTopic, it will arrive in each subscriber's virtual queue. However, if you have multiple threads processing from those queues (maxConcurrentConsumers > 1), then they will be single-threaded per message group, etc.
Overall, if you want certain subscribers to only process a subset of messages sent to a topic, then use message selectors to down select this subset.