(Picking up from a Github Issue)
We use RabbitMQ's consistent hash exchange which is useful to shard routing keys from an exchange to multiple queues.
We use the queues to dish out work to workers. Ideally we would like to dynamically scale these workers but this presents issues.
To scale up, you need to add a new queue and binding. On it's own this is not a huge deal - except for that fact that a sharded key may now start going to a different queue.
So Worker A may have been managing Thing1, but now as we add the new queue Worker B may end up getting messages for Thing1. It's important that Worker A has finished all of it's Thing1 processing, before Worker B starts getting Thing1 messages.
Are they any approaches or plugins that can ease this issue?
In this case I would use a worker which dispatches the messages instead of hash exchange.
producer1 ... producern => topic-exchange => queue => dispatcher worker => queue1 ... queuen => worker1 ... workern
This way the dispatcher worker can keep track of all messages. For this you can simple check how many messages are left within the queue or you can check the worker confirm message or you can use rabbitmqs RPC features.
Related
I have multiple producers that publish to their specific (durable and limited) queues using the amq.direct exchange and particular routing key
Queues:
producer.06
producer.07
...
Routing keys:
"producer.06" -> producer.06
"producer.07" -> producer.07
...
I also have multiple consumers. When they connect, they create their own (exclusive) queue and routing keys to receive all the messages from the queues that are of interest to them. This way multiple consumers can get the same messages.
Queues:
consumer.a
consumer.b
...
Routing keys:
"producer.06" -> consumer.a
"producer.06" -> consumer.b
"producer.07" -> consumer.b
...
I would like to populate the consumer's queue with a snapshot of messages of the relevant producer's queues, prior to binding the routing keys. Loosing a few messages in the interval between the message copy and routing key binding is acceptable, and a better alternative than out-of-order messages for my application. The consumer should not remove messages from producer's queues (as they would be needed by other consumers).
Is there a way to achieve this? -copying a snapshot of a queue into another one- or does anyone has a suggestion on how to achieve this?
I am running RabbitMQ 3.8.4 on Erlang 23.0.2, and using Rabbit .Net client 6.0.0.0 for the consumers.
I'm doing some research if rabbitMQ is the way to go for me.
Can I mix publish/subscribe and worker queues?
The goal is to have have a few queues which holds items of for specific process.
Each process can have a lot of workers.
The setup shown in this diagram is possible. If the exchange X is a direct or topic exchange, the 'specific process' would be represented by one of the two queues and routing of messages would be based on the routing key or topic of a message.
Consumers ('workers') would consume messages from the queues using round-robin.
We're seeing an issue where consumers of our message queues are picking up messages from queues at the top of the alphabetical range. We have two applications: a producer, and a subscriber. We're using RabbitMQ 3.6.1.
Let's say that the message queues are setup like so:
Our first application, the producer, puts say 100 messages/second onto each queue:
Our second application, the subscriber, has five unique consumer methods that can deal with messages on each respective queue. Each method binds to it's respective queue. A subscriber has a prefetch of 1 meaning it can only hold one message at a time, regardless of queue. We may run numerous instances of the subscriber like so:
So the situation is thus: each queue is receiving 100 msg/sec, and we have four instances of subscriber consuming these messages, so each queue has four consumers. Let's say that the consumer methods can deal with 25 msg/sec each.
What happens is that instead of all the queues being consumed equally, the alphabetically higher queues instead get priority. It's seems as though when the subscriber becomes ready, RabbitMQ looks down the list of queues that this particular ready channel is bound to, and picks the first queue with pending messages.
In our situation, A_QUEUE will have every message consumed. B_QUEUE may have some consumed in certain race conditions, but C_QUEUE/D_QUEUE and especially E_QUEUE will rarely get touched.
If we turn off the publisher, the queues will eventually drain, top to bottom.
Is it possible to configure either RabbitMQ itself or possibly even the channel to use some sort of round robin distribution policy or maybe even random policy so that when a channel has numerous bound queues, all with messages pending, the distribution is even?
to clarify: you have a single subscriber application with multiple consumers in it, right?
I'm guessing you're using a single RabbitMQ Connection within the subscriber app.
Are you also re-using a single RabbitMQ Channel for all of your consumers? If so, that would be a problem. Be sure to use a new Channel for each consumer you start.
Maybe the picture is wrong, but if it's not then your setup is wrong. You don't need 4 queues if you are going to have subscribers that listen to each and every queue. You'd just need one queue, that has multiple instances of the same subscriber consuming from it.
Now to answer, yes (but no need to configure, as long as prefetch is 1), actually rabbitmq does distribute messages evenly. You can find about about that here, and on the same place actually how your setup should look like. Here is a quote from the link.
RabbitMQ just dispatches a message when the message enters the queue.
It doesn't look at the number of unacknowledged messages for a
consumer. It just blindly dispatches every n-th message to the n-th
consumer.
I'm new to RabbitMQ and I'm wondering how to implement the following: producer creates tasks for multiple sites, there's a bunch of consumers that should process these tasks one by one, but only talking to 1 site with concurrency of 1, without starting a new task for this site before the previous one ended. This way slow site would be processed slowly, and the fast ones - fast (as opposed by slow sites taking up all the worker capacity).
Ideally a site would be processed only by one worker at a time, being replaced by another worker if it dies. This seems like a task for exclusive queues, but apparently there's no easy way to list and subscribe to new queues. What is the proper way to achieve such results with RabbitMQ?
I think you may have things the wrong way round. For workers you have 1 or more producers sending to 1 exchange. The exchange has 1 queue (you can send directly to the queue, but all that is really doing is going via a default exchange, I prefer to be explicit). All consumers connect to the single queue and read off tasks in turn. You should set the queue to require messages to be ACKed before removing them. That way if a process dies it should be returned to the queue and picked up by the next consumer/worker.
I have a Topic exchange from which I'd like to distribute messages to two queues on two servers part of a cluster, in order to reduce memory pressure on any particular server. My consumers are periodically slow, and I sometimes run into the high memory watermark.
The way I tried to resolve this is by routing messages using an intermediate direct exchange, with two queues bound to the exchange:
a (topic) -> a1 (direct) -> q1/q2 (bound to routing key "a")
But the messages were routed to both queues, as AMQP intends. Anyone has ideas? What I need is an exchange that routes to one and only one queue, even if the routing key matches many queues. I'd prefer not to change my routing keys, but that could be arranged.
I found Selective routing with RabbitMQ, which may mean I'll need to implement my own routing logic. Hopefully, this already exists somewhere else.
You could perhaps use the Shovel plugin - http://www.rabbitmq.com/shovel.html - to move messages from your intermediate exchange to the two queues.
If you set up two shovels, both consuming from a single queue on the direct intermediate exchange, they should be able to fight over the messages coming in (I'm assuming that you don't care too much if the two recipient queues don't get the incoming messages in a strict round robin fashion). The shovels then each publish to one of the two end queues, and can send through the ACKs from the end consumer.