Pretty new to RabbitMQ and we're still in the investigation stage to see if it's a good fit for our use cases--
We've readily come to the conclusion that our desired topology would have us deploying a few topic based exchanges, and then filtering from there to specific queues. For example, let's say we have a user and an upload exchange, where the user queue might receive messages where the topic is "new-registration" or "friend-request" and the upload exchange might receive messages like "video-upload" or "picture-upload".
Creating the queues, getting them routed to the appropriate queue, and then building listeners to handle the messages for the various queues has been quite straight forward.
What's unclear to me however is if it's possible to do a fanout on a topic exchange?
I.e. I have named queues that are bound to my topic exchange, but I'd like to be able to just throw tons of instances of my listeners at those queues to prevent single points of failure. But to the best of my knowledge, RabbitMQ treats these listeners in a straight forward round robin fashion--e.g. every Nth message always go to the same Nth listener rather than dispatching messages to the first available consumer. This is generally acceptable to us but given the load we anticipate, we'd like to avoid the possibility of hot spots developing amongst our consumer farm.
So, is there some way, either in the queue or exchange configuration or in the consumer code, where we can point our listeners to a topic queue but have the listeners treated in a fanout fashion?
Yes, by having the listeners bind using different queue names, they will be treated in a fanout fashion.
Fanout is 1:N though, i.e. each task can be delivered to multiple listeners like pub-sub. Note that this isn't restricted to a fanout exchange, but also applies if you bind multiple queues to a direct or topic exchange with the same binding key. (Installing the management plugin and looking at the exchanges there may be useful to visualize the bindings in effect.)
Your current setup is a task queue. Each task/message is delivered to exactly one worker/listener. Throw more listeners at the same queue name, and they will process the tasks round-robin as you say. With "fanout" (separate queues for a topic) you will process a task multiple times.
Depending on your platform there may be existing work queue solutions that meet your requirements, such as Resque or DelayedJob for Ruby, Celery for Python or perhaps Octobot or Akka for the JVM.
I don't know for a fact, but I strongly suspect that RabbitMQ will skip consumers with unacknowledged messages, so it should never bottleneck on a single stuck consumer. The comments on their FAQ seem to suggest that RabbitMQ will make an effort to keep things chugging along even in the presence of troublesome consumers.
This is a late answer, but in case others come across this question...
It sounds like what you want is fair dispatch rather than a fan out model (which would publish a given message to every queue).
Fair dispatch will give a message to the next available worker rather than using a simple round-robin approach. This should avoid the "hotspots" you are concerned about, without delivering the same message to multiple consumers.
If this is what you are looking for, then see the "Fair Dispatch" section on this page in the Rabbit docs. A prefetch count of 1 is the key here.
Related
I have a hard time understanding the basic concepts of RabbitMQ. I find the online documentation not perfectly clear.
So far I understand, what a channel, a queue, a binding etc. is.
But how would the following use case be implemented:
Use Case: Sender posts to one exchange with different topics. On the receiver side, depending on the topic, different receivers should be notified.
So the following should somehow be feasible with a topic exchange:
create a channel
within this channel, create a topic exchange
for each topic to be subscribed to, create a queue and a queue binding with this topic as property
My difficulty is that the callback would be related to the channel, not to the queue or the queue binding. I am not 100 % sure if I am right here.
So that's my question: in order to have multiple callbacks, IOW: different message handlers, depending on the subscribed topic - do you have to create multiple channels, one for each "different message handling"? All these channels should grab the same exchange and define their own queue + queue binding for that specific topic?
Please confirm if this is correct or if I am straying from the canonic path of AMPQ ... "queue" sounds so light-weight, so I intuitively thought of a queue or a queue binding as the right point to attach a consuming event handler to, but it seems that, instead, channel is my friend in this. Right?
Another aspect of my question:
If I really have to use multiple channels for this, do I have to declare the same exchange (exchange name and exchange type of "topic") for each channel? I hoped there was something like:
define the exchange with this name and the type of "topic" once
for each channel, "grab" this predefined exchange and use it by adding queues and queue bindings to this exchange
I find it helpful to think about the roles of the broker (RabbitMQ) and the clients (your applications) separately.
The broker, RabbitMQ, will receive messages from your publishers, route them to queues, and eventually send them to consumers. The message routing can be simple or complex. In your case, the routing is topic based with a few different queues.
You haven't said much about the publishers, likely because their job is simple. They send messages with a routing key to RabbitMQ.
The consumer side is where things can get interesting. At the simplest level, a consumer subscribes to a queue, receives messages from RabbitMQ, and processes them. The consumer opens a connection to RabbitMQ and will use a channel for a particular use (e.g., subscribing to a queue). The power of message brokers is that they allow designers to break up processes into separate apps if desired.
You don't give much insight into your application, other than the presence of different message topics. An important design choice for you to make is how to define the application(s). Are the different topics suitable for separate applications, or will a single application handle all types of messages.
For the former case, you would have one application for each queue. A single channel that subscribes to the queue is probably the most sensible decision unless your application needs to be threaded. For threaded applications, each thread would have its own channel and all threads can be subscribed to the same queue. Each application would have its own callback function for processing that type of message.
For the latter case (single application with multiple queues), the best approach would be to have at least one channel per queue. It sounds like each queue would require its own callback function, and you would assign the functions to the channels according to its subscription. You might have multiple channels per queue if your application can process multiple messages (of each topic) simultaneously.
Regarding your question about declaring exchanges, queues, and bindings, these items only need to be created once. But it is reasonable practice to have your clients declare them at connection time. Advantages of declaring them are that they will be created again if they were deleted and that any discrepancies between your declaration and what is on the broker will trigger errors.
I have an exchange that's going to receive roughly 50 messages per second. These messages have a unique identifier which relates to each unit in the field. This unique identifier will be the routing key. Every now and again we need to debug or analyse a unit. At that point in time we will spin up a queue, with the correct routing key, and bind it to the exchange. This way, that queue will start receiving the messages for that unit and any consumers monitoring that queue, will then receive the messages.
What this does mean is that 99% of the time, the exchange will have no queues and no routing key. Then, every now and again a queue and routing key will be created and subscribe.
It feels kind of wasteful to be sending 50 messages per second at an exchange, when its just going to immediately discard them. That said, it feels like this how RabbitMQ exchanges are supposed to be used. I guess from a developer perspective i feel like this is wasteful but I also think my understanding of rabbit says that this is the correct way to do.
Is there any overhead to doing this? Any performance concerns I should have? or maybe I am approaching this entirely wrong?
I did try to search before asking but nothing really describes a scenario where an exchange has no queue or routing key, but is still receiving messages.
This is basically how RabbitMQ works, as you have described. The broker is not responsible for how often and how many events you decide to publish. It will nonetheless protect from too much pressure. It has a credit based flow control mechanism. RabbitMQ flow control.
RabbitMQ has different ways in which unroutable messages can be handled.Unroutable Message Handling How to deal with unroutable messages
To sum up a bit the information you will find on those links:
If the publisher does not set the message as mandatory, it will either be discarded or republished to a different alternate exchange that you can configure. This only makes sense if you want to persist all unroutable messages regardless of the source in a single queue, that you can handle later.
If the publisher sets the message as mandatory, the message will be returned to the publisher and the publisher can have a returned message handler setup in order to handle those events.
These strategies in addition to the flow control mechanism, also assure RabbitMQ reliability and protection.
In your situation if you want to limit the messages from producer even more, you need to create a mechanism, as an example, so the producer will not start publishing only when a consumer becomes active. So basically the consumer process will communicate the producer process that it is active and it can start publishing. But from my experience I don't think it's worth the overhead, at least at first, because 50 messages per seconds isn't much. You can monitor the RabbitMQ server and check how is the resource consumption to check if you need to optimize, at first. Optimization is best done with metrics and understanding.
So we decided to go with RabbitMQ as a message/event bus in our migration to micro-services architecture, but we couldn't find a definite answer on what is the best way of putting our queues, we have two options to go with:
One main exchange which will be a Fanout exchange, which in turn will fan messages out to a main queue for logging and other purposes and another sub exchange which will be a topic exchange and route the messages to each desired queue using the message routing key. We expect the number of queues behind the sub-exchange to be some how a large number. This can be explained by this graph:
One main exchange, which will be a Topic exchange, with still one main queue bound to that exchange using "#" routing key. That main exchange will also handles main routing to other sub exchanges, so routing keys might be "agreements.#", "assignments.#", "messages.#", which are then used to bind multiple topic sub-exchanges, each will handle sub routing, so one sub exchange might be handling all "assignments" and queues bound to that exchange could be bound by routing keys like "assignments.accepted", "assignments.deleted"...In this scenario, we feel like the huge number of queues will be less per exchange, they will be somehow distributed between exchanges.
So, which of these scenarios could be the best approach? Faster on RabbitMQ, less overhead.
Taking in mind, all queues, exchanges and bindings will be done on the fly from the service which will be either publishing or subscribing.
You can find some explanation in this topic: RabbitMQ Topic exchanges: 1 Exchange vs Many Exchanges
I am using RabbitMQ in a very similar way that you showed in the case 2, as I found the same benefits as described in this article: https://skillachie.com/2014/06/27/rabbitmq-exchange-to-exchange-bindings-ampq/
Exchange-to-exchange bindings are much more flexible in terms of the topology that you can design, promotes decoupling & reduce binding churn
Exchange-to-exchange bindings are said to be very light weight and as a result help to increase performance *
Based in my own experience with exchange-to-exchange, the case 2 is great and it will allows to create/change messages flow topologies in a very fast way.
I'm going to first re-summarize what I think is your question, since I'm sure it's buried somewhere in your post.
It is desirable to have a tracer/logging queue, in addition to a series of work-specific queues for actual message processing. What exchange topology is best for this scenario?
First off, neither option makes much sense given your application. Option 1 will create an exchange that will publish a message to every queue bound to it, regardless. This is clearly not what you want. Option 2 will give you a rather complex routing topology for which the benefit is unclear, and the drawback is painful maintenance and a steep learning curve. (Just because you can do something does not mean you should do it.)
What should be done?
It is important to remember that in RabbitMQ, it is the queues which consume the resources of the broker. Exchanges merely connect queues with publishers. The exchange is a means to an end, while the queue is the end itself.
What instead I think you should do is set up a single topic exchange. Bind your tracing queue to routing key # so that you receive all messages. Then, bind your work queues appropriately so that they receive only the messages that need to flow into them. For example, it is common to route messages by message type, where each queue holds exactly one type of message. This is both simple and effective.
The advantage of a single topic exchange is that you get the benefits of both a Direct Exchange and a Fanout Exchange depending on the binding key used. Further, configuration changes are easy to achieve and can often be done without disrupting any system processing at all (let's say that you want to stop tracing certain messages - this can be done with ease using a topic exchange, assuming your routing keys are rational).
Exchange-to-exchange bindings is semantically identical to exchange-to-queue bindings.
https://www.rabbitmq.com/e2e.html
I need a way to publish messages to unknown number of subscribers. The messages should be durable/persisted and categorized into three priorities (high, medium and low). One of the subscribers can only handle a limited load and some messages are just more important. High-prioritized messages processed first etc.
How do I do that with Rebus? I guess I need three queues per subscriber?
Where can I find a publish/subscribe example with durable queues and MSMQ?
First, some info: Rebus likes to work with durable queues, durable messaging, and guaranteed delivery. In fact, unless you actively do stuff to opt out, that's the way everything works. So if you manage to make pub/sub work with Rebus, it's durable :)
Publishing by definition works with an "unknown number of subscribers" - at least that's a bus concern, and not an application concern.
In reality, subscribers initiate pub/sub conversation by issuing a SubscriptionMessage (which can be seen as a subscription request), which is then followed by the publisher publishing some number of events (which can be seen as "subscription replies"). The "bus part" of the publisher keeps track of who subscribed to any given event type.
So far, so good.
Regarding priorities, there's no out-of-the-box way to achieve that with Rebus. One way to ensure a maximum latency on certain message types is, as you're suggesting, by making separate endpoints whose input queues will not be clogged by low priority messages.
But there is some stuff around how Rebus is configured that strongly suggests having only one single input queue in each process, so that would probably imply that you should create separate processes that subscribe to those high priority message types.
I know that MSMQ supports some kind of priority on messages, so I guess it could be supported by having MsmqMessageQueue understand certain headers (similar to how express delivery and time-to-be-received are implemented - see here) - pull requests are happily accepted and strongly encouraged :)
I have found this image is very similar to my bussiness model. I need to split message to some queue.
for some heavy work. I can add more worker thread for them. But for some no much heavy work. I can
let single consumer to subscribe their message. But how to do that in rabbitMQ.
Through their document. I just found that single-queue-multi-consumer model.
You can add multiple workers to a queue
There can be multiple queues bound to an exchange.
In RabbitMQ, the producer always sends the message to an exchange. So, in your case, I hope only one exchange is enough. If you want to load balance at the consumer side, you have the above said two options.
You can also read my article:
https://techietweak.wordpress.com/2015/08/14/rabbitmq-a-cloud-based-message-oriented-middleware/
RabbitMQ has a very flexible model, which enables a wide variety of routing scenarios to take place.
I need to split message to some queue. for some heavy work. I can add more worker thread for them.
Yes, this is supported via a direct exchange. Publish a message using a routing key that is the same as the name of the queue. For convenience, let's say you use the fully-qualified object name (e.g. MyApp.Objects.DataTypeOne). All you need to do is subscribe multiple consuming processes to this queue, and RabbitMQ will load-balance using a round-robin approach.
But for some no much heavy work. I can let single consumer to subscribe their message.
Yes, you can do this also. Same process as in the paragraph above. Just don't attach multiple consuming processes.
I have found this image is very similar to my business model.
The diagram isn't very useful, because it lacks information about the type of messages being published. In that sense, it is only an interconnect diagram. The interesting lines are the ones connecting the queues to the exchange, as that is what you specify within RabbitMQ via Queue Bindings. You can also bind exchanges to one another, but that's a bit further than we probably need to go.
Everything else on the diagram is fully under your control as the user of the RabbitMQ/AMQP system. You can create an arbitrary number of publishers and have an arbitrary number of consuming processes each consuming from an arbitrary number of queues. There are no hard and fast limits, though there are some practical aspects you probably will want to think about to ensure your system is maintainable.