I have a service that consumes messages from a RabbitMQ queue (posting to the queue is done through a topic exchange). Assuming that the service can theoretically fail and lose its state, possibility to back up all the messages for disaster recovery would come in handy.
The first idea that comes to mind is adding another binding for the topic exchange so that the messages are also posted to another queue, and creating a custom service for backing up messages that would listen on that queue. But this sounds much like a potential reinvention of the wheel. Is there a simpler way to do this with RabbitMQ (plugin/existing service/etc)?
Found out that it's possible to do with a combination of a firehose and a tracing plugin.
RabbitMQ cluster, as specified in Clustering Guide and Highly Available Queues will do what you want in the right way.
Related
I want to create a UI to see all the messages that are flowing through all exchanges in RabbitMQ server (of course other than the management console).
I am also using Mass Transit over rabbit but i am not sure if this matters.
Is this at all possible without having to code a consumer for each one of them one by one? If yes, any starting points?
The message exchanges used for publishing, as well as sending, are all bound to an exchange that has the same name as the queue for message delivery. So you could bind your own wire tap exchange on the broker to any queue exchange, and wiretap the messages to another queue of your choosing.
You can view the RabbitMQ topology layout in the documentation.
It was specifically done this way to make it easy to wiretap any endpoint, since all messages flow through a single fanout exchange.
This is a pretty broad question because it's not entirely obvious what you mean by "see", but regardless, you could create an observer on your bus. It's documented here and I think it's fairly straightforward: https://masstransit-project.com/MassTransit/usage/observers.html
In the observer you can handle various events when any message hits the MT message bus, and perform some kind of operation (like print the message, add logging, metrics, etc). If you have a microservice scenario it might be a good idea to add an observer to your shared library and add it to the bus in your individual applications.
In discussing the differences between Kafka and RabbitMQ, "dumb broker" and "smart broker" keeps popping up in their interactions with consumers. Kafka is described as having a dumb broker while RabbitMQ is said to have a smart broker/dumb consumer model.
What exactly does this mean? I'm familiar with the basics of Kafka and a little bit more about RabbitMQ. However, what features of RabbitMQ makes the broker smarter than Kafka's?
This is a question that bothered me for sometime too :) Here's what I have understood so far...
In the case of RabbitMQ the broker makes sure the messages are delivered to the consumers and dequeue them only when it gets acknowledgement from all the consumers that need that message. It also keeps track of consumer state.
Kafka does not keep track of "which messages were read by consumers". The Kafka broker keeps all messages in queues for a fixed duration of time and it's the responsibility of the consumer to read it from the queue. It also does not have this overhead operation of keeping track of consumer state.
You can read more about it in this awesome Pivotal blog post comparing RabbitMQ and Kafka.
The point about Kafka using a dumb broker while Rabbit MQ using a smart broker is one of the points used while deciding which Messaging System to use. Since RabbitMQ is a smart broker implementing global startegies for retry is far easier and listener agnostic than in Kafka.
Given a set of microservices accessed through an API gateway I believe that the above point, combined with the advantages of Rabbit MQ being much more maintainable and the knowledge that the data passed across microservices will never amount to the same load as that of streaming data, makes Rabbit MQ a far better choice than Kafka for Inter Service Communication
Dumb vs Smart broker means that the Broker can be smart to route messages based on certain conditions.
In the case of RabbitMQ, producer sends message to Exchange and Exchange routes the message to Queue. Here "Exchange" does the routing and thats what they call as Smart broker. Again people have made Brokers really smart and ended up with ESB which we all know what happened and Industry is moving away from Bloated ESB's.
In the case of Kafka, broker doesn't route messages. It is up to the user to create topics, producers partition the events into topic-partitions, and consumer groups and decide which consumer groups listens to which topic.
Smart vs Dumb broker has nothing to do with Message acknowledgment. In case of RabbitMQ, it tracks the status of each message to see whether it is consumer or not. In the case of Kafka, it happens but differently by using offsets on partitions and offset is stored in Kafka itself ( consumer can also store). But both provide the functionality.
I am putting together a queue based distributed system, all standard stuff. We are using the latest version of RabbitMQ to provide our messaging transport tier.
I have some questions regarding achieving high availability (for my applications and not actually RabbitMQ) that I couldn't answer by reading the documentation. Would appreciate some advice, it's very likely my lack of understanding of Rabbit/AMQP that is causing the problem :)
Problem: I have a message producer (called the primary). There is one and only 1 message producer. There is a secondary producer (called the backup) which should take over from the primary should it fail.
How could I achieve this using existing RabbitMQ capabilities?
Thoughts: Use an "exclusive" queue, to which the primary will be connected to. The backup will attempt to connect to to this queue. When the primary fails, the backup will gain connectivity to the queue and establish control over the process.
What is the correct pattern I should be using to achieve this? I couldn't find any documentation on competing producers etc, would appreciate your advice! How do others do this?
Kind regards
TM
If you want to have only one producer at a time - you can't afford it with RabbitMQ mechanism (unless you'll get some plugin but I don't know such of a kind). You can gain control on producers number on application level.
P.S.:
Looks like you don't get AMQP idea well, producers publish messages to exchanges, while consuming get them from queue. The broker (RabbitMQ) route messages from exchange to on or more queues (in fact, it can also route messages to other exchange, but that's another story).
I'm using NServiceBus 4.x with RabbitMQ 3.2.x as my transport.
I made the assumption that by using RabbitMQ as my transport I would be given the competing consumer model as an option. I understand that NServiceBus employs the "Fannout" exchange type for all exchanges and does not support round robin at this time. However is there a way to configure NServiceBus to take advantage of the levels of indirection via Exchanges and channels that RabbitMQ offers.
I have several consumers I would like to compete for messages from a given queue. What I am observing is subscribers' blocking access to further message retrieval from the queue until the message is consumed. So having more then one consumer at this point does me no good other then redundancy.
After reading some documentation on RabbitMQ I'm assuming that it's normal to block until the Ack receipt is sent from the subscriber. But I had assumed that subscriber #2 would have free access to the queue to fetch another message.
There is mention of increasing the prefetch count on RabbitMQ channel.
Example:
channel.BasicQos(0,prefetchcount,false)
I don't see anywhere that I can change this setting via configuration in NServiceBus. Furthermore as I read what prefetch does I'm really not sure this what i'm looking for.
Is it possible to use RabbitMQ with out a distirbutor type pattern used with MSMQ? Or should I move to MassTransit or Rebus?
Put prefetchcount=2 in your connection string. Any value above 1 will tell the broker to allow more than X unacked message to go out. You need to fiddle with this setting to find the optimum for your scenario.
Pretty new to RabbitMQ and we're still in the investigation stage to see if it's a good fit for our use cases--
We've readily come to the conclusion that our desired topology would have us deploying a few topic based exchanges, and then filtering from there to specific queues. For example, let's say we have a user and an upload exchange, where the user queue might receive messages where the topic is "new-registration" or "friend-request" and the upload exchange might receive messages like "video-upload" or "picture-upload".
Creating the queues, getting them routed to the appropriate queue, and then building listeners to handle the messages for the various queues has been quite straight forward.
What's unclear to me however is if it's possible to do a fanout on a topic exchange?
I.e. I have named queues that are bound to my topic exchange, but I'd like to be able to just throw tons of instances of my listeners at those queues to prevent single points of failure. But to the best of my knowledge, RabbitMQ treats these listeners in a straight forward round robin fashion--e.g. every Nth message always go to the same Nth listener rather than dispatching messages to the first available consumer. This is generally acceptable to us but given the load we anticipate, we'd like to avoid the possibility of hot spots developing amongst our consumer farm.
So, is there some way, either in the queue or exchange configuration or in the consumer code, where we can point our listeners to a topic queue but have the listeners treated in a fanout fashion?
Yes, by having the listeners bind using different queue names, they will be treated in a fanout fashion.
Fanout is 1:N though, i.e. each task can be delivered to multiple listeners like pub-sub. Note that this isn't restricted to a fanout exchange, but also applies if you bind multiple queues to a direct or topic exchange with the same binding key. (Installing the management plugin and looking at the exchanges there may be useful to visualize the bindings in effect.)
Your current setup is a task queue. Each task/message is delivered to exactly one worker/listener. Throw more listeners at the same queue name, and they will process the tasks round-robin as you say. With "fanout" (separate queues for a topic) you will process a task multiple times.
Depending on your platform there may be existing work queue solutions that meet your requirements, such as Resque or DelayedJob for Ruby, Celery for Python or perhaps Octobot or Akka for the JVM.
I don't know for a fact, but I strongly suspect that RabbitMQ will skip consumers with unacknowledged messages, so it should never bottleneck on a single stuck consumer. The comments on their FAQ seem to suggest that RabbitMQ will make an effort to keep things chugging along even in the presence of troublesome consumers.
This is a late answer, but in case others come across this question...
It sounds like what you want is fair dispatch rather than a fan out model (which would publish a given message to every queue).
Fair dispatch will give a message to the next available worker rather than using a simple round-robin approach. This should avoid the "hotspots" you are concerned about, without delivering the same message to multiple consumers.
If this is what you are looking for, then see the "Fair Dispatch" section on this page in the Rabbit docs. A prefetch count of 1 is the key here.