RabbitMQ 3.6.x
I have a fanout exchange with 2 queues registered. All works fine.
But silly question in pub/sub fanout model, can/should the consumer do acknowledgements?
consumers always have to acknowledgement messages!
Maybe the question is auto_ack yes or not.
But silly question in pub/sub fanout model, can/should the consumer do
acknowledgements?
Even if pub/sub usually implements auto-delete, exclusive queues with auto_ack = true, there is not a specific rule !
it depends from your application! :)
Related
RabbitMQ: version 3.11.2
I wish to configure a fanout exchange for which there will be two consumers and a single producer. Each of the two consumers can go offline for several minutes at a time; in the worst case scenario, one of the consumers could go offline for hours.
QUESTION: how should the fanout exchange and/or consumer queues be configured such that no message is ever lost, even if and when one of the consumers goes offline for several minutes or hours?
It is enough to bind queues to your exchange.
See here for more details https://www.rabbitmq.com/tutorials/tutorial-three-python.html
result = channel.queue_declare(queue='')
At this point result.method.queue contains a random queue name. For example it may look like amq.gen-JzTY20BRgKO-HjmUJj0wLg.
Secondly, once the consumer connection is closed, the queue should be deleted. There's an exclusive flag for that:
result = channel.queue_declare(queue='', exclusive=True)
channel.queue_bind(exchange='logs',
queue=result.method.queue)
Note:
If you need to handle the offline consumers, you need persistent queues and persistent messages instead of exclusive and auto-delete queues
I'm doing some research if rabbitMQ is the way to go for me.
Can I mix publish/subscribe and worker queues?
The goal is to have have a few queues which holds items of for specific process.
Each process can have a lot of workers.
The setup shown in this diagram is possible. If the exchange X is a direct or topic exchange, the 'specific process' would be represented by one of the two queues and routing of messages would be based on the routing key or topic of a message.
Consumers ('workers') would consume messages from the queues using round-robin.
In discussing the differences between Kafka and RabbitMQ, "dumb broker" and "smart broker" keeps popping up in their interactions with consumers. Kafka is described as having a dumb broker while RabbitMQ is said to have a smart broker/dumb consumer model.
What exactly does this mean? I'm familiar with the basics of Kafka and a little bit more about RabbitMQ. However, what features of RabbitMQ makes the broker smarter than Kafka's?
This is a question that bothered me for sometime too :) Here's what I have understood so far...
In the case of RabbitMQ the broker makes sure the messages are delivered to the consumers and dequeue them only when it gets acknowledgement from all the consumers that need that message. It also keeps track of consumer state.
Kafka does not keep track of "which messages were read by consumers". The Kafka broker keeps all messages in queues for a fixed duration of time and it's the responsibility of the consumer to read it from the queue. It also does not have this overhead operation of keeping track of consumer state.
You can read more about it in this awesome Pivotal blog post comparing RabbitMQ and Kafka.
The point about Kafka using a dumb broker while Rabbit MQ using a smart broker is one of the points used while deciding which Messaging System to use. Since RabbitMQ is a smart broker implementing global startegies for retry is far easier and listener agnostic than in Kafka.
Given a set of microservices accessed through an API gateway I believe that the above point, combined with the advantages of Rabbit MQ being much more maintainable and the knowledge that the data passed across microservices will never amount to the same load as that of streaming data, makes Rabbit MQ a far better choice than Kafka for Inter Service Communication
Dumb vs Smart broker means that the Broker can be smart to route messages based on certain conditions.
In the case of RabbitMQ, producer sends message to Exchange and Exchange routes the message to Queue. Here "Exchange" does the routing and thats what they call as Smart broker. Again people have made Brokers really smart and ended up with ESB which we all know what happened and Industry is moving away from Bloated ESB's.
In the case of Kafka, broker doesn't route messages. It is up to the user to create topics, producers partition the events into topic-partitions, and consumer groups and decide which consumer groups listens to which topic.
Smart vs Dumb broker has nothing to do with Message acknowledgment. In case of RabbitMQ, it tracks the status of each message to see whether it is consumer or not. In the case of Kafka, it happens but differently by using offsets on partitions and offset is stored in Kafka itself ( consumer can also store). But both provide the functionality.
I am new to RabbitMQ and trying to figure out the difference between a broker and an exchange.
From what I've read, the terms seem to almost be used interchangeably and in the diagrams, a broker seems to encompass both the exchange and the queues.
From "RabbitMQ Succinctly" book:
Exchanges are AMQP entities where messages are sent to the message
broker. Exchanges take a message and then route it to one or more
queues
So what is a broker? In the RabbitMQ management there is a tab for "Exchanges", but not for brokers. Can I interact with a broker directly or is this only done by the exchange?
"Broker" is a generic term for the type of messaging system that RabbitMQ is. It's a centralized messaging system, with a server that handles the routing and delivery of messages, etc.
This paper from ZeroMQ is good to understand the differences between brokered and brokerless: http://zeromq.org/whitepapers:brokerless (although this paper is fairly biased toward brokerless model, both are good and both have uses. i tend to prefer rabbitmq / brokered system, but not always)
From the other perspective, here is RabbitMQ's broker vs brokerless post: https://www.rabbitmq.com/blog/2010/09/22/broker-vs-brokerless/
For the most part, just substitute "rabbitmq server" in your mind, when you see the work "broker" and you'll be good to go.
The exchange, as you've noted, the thing through which you publish a message, in RabbitMQ. It handles the bindings and routing of the messages, depending on the exchange type.
A broker stands between producer(s) and consumer(s).
Here is the post-office analogy to understand the components in the Rabbitmq-based messaging system without going into the details.
An exchange is like a parcel-delivery man. A queue is the recipient of the parcel. A producer is the sender of the parcel. A set of rules that an exchange follows to deliver a parcel (that is, the message) to a queue are called "bindings". The routing key and/or the header are like the address on the parcel. The exchange determines which queue a message goes to based on the routing key/header. The producer sends messages to the exchange, not to the queue.
This entire business of accepting messages from the producers and delivering them to the consumers is what the "broker" does.
Rabbitmq is the implementation of AMQP protocol (an application-layer protocol) which is an asynchronous message-brokerage middleware.
Is there any way in RabbitMQ to have multiple consumers get the same message from the same queue?
I need to send the same message to anyone who's listening but also ensure that someone deals with it. Basically, I need the fanout functionality of an exchange combined with the basic.ack functionality of a queue. Is there any way to accomplish this in a scalable way?
If you are trying to ensure that the message is properly processed, acknowledgement already provides this capability. If your consumer is unable to process the message and does not provide an ack it will be requeued and processed again by the next available consumer. Implementing multiple competing consumers on the same queue will give you round-robin delivery, allowing the other consumers a chance for success.
How scalable this will be depends on how long it takes to process each message compared to the incoming rate, queue durability, prefetch and how many competing consumers you have on the queue.