Producer sends messages rabbitmq, and the consumer receives messages from rabbitmq, then consumer send messages back to producer via rabbitmq.
Is it possible?
Are you trying to implement an RPC type of system, or do you just want to know how to send messages in both directions?
The basic pipe is unidirectional. You cannot send messages from the consumer to the producer through the same queue that the consumer received messages from the producer.
If you want send messages the other way, your consumer will need to be a producer as well, and your producer will need to be a consumer as well.
Yes its possible, but the producer will have to listen to a queue as well on which the client will publish/produce... , you can send the queueName/routingkey(of the producer) in the first msg sent from producer to client.. the client then after recieving the msg with the routingkey inside can send msg to original producer
Producer needs another queue to get the response back from Consumer, so called callback queue. Producer can send request with reply_to and correlation_id. Then, Consumer can utilize reply_to as the routing key to point to the correct callback queue and Producer can dequeue the callback queue and match the correlation_id.
Related
I'm trying to understand how message acknowledgement works exactly under the AMQP protocol (specifically RabbitMQ) with a direct exchange with multiple consumers subscribed to the same routing key. It is essentially a fanout exchange, but I have it so it can fan out to different consumers based on the routing_key. My current mental model looks like this:
Publisher creates "reply_to" queue and publishes to routing key with a message telling consumers to send response to queue (RPC protocol), along with a correlation id which is passed back so that all future results are tied to that unique identifier
Exchange sends out message to all queues bound to that routing key. Here, there are two queues for two consumers, each bound to routing key "pumps"
After some time, the consumer replies back to the reply_to queue, and then acknowledges message so that THEIR EXCLUSIVE QUEUE deletes the message that was sent to its queue. Each consumer that received a message does this.
The broker sends the responses to the RPC queue. The publisher acknowledges each message it gets, acknowledging messages it receives for that
I know its confusing.. basically it comes down to this question - what is a message bound to? It is obvious in a round-robin scenario. Each message is being sent to one queue, and the consumer can ack it; however, if there are multiple consumers for the same message, it made sense to me that really, each queue (and each consumer bound to it) has its own message to the consumer, each of which must be acknowledged. Is that the case?
RabbitMQ has this to say:
https://www.rabbitmq.com/confirms.html#acknowledgement-modes
Depending on the acknowledgement mode used, RabbitMQ can consider a message to be successfully delivered either immediately after it is sent out (written to a TCP socket) or when an explicit ("manual") client acknowledgement is received.
Unfortunately, this mentions nothing about queues, and what happens when there are multiple of them with their own consumers.
With RabbitMQ, for true FanOut method, its best for each consumer to have their own queue bind to the exchange, with this each consumer will receive and consume the message without affecting other consumers
with a scenario like, Sale Rep sends orders to Cashiers, where there are multiple sale reps and multiple cashiers.
Sale rep sends order
Channel.ExchangeDeclare(exchange: "cashieradd", ExchangeType.Fanout);
var jsonResult = JsonConvert.SerializeObject(new CashierQueue()
{
transactionId = transactionId
});
var body = Encoding.UTF8.GetBytes(jsonResult);
Channel.BasicPublish(exchange: "cashieradd", routingKey: "", basicProperties: null, body: body);
Each cashier would subscribe to the exchange
{
var cashierAddQueue = Channel.QueueDeclare().QueueName;
Channel.QueueBind(queue: cashierAddQueue, exchange: "cashieradd", routingKey: "");
var consumer = new EventingBasicConsumer(Channel);
Channel.BasicConsume(queue: cashierAddQueue, autoAck: true, consumer: consumer);
return consumer;
}
this uses the RabbitMQ Dynamic Queue, however, any queue unique to the consumer would have the same effect.
A routingkey here is not necessarily required
I have a producer and a consumer. Multiple instances of the consumer are running. When producer publishes a message, my intention is to consume the message by all the instances. So, I am using the direct exchange. Producer publishes a message to the direct exchange with a topic. Consumers are listening to that topic with the exclusive queue. This process is working fine when the consumer is up and producer publishes a message. But when consumers are down and producer publishes a message, consumers are not consuming this message when up.
I googled about the issue. A suggestion was to use named queue. But if I use named queue, messages will be consumed following the round-robin algorithm. That does not meet my expectation to consume the same message by all the consumers.
Is there any other solution?
Appreciated your help.
There are two solutions to your issue.
Using named queue is one of them.
Set your exchange in fanout mode and subscribe your named queues to it. Doing so, when a publisher send a message in your exchange, it will be dispatched to all the queues listening.
You can then have one or more consumer for each queue (allowing you to scale). You'll have to define a named queue / consumer. When one consumer disconnect, his queue still receive messages and when he comes back he can consume them.
You should be able to do what you want that way.
The other way is more for your personnal knowledge since you said you want to use RabbitMQ. But in that particular case you could use Kafkha, your consummer could then, after reconnection, resume at the message index he was when he disconnected.
Please update me if it doesn't work :)
I've defined one topic exchange (alarms) and multiple queues, each with its own routing key:
allAlarms, with routing key alarms.#: I want this to be used for receiving all alarms in a monitoring application
alarms_[deviceID], with routing key alarms.[deviceID], where the number of devices can vary at any given time
When sending an alarm from the device, I publish it using the routing key alarms.[deviceID]. The monitoring app, however, only consumes from the allAlarms queue. This leads to the following problem:
The messages in the allAlarms queue have been consumed, while the messages in the remaining queues are ready. Is there a better way of handling messages from multiple consumers? Ideally, I'd like to be able to also send commands back to the devices using the same queues where the devices publish their alarms.
It looks like you have consumers bound to the allAlarms queue but not to any of the alarms_[deviceID] queues.
In AMQP, a single consumer is bound to a single queue by name (and each queue can have multiple consumers bound to it). Messages are delivered to the consumers of a queue in round robin such that for a given message in a queue there is exactly one consumer that will receive the message. That is, consumers cannot listen to multiple queues.
Since you're using a topic exchange, you're correctly routing a single message to multiple queues via the routing key and queue bindings. This means that you can have a consumer for each queue and when a message is delivered to the exchange, each queue will get a copy of the message and each queue will deliver the message to exactly one consumer on each queue.
Thus, if allAlarms is consuming messages, it's because it has a consumer attached to the queue. If any of the alarms_[deviceID] are not consuming messages then they must not have consumers bound to those individual queues. You have to start up consumers for each alarms_[deviceID] by name. That will allow you to also have different consumer logic for different queues.
One last thing:
Ideally, I'd like to be able to also send commands back to the devices using the same queues where the devices publish their alarms.
You don't want to do this using the same queue because there's nothing that will stop the non-device consumers on the queue from picking up those messages.
I believe you're describing RPC over RabbitMQ. For that you will want to publish the messages to the alarms queues with a reply-to header which is the name of a temporary queue. This temp queue is a single-use queue that the consumer will publish to when it's done to communicate back to the device. The device will publish to the alarms exchange and then immediately start listening to the temp queue for a response from the consumer.
For more info on RPC over RabbitMQ check out this tutorial.
I don't think you need any of the queues for the devices - the alarm_[deviceid] queues.
You don't have any consumer code set up on these queues, and the messages are backed up and waiting for you to consume them.
You also haven't mentioned a need to consume messages from these queues. Instead, you are only consuming messages form the alarmAll queue.
Therefore, I would drop all of the alarm_[deviceid] queues and only have the alarmAll queue.
Just publish the alarms through your exchange and route them all to the alarmAll queue and be done with it. No need for any other routing or queues.
I have a producer and broker on the same machine. The producer sends messages like so:
channel = connection.createChannel();
//Create a durable queue (if not already present)
channel.queueDeclare(merchantId, true, false, false, null);
//Publish message onto the queue
channel.basicPublish("", consumerId, true, false,
MessageProperties.MINIMAL_PERSISTENT_BASIC, "myMessage");
The consumer sits on another machine and listens to messages. It uses explicit acknowledgement like so:
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
//Handle message here
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
From what I understand, the ack is meant for the broker to dequeue the message.
But how can my producer come to know about the ack that the consumer sent?
Producers and consumers normally don't interact. This is by AMQP protocol design. For example, consuming a specific message may be done a long time after it was published, and there is no sense in leaving the producer up and running for a long time. Another example is when a publisher sends one message to a broker, and due to routing logic that message gets duplicated to more than one queue, leading to ambiguity (because multiple consumers can acknowledge the same message). AMQP protocol is asynchronous (mostly), and letting the publisher know about its message being consumed just doesn't fit the AMQP async model.
There are exceptions from that, notably, RPC calls. Then the producer becomes a producer-consumer. It sends a message and then immediately waits for a reply (there is a good RabbitMQ manual - Direct reply-to related to RPC with RabbtiMQ).
In general, you can ensure that a message is delivered to a broker with Confirms (aka Publisher Acknowledgements) alongside with Dead Letter Exchanges and Alternate Exchanges. Those cover most cases under which a message can be lost from its normal flow.
In ActiveMQ, I was sending a message to a consumer, the consumer then forwards the message to a different process. I wanted to know if there is any way by which the acknowledgment can be send to the broker from the other process??
I tried sending the Message Object using a socket connection to the other process and then calling the acknowledge() method on it, it is not working.
I tried the sending the message to some other class object(in the same JAVA process) and then calling the acknowledge() method, it worked.
I guess it depends on how you are sending the message to the other process...I'd just call acknowledge() in first consumer after the call to deliver it to the other process...that should guarantee that its been delivered (assuming your delivery to the second process is sound)...