I have configured a topic in ActiveMQ with lastImageSubscriptionRecoveryPolicy. It works as expected in that the broker persists the last message sent to the topic and when a new consumer subscribes to that topic it receives that last message:
producer publishes N messages
consumer A subscribes to the topic; 1 message is received
consumer B subscribes to the topic; same 1 message is received
However, the message is lost when I restart the broker. This is the sequence of events:
producer publishes N messages
broker is restarted
consumer subscribes to the topic; nothing is received
Is it possible to enable persistence of that last message (Last Image) so that it is persisted even across broker restarts?
No this is not possible. If you need message durability then you either need to use a Queue, or use a Durable Topic subscription in order to keep messages around after restart. The broker has some convenience features for Topics such as recovery policies but they are no real substitute for the stronger guarantees that exists for Queue based messaging, so if you need that then you must use those mechanisms.
Related
I have a producer and a consumer. Multiple instances of the consumer are running. When producer publishes a message, my intention is to consume the message by all the instances. So, I am using the direct exchange. Producer publishes a message to the direct exchange with a topic. Consumers are listening to that topic with the exclusive queue. This process is working fine when the consumer is up and producer publishes a message. But when consumers are down and producer publishes a message, consumers are not consuming this message when up.
I googled about the issue. A suggestion was to use named queue. But if I use named queue, messages will be consumed following the round-robin algorithm. That does not meet my expectation to consume the same message by all the consumers.
Is there any other solution?
Appreciated your help.
There are two solutions to your issue.
Using named queue is one of them.
Set your exchange in fanout mode and subscribe your named queues to it. Doing so, when a publisher send a message in your exchange, it will be dispatched to all the queues listening.
You can then have one or more consumer for each queue (allowing you to scale). You'll have to define a named queue / consumer. When one consumer disconnect, his queue still receive messages and when he comes back he can consume them.
You should be able to do what you want that way.
The other way is more for your personnal knowledge since you said you want to use RabbitMQ. But in that particular case you could use Kafkha, your consummer could then, after reconnection, resume at the message index he was when he disconnected.
Please update me if it doesn't work :)
I publish the message, it does not stay on queue. Once subscribe, the message queue starts to stay. I want to keep the message in the queue even if the user is not subscribed at all. I am using qos = 1.
It is important to remmber that MQTT is a Pub/Sub system not a Message Queuing system.
With MQTT messages will only be queued for offline clients that already have a subscription (at QOS 1 or 2), a new client subscribing to a topic will only receive new messages.
You can use the retained flag to ensure the last message (with a retained flag set) is always delivered to a client when it subscribes to a topic before new messages, but this is a single message.
this was asked to me in an interview.
what happens when message published on queue with no Subscriber at 10 AM? and a subscriber with proper filter subscribes at 10.02 AM to same Queue. Does the message gets delivered when a subscriber subscribes after the message has reached to the broker (I mean does it store in the memory until it finds the subscriber)? what is default behavior? also is it different in JMS, STOMP and AMQP standerds?
In AMQP brokers, messages are either deliver to consumers that are subscribing to queues, or when consumers fetch/pull messages from queues on demand. Messages will stay in the queue be default even if no subscriber is active at the moment. A message will be stored on disk if it's a persistent message and in memory if not persistent. https://www.rabbitmq.com/tutorials/amqp-concepts.html
STOMP: The RabbitMQ STOMP adapter supports a number of different destination types. Messages sent when no subscriber exists will be queued until a subscriber connects to the queue (while a topic will drop messages when there are no connected subscribers). https://www.rabbitmq.com/stomp.html
JMS:
JMS is an API, it doesn't use any protocol. AMQP on other hand is a protocol between a messaging client and messaging server. A JMS client can use AMQP as the protocol to communicate with the messaging server. (check this article for more information about the subject https://spring.io/understanding/AMQP)
However, messages sent to a queue remain in the queue until the message consumer for that queue consumes them.
http://docs.oracle.com/javaee/6/tutorial/doc/bnceh.html and
https://en.wikipedia.org/wiki/Java_Message_Service
As the question mentions about Publisher and Subscriber, I think the question is for Publish-Subscribe messaging pattern. In Pub/Sub pattern, publications are made to a topic, not queue.
The behavior depends on messaging provider. A messaging provider may discard a publication if there are no subscribers. So if a message was published to a topic at 10AM, the publication is discarded as there are no subscribers. Now when a new subscriber comes in 10:02AM, the publication will not be delivered to the subscriber.
There is a concept of "Retain Publication" in IBM MQ. When a publication has the "Retain Publication" attribute set, IBM MQ Queue Manager will keep a copy of such publication for a topic until a new publication is made for the same topic. Assuming a publication with "Retain Publication" is made at 10AM, when a subscriber comes at 10:02AM, the subscriber will get that publication.
Hope this helps.
Let's say that a consumer for a queue has been disconnected for some time during which many number of messages are produced.
How long does RabbitMQ keep the messages for the disconnected consumer without durable mode?
(Will it discard the queue right after the consumer is disconnected? or will it keep the queue until the memory allows?)
Does the durable mode will give a functionality for a consumer to consume any message which is published until now? (i.e. random access to the queue, fetching messages out-of-order, or consuming from the beginning of the queue)
There are some TTL extensions.
TTL can be set for a given queue by setting the x-message-ttl argument
to queue.declare, or by setting the message-ttl policy.
No it doesn't. The messages are kept in queue until they are acknowledged, regardless of durability. (unless of course the server dies, then the messages are gone if not previously marked as durable).
I've defined one topic exchange (alarms) and multiple queues, each with its own routing key:
allAlarms, with routing key alarms.#: I want this to be used for receiving all alarms in a monitoring application
alarms_[deviceID], with routing key alarms.[deviceID], where the number of devices can vary at any given time
When sending an alarm from the device, I publish it using the routing key alarms.[deviceID]. The monitoring app, however, only consumes from the allAlarms queue. This leads to the following problem:
The messages in the allAlarms queue have been consumed, while the messages in the remaining queues are ready. Is there a better way of handling messages from multiple consumers? Ideally, I'd like to be able to also send commands back to the devices using the same queues where the devices publish their alarms.
It looks like you have consumers bound to the allAlarms queue but not to any of the alarms_[deviceID] queues.
In AMQP, a single consumer is bound to a single queue by name (and each queue can have multiple consumers bound to it). Messages are delivered to the consumers of a queue in round robin such that for a given message in a queue there is exactly one consumer that will receive the message. That is, consumers cannot listen to multiple queues.
Since you're using a topic exchange, you're correctly routing a single message to multiple queues via the routing key and queue bindings. This means that you can have a consumer for each queue and when a message is delivered to the exchange, each queue will get a copy of the message and each queue will deliver the message to exactly one consumer on each queue.
Thus, if allAlarms is consuming messages, it's because it has a consumer attached to the queue. If any of the alarms_[deviceID] are not consuming messages then they must not have consumers bound to those individual queues. You have to start up consumers for each alarms_[deviceID] by name. That will allow you to also have different consumer logic for different queues.
One last thing:
Ideally, I'd like to be able to also send commands back to the devices using the same queues where the devices publish their alarms.
You don't want to do this using the same queue because there's nothing that will stop the non-device consumers on the queue from picking up those messages.
I believe you're describing RPC over RabbitMQ. For that you will want to publish the messages to the alarms queues with a reply-to header which is the name of a temporary queue. This temp queue is a single-use queue that the consumer will publish to when it's done to communicate back to the device. The device will publish to the alarms exchange and then immediately start listening to the temp queue for a response from the consumer.
For more info on RPC over RabbitMQ check out this tutorial.
I don't think you need any of the queues for the devices - the alarm_[deviceid] queues.
You don't have any consumer code set up on these queues, and the messages are backed up and waiting for you to consume them.
You also haven't mentioned a need to consume messages from these queues. Instead, you are only consuming messages form the alarmAll queue.
Therefore, I would drop all of the alarm_[deviceid] queues and only have the alarmAll queue.
Just publish the alarms through your exchange and route them all to the alarmAll queue and be done with it. No need for any other routing or queues.