this was asked to me in an interview.
what happens when message published on queue with no Subscriber at 10 AM? and a subscriber with proper filter subscribes at 10.02 AM to same Queue. Does the message gets delivered when a subscriber subscribes after the message has reached to the broker (I mean does it store in the memory until it finds the subscriber)? what is default behavior? also is it different in JMS, STOMP and AMQP standerds?
In AMQP brokers, messages are either deliver to consumers that are subscribing to queues, or when consumers fetch/pull messages from queues on demand. Messages will stay in the queue be default even if no subscriber is active at the moment. A message will be stored on disk if it's a persistent message and in memory if not persistent. https://www.rabbitmq.com/tutorials/amqp-concepts.html
STOMP: The RabbitMQ STOMP adapter supports a number of different destination types. Messages sent when no subscriber exists will be queued until a subscriber connects to the queue (while a topic will drop messages when there are no connected subscribers). https://www.rabbitmq.com/stomp.html
JMS:
JMS is an API, it doesn't use any protocol. AMQP on other hand is a protocol between a messaging client and messaging server. A JMS client can use AMQP as the protocol to communicate with the messaging server. (check this article for more information about the subject https://spring.io/understanding/AMQP)
However, messages sent to a queue remain in the queue until the message consumer for that queue consumes them.
http://docs.oracle.com/javaee/6/tutorial/doc/bnceh.html and
https://en.wikipedia.org/wiki/Java_Message_Service
As the question mentions about Publisher and Subscriber, I think the question is for Publish-Subscribe messaging pattern. In Pub/Sub pattern, publications are made to a topic, not queue.
The behavior depends on messaging provider. A messaging provider may discard a publication if there are no subscribers. So if a message was published to a topic at 10AM, the publication is discarded as there are no subscribers. Now when a new subscriber comes in 10:02AM, the publication will not be delivered to the subscriber.
There is a concept of "Retain Publication" in IBM MQ. When a publication has the "Retain Publication" attribute set, IBM MQ Queue Manager will keep a copy of such publication for a topic until a new publication is made for the same topic. Assuming a publication with "Retain Publication" is made at 10AM, when a subscriber comes at 10:02AM, the subscriber will get that publication.
Hope this helps.
Related
I'm having a little trouble figuring if Rabbit MQ can publish a message to a single queue with multiple subscribers, where the message will not get deleted until all subscribers to that queue have gotten the message.
The closest I can find is https://www.rabbitmq.com/tutorials/amqp-concepts.html, where it states:
AMQP 0-9-1 has a built-in feature called message acknowledgements (sometimes referred to as acks) that consumers use to confirm message delivery and/or processing. If an application crashes (the AMQP broker notices this when the connection is closed), if an acknowledgement for a message was expected but not received by the AMQP broker, the message is re-queued (and possibly immediately delivered to another consumer, if any exists).
Does this mean if the queue has more than one subscriber, it will wait until the message is consumed by all subscribers?
You should use multiple queues bound to the same exchange, using the same binding. Then, when a message matches the binding, it will be delivered to all queues, which presumably each have a consumer.
If you have multiple consumers on a single queue, RabbitMQ will round-robin deliveries among those consumers (which is not what you want).
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I have configured a topic in ActiveMQ with lastImageSubscriptionRecoveryPolicy. It works as expected in that the broker persists the last message sent to the topic and when a new consumer subscribes to that topic it receives that last message:
producer publishes N messages
consumer A subscribes to the topic; 1 message is received
consumer B subscribes to the topic; same 1 message is received
However, the message is lost when I restart the broker. This is the sequence of events:
producer publishes N messages
broker is restarted
consumer subscribes to the topic; nothing is received
Is it possible to enable persistence of that last message (Last Image) so that it is persisted even across broker restarts?
No this is not possible. If you need message durability then you either need to use a Queue, or use a Durable Topic subscription in order to keep messages around after restart. The broker has some convenience features for Topics such as recovery policies but they are no real substitute for the stronger guarantees that exists for Queue based messaging, so if you need that then you must use those mechanisms.
I have a producer and a consumer. Multiple instances of the consumer are running. When producer publishes a message, my intention is to consume the message by all the instances. So, I am using the direct exchange. Producer publishes a message to the direct exchange with a topic. Consumers are listening to that topic with the exclusive queue. This process is working fine when the consumer is up and producer publishes a message. But when consumers are down and producer publishes a message, consumers are not consuming this message when up.
I googled about the issue. A suggestion was to use named queue. But if I use named queue, messages will be consumed following the round-robin algorithm. That does not meet my expectation to consume the same message by all the consumers.
Is there any other solution?
Appreciated your help.
There are two solutions to your issue.
Using named queue is one of them.
Set your exchange in fanout mode and subscribe your named queues to it. Doing so, when a publisher send a message in your exchange, it will be dispatched to all the queues listening.
You can then have one or more consumer for each queue (allowing you to scale). You'll have to define a named queue / consumer. When one consumer disconnect, his queue still receive messages and when he comes back he can consume them.
You should be able to do what you want that way.
The other way is more for your personnal knowledge since you said you want to use RabbitMQ. But in that particular case you could use Kafkha, your consummer could then, after reconnection, resume at the message index he was when he disconnected.
Please update me if it doesn't work :)
I'm working for a company where we're considering Mule ESB. We would need to set up Mule in a clustered configuration to get what Mule coins a Mule High Availability (HA) Cluster.
Now, we need to persist incoming messages to a queue in case of power outage or disk failure. As far as I understand, we can either go with the default Mule Object Store which "persists" messages to a shared memory grid. However, my first thought here is that this can't be any good if we get a power outage which takes the entire cluster out of action.
Our other option is to use a separate queue product such as RabbitMQ or ActiveMQ. However, do these integrate alright with a HA cluster? Are there any mechanism in these products which ensures that the same message won't be picked up by two machines at the same time?
Consider this scenario (based on the observer pattern):
Mule receives a message, puts it on a queue and responds with an OK
to the client which delivered the message.
Mule picks up a message from the queue, and attempts to deliver it to a subscriber.
The subscriber accepts the message, and Mule removes it from the queue.
What happens if another Mule instance in the HA cluster attempts to pick up the message between 2 and 3 above? Is there a mechanism where Mule can indicate that a message is picked up from the queue to be "attempted delivered" but then, if the delivery fails, update the message on the queue as "not delivered" if delivery fails?
Both RabbitMQ and ActiveMQ will give you the once-and-only-once functionality I think you are looking for.
Both platforms ensure that each message in a queue is received by only one subscriber.
In ActiveMQ, to return a message to a queue in the event of a failure, you can use explicit message acknowledgement or JMS transactions. Here's a quick overview.
In RabbitMQ, you do it using acknowledgements.
Also, you might want to consider reliability for your message broker. Both ActiveMQ and RabbitMQ offer highly available broker configuration options.
I need to build a system that uses a Publish/Subscribe bus (e.g. Mule, ZeroMQ, RabbitMQ), but the literature all implies that subscriber applications are reliably available to receive messages from topics to which they subscribe as soon as the Pub/Sub bus is able to deliver the message.
I have a system where some of the applications will be reliably connected to the Publish/Subscribe bus, but other applications will not be active or connected to the bus all the time.
The obvious solution is to have some sort of "presence" protocol between the unreliable application and the Publish/Subscribe bus so that "present" applications get their messages delivered immediately, and "not present" applications have their messages queued up in a persistent buffer of some kind, and as soon as they complete the "presence handshake", the queued messages are delivered to the newly present application.
Are there any Publish/Subscribe buses which have this kind of feature built in, or are there any open-source add-ons which do this? Can you point me to any URLs which describe this?
You can achieve this behaviour quite easily with any AMQP-compliant broker (such as RabbitMQ).
Choose the correct exchange type for your usage model. You'll want to use a direct exchange if you're always sending to absolutely named destinations, something like chat.messages.
If you want to do pattern-based routing, you'll want to use topic exchange. Then you can route based on patterns such a chat.messages.*.
Routing is described in more detail in the RabbitMQ Tutorials.
To create the kind of persistent subscription that you mention, have each subscriber create a queue that is private to that subscriber. The queue is then bound to the relevant routing keys on your chosen exchange.
Since each subscriber has its own queue, messages will be consumed by the subscriber when active and stored when subscriber is inactive or disconnected.
You haven't mentioned your language of choice, but in Java you can accomplish this with JMS using durable subscribers. Any implementation of JMS (there are many, including the aforementioned RabbitMQ) will support this feature.