Given that rabbitmq implements flow control, which can slow some of the message publishing down, How can one use timeouts. Also, as pika also doesnt implement publisher confirms,
which language and library are recommended for publishing code
Related
I understand that every messaging platform (rabbitmq, activemq) provides some mechanism that guarantees message delivery between the server (rabbitmq) and the consumers, so if a consumer is down, the message will still be in a queue and redelivered. But what happens in case the messaging platform is down and the client (producer of the message) could not deliver the message to the platform (Rabbitmq)? In that case, a retry method to redeliver the message is needed, but I'm not sure if I have to implement it myself, or clients libraries have this option , is it declared as a parameter in the connection/channel creation? is it platform specific? Also, protocols like AMQP specifies this? I heard also from Spring Retry as a option.
A spring-retry RetryTemplate can be wired directly into Spring AMQP's RabbitTemplate.
Kafka has retry capability built into the Producer (see the retries property).
For JMS, you can wrap the JmsTemplate into a RetryTemplate.execute() method.
The retry policy, back-off etc is all configurable.
I am using Spring AMQP to publish messages to RabbitMQ. Consider a scenario:
1. Java client sends a message to MQ usin amqpTemplate.convertAndSend()
2. But RabbitMQ is down OR there's some n/w issue
In this case the message will be lost? OR
Is there any way it'll be persisted and will be retried?
I checked the publish-confirm model as well but as I understood, ultimately we've to handle the nack messages through coding on our own.
The RabbitTemplate supports adding a RetryTemplate which can be configured whit whatever retry semantics you want. It will handle situations when the broker is down.
See Adding Retry Capabilities.
You can use a transaction or publisher confirms to ensure rabbit secured the message.
Tech Stack:
.Net 4.6.1
Erlang 18.3
Language: C# 6
NServiceBus 6
RabbitMQ 3.6.3
Windows 7
Context:
We are trying to implement a dumb event publisher with smart subscription in NServiceBus on the RabbitMQ transport.
All interested receivers subscribe to an event.
A Publisher publishes the event to an event channel.
All subscribers receive a copy of the event.
Note that I did not say each TYPE of subscriber receives a copy of the message. If there are multiple instances of a service running, and they all have active subscriptions to an event, each INSTANCE of the subscribing service should get a copy of the message.
However, NServiceBus' notion of Publish-Subscribe delivers a published event to one and only one receiver on a given channel. In our case, one instance of a given subscribed service.
I hesitate to list messaging "patterns" because they don't seem to be named or described particularly consistently. However, I believe we are esentially trying to implement the "Multicast" version of the Publish-Subscribe Channel messaging pattern from the Enterprise Integration Patterns (Hohpe and Woolf) Book.
Problem Statement:
Our business case is this:
We have a configuration service that provides application configuration to all other services in our broader application.
Each service requests its configuration from the configuration service on startup by sending a ConfigurationRequest message.
The configuration services replies to the specific instance that made the ConfigurationRequest with a ConfigurationResponse. This is done using NServiceBus' Full-Duplex (commonly called Request-Response) feature.
There is a website that can modify configuration globally. When it does so, it notifies the configuration service with an UpdateConfiguration command.
The configuration service publishes a ConfigurationUpdated event that all other services are subscribed to.
Each service can have multiple instances running on more than one server. ALL instances of a service need to update their configuration, not just one instance.
Each service instance is calling NServiceBus' EndpointConfiguration.MakeInstanceUniquelyIdentifiable with a distinct discriminator.
Currently when we publish the ConfigurationUpdated event, only one instance of each service type gets the message. The events are distributed round-robin, instead of each instance getting a copy of the message.
We have worked around this problem by keeping a record of running services instances and sending the ConfigurationUpdated event (as a Command in NServiceBus) to each of them individually, but Pub-Sub implies that we should have dumb publishers and smart subscribers, and our workaround is the opposite... Our publisher looks up a listof each subscriber and sends to it implicitly. Is there some configuration I am missing on the subscriber side that would allow each instance of each service to get a copy of the published ConfigurationUpdated event? If not, where should I look in NServiceBus to start implementing such a feature? Routing topology, perhaps?
i'm not familiar enough with nservicebus to know how to do it with that toolset, but the RabbitMQ implementation would be a "fanout" exchange with a queue per consumer.
Every queue bound to a fanout exchange will get a copy of the message. If there is a single consumer for each of those queues, then you will be sending a copy of the message to each of the consumers.
It sounds like you have multiple consumers connected to the same queue. Maybe there's a way to tell nservicebus to create a queue for each consumer instance?
p.s. you're right about pub-sub pattern being multicast. i talk about this in my RMQ Patterns ebook (https://leanpub.com/rmq-patterns) and that's what the EIP book would say, as well.
Does NServiceBus, at any point, for any reason, have to post empty messages to MSMQ, and if so, why and when does it happen? Longer explanation below.
A project I work on makes use of NServiceBus version 4. That version does not allow "multi-hosting" of event handlers for different queues in a single process, which may be inconvenient if your project contains 40 or so different queues.
To overcome this problem in development, I made a small "router" app, which listens to all the necessary MSMQ queues and simply forwards all messages from them into a single "unified" queue. That "unified" queue is specified as the queue name for the "unified endpoint" process, which references all the handlers for all the messages that would normally be handled from those various queues.
The setup kind-of works, it seems (with most handlers, at least), but there is one mysterious behaviour (which, I presume, may have something to do with the set-up not working with some other handlers). Namely, as soon as the project starts up, my "router" immediately discovers a number of empty MSMQ messages posted to the queues it has to listen to. Apparently, NSB is publishing those messages during start-up for some reason (and most probably the router is snitching them up before NSB has the chance to look at them again).
I am sure this is not an artefact of my implementation as this does not happen unless NSB is also started. I am curious about the reasons.
NServiceBus, by default, autosubscribes to all handled events if it knows the endpoint which publishes them. These empty messages you see might be the subscribe messages because they are being sent during the endpoint start-up phase.
The mechanics behind the subscribe messages are documented here. TL;DR for transports that do not provide publishing natively (e.g. MSMQ, Azure Storage Queues) NServiceBus emulates it using subscribe messages and internal subscription lists (storages).
You can verify this by checking the message intent header. If they are not subscribe messages, please share the complete list of headers of such a message for further investigation.
How can manual retry work in RabbitMQ after a message has been put onto dead letter queue?
Does RabbitMQ provide an user interface through which you can do this? I assume here that RabbitMQ console does not provide you this capability.
The Rabbit MQ management interface would let you do this crudely, you can go into the deadletter queue, 'get' the message then copy the content. Go to the queue you want to retry the message on and 'publish' it directly to that queue.
Alternatively, you can enable the shovel plugin which allows you to move messages from one queue to another. The RabbitMQ Management plugin directly contains instructions on how to do this.
You can write a consumer / producer using a number of various client libraries. For python a popular library is pika (https://pypi.python.org/pypi/pika).
The script can consume all the messages in a queue, then publish them to another queue.