Is it possible to use the AsyncRabbitTemplate in a multi instance environment?
In my case, the instance that sends the request might not be the one handling the response. AsyncRabbitTemplate keeps track of sent messages in a ConcurrentHashMap so I'm wondering if the consumer only consumes messages that it has a reference to.
If this is an entirely wrong approach, can you point me in the right direction?
You can have multiple instances but the replies have to go back to the sending instance (they can't use the same reply queue).
If a late (or unknown) reply is received, it is logged and discarded.
Related
Setting up a CMS consumer with a listener involves two separate calls: first, acquiring a consumer:
cms::MessageConsumer* cms::Session::createConsumer( const cms::Destination* );
and then, setting a listener on the consumer:
void cms::MessageConsumer::setMessageListener( cms::MessageListener* );
Could messages be lost if the implementation subscribes to the destination (and receives messages from the broker/router) before the listener is activated? Or are such messages queued internally and delivered to the listener upon activation?
Why isn't there an API call to create the consumer with a listener as a construction argument? (Is it because the JMS spec doesn't have it?)
(Addendum: this is probably a flaw in the API itself. A more logical order would be to instantiate a consumer from a session, and have a cms::Consumer::subscribe( cms::Destination*, cms::MessageListener* ) method in the API.)
I don't think the API is flawed necessarily. Obviously it could have been designed a different way, but I believe the solution to your alleged problem comes from the start method on the Connection object (inherited via Startable). The documentation for Connection states:
A CMS client typically creates a connection, one or more sessions, and a number of message producers and consumers. When a connection is created, it is in stopped mode. That means that no messages are being delivered.
It is typical to leave the connection in stopped mode until setup is complete (that is, until all message consumers have been created). At that point, the client calls the connection's start method, and messages begin arriving at the connection's consumers. This setup convention minimizes any client confusion that may result from asynchronous message delivery while the client is still in the process of setting itself up.
A connection can be started immediately, and the setup can be done afterwards. Clients that do this must be prepared to handle asynchronous message delivery while they are still in the process of setting up.
This is the same pattern that JMS follows.
In any case I don't think there's any risk of message loss regardless of when you invoke start(). If the consumer is using an auto-acknowledge mode then messages should only be automatically acknowledged once they are delivered synchronously via one of the receive methods or asynchronously through the listener's onMessage. To do otherwise would be a bug in my estimation. I've worked with JMS for the last 10 years on various implementations and I've never seen any kind of condition where messages were lost related to this.
If you want to add consumers after you've already invoked start() you could certainly call stop() first, but I don't see any problem with simply adding them on the fly.
I am currently learning RabbitMQ and AMQP in general. I started working with some tutorials I found online and all of them show more or less the same example - a Spring Boot web app that, upon a REST call, produces a message and puts in onto a RabbitMQ queue and then, another class from the same app, which is configured as the Consumer of that message consumes it and processes the handler method.
I can't wrap my head around why this is beneficial in any way. The upside I understand is that the handler is executed in a separate thread, while the controller method can return right after sending the message to the queue. However, why would this be in any way better than just using Spring's #Async annotation on that handler method and calling it explicitly? In that case I suppose we would achieve the same thing, while not having to host and manage a seperate instance of a message broker like RabbitMQ.
Can someone please explain? Thanks.
Very simply:
with RabbitMq you can have persistent messages and a much safer and consistent exception management. In case the machine crashes, already pushed messages are not lost.
A message can be pushed to an exchange and consumed by more parallel consumers, that helps scaling the application in case the consumer code is too slow.
and a lot of other reasons...
In my app(multiple instances), we occasionally see the case where connection is lost between my app and rabbitmq due to network issues(my app and rabbitmq are both alive), then after connection is recovered(re-established) we will receive messages that are unacked.
This creates an issue for us, because my app wasn't dead, and it is still processing the same message it received before, but now the message is redeivered, and it causes the app to process the message again (which can be fatal to us).
Since the app has multiple instances, it is not easy for an instance to check if another instance is processing the same message at the same time. We can't simply filter out redelivered message, because we need this feature to handle instance/app crashes/re-deployments.
It doesn't seem that there is an api to tell rabbitmq when to not redeliver unacked messages.
So what is the recommended practice to handle this situation ?
Thanks,
The general solution for such scenario is to make the consumers handle the messages in an idempotent manner . Generally what I do is from the producer side ( in case there is no unique identifier in the message body ) I add an attribute idempotencyId to the message body which is a guid and on the consumer side for each message this id is validated against the stored value in database , any duplicates are rejected.
This approach also works for messages which might be shoveled from another cluster or if in a same cluster multiple instances of consumers are listening then too this approach guarantee one time processing.
Would suggest to go over the RabbitMQ Reliability Guide here
Yeah, exactly-once delivery is not something RabbitMQ is good at. In fact, I'd say you should probably not be using it for these kinds of problems. Honestly, the only way to truly fix this is to use distributed transactions or locking.
Anyway, you could turn the problem on its head by ack'ing the message as soon as the consumer gets it, before it starts working on it. That would avoid the RabbitMQ-related duplication issue at least. This is at-most-once delivery.
Of course, it means that if the consumer crashes, the message is lost forever. So you need to persist the message right before you ack it so you can recover it later and also the consumer should remove it once it's complete.
Considering that crashes are rare, you can then have a single dedicated process that just works on those persisted messages. Or for that matter, handle them manually.
Just be aware that you are pushing the duplication problem in front of you, because the consumer might fail to remove the persisted message after it's done working with it anyway, but at least you have the option to implement it however you want.
Storage in this case could be anything from files, a RDBMS or something like ZooKeeper or Redis to lock/unlock in-flight messages.
We want to use Akka to implement a scenario when messages are fetched from a message queue (RabbitMQ) and then processed by a chain of actors. The queue is durable and messages must not be lost. So we need to send an acknowledgement (BasicAck in RabbitMQ) back to the queue in order to finalize the dequeued message. Because of that the very last actor in the processing chain needs to do the acknowledgement. This seems to be rather common need, and I wonder if there is a known pattern for this. Vaughn Vernon in his book writes about using Return Address, so all messages sent along the chain will have the return address (of the MQ channel actor) and the correlation identifier that specifies the queue message tag. Is this the proper way to do it?
An alternative is to ack the message right after the receival and then use persistent actors to provide its guaranteed delivery, but I was adviced against such approach because use of AMPQ eliminates the need for actor persistance for this particular scenario.
I'm not really familiar with Akka, but I think I get the gist of what it does (very similar to "process" in Erlang - i think - which is what RMQ is built on).
In general, your first suggestion from Vaughn Vernon's book is the way to go.
In my specific scenarios, I have taken a "middleware" approach to what you are suggesting. My specific middleware implementation forwards the message itself through a chain of commands that process the message. Each command calls an action.next() method to continue forwarding to the next command.
Prior to sending the message through the middleware, I create a default last-command-in-the-chain. This default command simply calls actions.ack() - which, behind the scenes, acknowledged the message.
I do things this way so that the commands never have to know anything about how to actually implement the mechanics of completing and moving on to the next thing. They have an API specific to themselves, being commands in a chain.
This allows me to change the implementation of acknowledging the message, or how i handle messages from RMQ, etc, without changing the commands directly.
Ack'ing the message immediately introduces danger, as your actor could crash, Akka itself could crash, and a host of other problems can (and will) occur, and you'll be more likely to lose the message.
Remember, though - there is not 100% perfect setup. You will, at some point, lose a message or process the same message twice. Your system needs to handle these scenarios in some way, at some point. Everything your doing is heading down the right path to make this less likely, but nothing will ever prevent crashes and message loss 100% of the time.
I am trying to set up broadcast messaging to all nodes in the system. When a new node joins the system, it publishes a message to everyone else to announce its entry. The way I have designed is that, a exchange exists to which all nodes will bind its own queue. Whenever a new node joins the system, it will bind its queue as well to the exchange and publish a message to the exchange. All nodes will receive this msg(including itself) and all other nodes(except this message) will send a "ack" message so that the new node will get to know the available nodes in the system. But somehow I couldn't able to get this working. My broadcast messages doesn't propagate to every node in the system. A simple one node publishing and rest consuming is working. But same node publishing and consuming is somehow screwed up somewhere.
Is there any other efficient way of doing this apart from the logic mentioned above? Or is there any restriction from rabbitmq perspective to achieve the above or is my code buggy and do I have to take a closer look at it.
The way you described it, your solution should work. However, without more detailed code examples (of the consume/publish logic in the "announcer" and the consume/acknowledge-publish logic in the other peers) it's difficult to debug.
A couple common problems could be tripping you up, though:
If you're considering "did I get responses back from all the other nodes" as the authority for "did the other nodes get my announce message?", you might need to acknowledge (basic.ack in AMQP) the messages your announcer is receiving as it gets them. Otherwise, it's possible you're not seeing subsequent messages due to consumer prefetch, though in most client libraries you'd have to be explicitly turning that on somewhere first.
Make sure your other peers (the ones receiving the "announce" and sending a message back) are acknowledging the message as well, or are consuming in "no-ack" mode. Otherwise, if they get blocked (via flow, rate-limiting, or prefetch), they will probably receive announces for awhile and then stop.
Make sure you're using a "fanout" type exchange. It sounds like you want unconditional-fanout behavior, so you don't need to muck about with topic routing. If you're using a topic or direct exchange, you may have a bug in your routing logic, in which case switching to fanout will work. I suspect you're already doing this though.
This is likely not the issue, but: you mention that your peers (not the announcer) are "acknowledging" the announce. Make sure that they acknowledge the announce by publishing a new message back to the announcer's queue directly (with no exchange, just a routing key), not by sending a basic.ack to RabbitMQ (that doesn't notify the sender of anything), and not by publishing an announce-received to the fanout exchange.
As an aside, I don't know why you're doing declare-queue/bind/publish as opposed to publish/declare-queue/bind; is there a good reason you need an announcing node to receive its own announce message? If you're after a "self-test" behavior, I think it's probably better to just implement a periodic "can things announce successfully?" health-check somewhere instead, though that's entirely subjective.
Have you tried the RPC style message, with a callback queue that you identify in the broadcast message's propeties? Like at the rabbitmq tutorial.