Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 months ago.
Improve this question
When building webhooks, it's a best practice for the consumer of the webhook (e.g. the receiver of the webhook) to immediately drop any messages received into a queue to prevent it from "backing up" the delivery of subsequent messages. This seems to be the "best practice" for years regarding webhook architectures. Nowadays, with the advent of internet-accessible queues (e.g. Amazon SQS), why are we not flipping the script on webhook architecture such that the consumer becomes responsible to "pull" messages off a queue, rather than receive message via an http post?
Yes, this essentially is no longer a "webhook", but the concept is the same. The problem being solved here is that a consumer wants to be made "aware" of events happening in another system. Why not have the publisher of these events store all relevant events in a queue dedicated to a single consumer such that the consumer can pull those messages off the queue at their own leisure, pace, etc. I see many benefits to this, mainly the transfer of responsibility to the consumer to "dequeue" messages according to their own abilities. The publisher drops messages as quickly as they can into the queue, and the consumer pulls them off as quickly as they can. If the consumer goes down for any reason, the messages will still remain in the queue for as long as they need. Once the consumer is back up, they can continue pulling message off. No messages ever lost in this scenario. Right?
The way I see it is mostly an opinion, not necessarily the ultimate answer.
While theoretically there's a good point in advocating for pushing messages straight to the queue by the producers, there's a real world constraint that will be imposed on those producers. Every messaging system has some nuances. This means producers have to be aware of those nuances in order to be able to publish to various messaging services. Authentication is another nuance. All this turns into a nightmare for any producer that issues notifications to various consumers. This is what webhooks have solved. Ubiquitous, established protocol, authentication, etc.
Related
I am a newcomer, and I have a question I want to ask for your help. In my project I used ActiveMQ to implement communication. Now there is a requirement that the producer needs to sense whether the consumer has started, and how should I implement it?
As others have chimed in, it is best to not have producers and consumers know about each other. Essentially, that is one of the primary benefits of event-driven-design-- applications are decoupled at runtime.
If there is some hard-and-fast rule to have producers and consumers communicate, then you can implement a communication channel using a dedicated Queue or Topic to send events. This design has a number of names-- side-channel, control channel or out-of-band communication.
Saw similar questions but different expected answers. My question is can I can create a consumer to focus on a single queue until it empties then switch to the other queue, until new work is sent to their main queue?
For example: 1 queue contains large amount of work to be processed in longer time frame and its own dedicated consumers (3 for instance). The 2nd queue receives much less work that requires less processing. If the consumers for the 2nd queue complete their work can I make it so they switch to the first queue until their queue receives more work?
I think for this question, it's important to keep in mind that there is a difference between a "consumer" in the canonical sense vs. a "consumer" in the RabbitMQ sense.
A RabbitMQ Consumer is a contrivance of the protocol - basically, it is a designation that the channel/connection would like to have messages pushed to it, under a designated consumer tag. In this sense, it is merely a notification to the broker to immediately send messages.
In the canonical sense, a message consumer is any piece of code that processes messages.
So, the answer to your question is "yes, go ahead and write your program to do that." You have control over the canonical consumer code. It is up to your software to determine what to do with a message that arrives from a queue.
Now, if you're wondering if RabbitMQ can re-subscribe a consumer to a different queue, the answer is "that's not how it works." In RabbitMQ, a consumer is simply a response to a request to subscribe to a queue - it is a "consumer tag" object. The ongoing nature of the subscription is tied to the channel/connection pair.
What should you do? While your code doesn't specify any particular coding language, in my opinion, you're off-track by even asking this question. Subscribe to both queues. If there is nothing for the worker to do, I think the computer would be perfectly happy with that. If you're worried about a particularly busy queue issuing too much work, you can use a number of techniques to throttle messages coming into that consumer. One popular technique is prefetch.
I have an queue and messages in it. Also i have two consumer in separate processes. I take message by one, and decide that this message is not mine, and reject it with requeue flag. In documentation I found the next phrase "The server MUST NOT deliver the message to the same client within the context of the current channel". Is that mean that the rejected message should be deliver to another consumer or not?
So, there are a couple of things going on here that I'd like to touch on.
First, your question as to the behavior of RabbitMQ. The rule referenced above comes from the AMQP-0-9-1 specification. As with most implementation of open specs, RabbitMQ is not fully-conforming. This page describes in precise detail exactly which portions of the specification are implemented, and where any deviations occur.
On that page, it stipulates that "No attempt is made to prevent redelivery to the same client." RabbitMQ lists this as a planned addition in a future release, but it has been planned for quite a few years now.
Should Consumers Be Picky?
The more important question is the one you haven't directly asked, but that is "should my consumer be picky about which messages from the queue it processes?
The answer to this is a definitive "no." One of the key design assumptions about message queues is that any consumer subscribed to the queue should be able to process any message in the queue. Thus, it should be considered proper design that all consumers attached to the queue are running identical code (same code base, same version). If not, you're going to have some serious problems with your application sooner or later.
Reject should only be used to tell the broker that there is a problem with a particular message. If there is a problem with a particular consumer (e.g. loses connection to a database), it should not reject the message, but instead should close the connection, triggering redelivery to another, working consumer. By design, messages that need to be processed by a specialized or different consumer should be deposited in a different queue.
In our application the publisher creates a message and sends it to a topic.
It then needs to wait, when all of the topic's subscribers ack the message.
It does not appear, the message bus implementations can do this automatically. So we are leaning towards making each subscriber send their own new message for the client, when they are done.
Now, the client can receive all such messages and, when it got one from each destination, do whatever clean-ups it has to do. But what if the client (sender) crashes part way through the stream of acknowledgments? To handle such a misfortune, I need to (re)implement, what the buses already implement, on the client -- save the incoming acknowledgments until I get enough of them.
I don't believe, our needs are that esoteric -- how would you handle the situation, where the sender (publisher) must wait for confirmations from multiple recipients (subscribers)? Sort of like requesting (and awaiting) Return-Receipts from each subscriber to a mailing list...
We are using RabbitMQ, if it matters. Thanks!
The functionality that you are looking for sounds like a messaging solution that can perform transactions across publishers and subscribers of a message. In The Java world, JMS specifies such transactions. One example of a JMS implementation is HornetQ.
RabbitMQ does not provide such functionality and it does for good reasons. RabbitMQ is built for being extremely robust and to perform like hell at the same time. The transactional behavior that you describe is only achievable with the cost of reasonable performance loss (especially if you want to keep outstanding robustness).
With RabbitMQ, one way to assure that a message was consumed successfully, is indeed to publish an answer message on the consumer side that is then consumed by the original publisher. This can be achieved through RabbitMQ's RPC procedure calls which might help you to get a clean solution for your problem setting.
If the (original) publisher crashes before all answers could be received, you can assume that all outstanding answers are still queued on the broker. So you would have to build your publisher in a way that it is capable to resume with processing those left messages. This might turn out to be none-trivial.
Finally, I recommend the following solution: Design your producing component in a way that you can consume the answers with one or more dedicated answer consumers that are separated from the origin publisher.
Benefits of this solution are:
the origin publisher can finish its task independent of consumer success
the origin publisher is independent of consumer availability and speed
the origin publisher implementation is far less complex
in a crash scenario, the answer consumer can resume with processing answers
Now to a more general point: One of the major benefits of messaging is the decoupling of application components by the broker. In AMQP, this is achieved with exchanges and bindings that allow you to move message distribution logic from your application to a central point of configuration.
If you add RPC-style calls to your clients, then your components are most likely closely coupled again, meaning that the publishing component fails if one of the consuming components fails / is not available / too slow. This is exactly what you will want to avoid. Otherwise, why would you have split the components then?
My recommendation is that you design your application in a way that publishers can complete their tasks independent of the success of consumers wherever possible. Back-channels should be an exceptional case and be implemented in the described not-so coupled way.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I have used N Service Bus in one of my projects recently and although I like it but I am still looking for alternate options. I have stumbled on RabbitMQ but before I give it a try I want to find out the following things:
1- Is it reliable? (With N Service Bus if a message gets published to the queue the listener always receives it).
2- If the client is down does it automatically gets the message when it becomes available? like in NSB.
3- Is it lightweight on resources (NSB is very lightweight)?
4- Is it easy to integrate with .Net?
5- Is the Admin panel available with Open Source free version?
6- Is it easy to track down problems if messages are not getting published etc? (This is the pain with NSB)
7- Does it support complex scenarios where there might be N number of Listeners for a message or a single listener that needs to listen to multiple messages etc?
8- Is it configurable from code? (Personally I don’t like to use heave Config files, just personal choice )
9- Is the .Net API of RabbitMQ clean or does it make the code messy?
Kindly give me your feedback. All the above questions are relevant to the Open Source version, I don’t want to buy licensed version yet.
Also suggest if there are any other options available out there.
Thanks,
Is it reliable? (With N Service Bus if a message gets published to the queue the listener always receives it).
Yes. In addition you have more control over the 'reliability' in rabbitmq, for example you can specify that a queue is durable (which means that messages are persisted to disk before being delivered).
2- If the client is down does it automatically gets the message when it becomes available? like in NSB.
Yes.
3- Is it lightweight on resources (NSB is very lightweight)?
RabbitMq is written in erlang and runs as it's own process. You have a lot of insight into how it is consuming memory, but the actual resource usage will be dependent on your workload.
4- Is it easy to integrate with .Net?
Yes. The basic rabbitmq C# wrapper is very easy to use and offers a very simple abstraction over the rabbitmq concepts. There are higher level libraries available if you're coming from NServiceBus (which I believe has a RabbitMQ adapter). You should look at MassTransit, which can use RabbitMQ as well as MSMQ as a transport, and libraries like my own chinchilla or EasyNetQ which are RabbitMQ only.
5- Is the Admin panel available with Open Source free version?
Yes.
6- Is it easy to track down problems if messages are not getting published etc? (This is the pain with NSB)
Yes. Using the admin tool you can see bindings between exchanges and queues.
7- Does it support complex scenarios where there might be N number of Listeners for a message or a single listener that needs to listen to multiple messages etc?
Yes.
8- Is it configurable from code? (Personally I don’t like to use heave Config files, just personal choice )
Yes.
9- Is the .Net API of RabbitMQ clean or does it make the code messy?
Yes.