I've been tasked with rearchitecting/repairing a flawed RabbitMQ/Elixir application. The guy who originally wrote it is unavailable.
One thing he did was to put the data of the message in the header property rather than in the payload. I don't know which is the appropriate way. I tend to lean toward payload because the data isn't a header in my mind. I've been looking for example RabbitMQ messages and while I found hundreds or sites addressing details about the message data/content/format, I can't find any tangible examples.
So my question is two-fold: does anyone know where I can find examples of the shape/format of the message, and which is the right place to put the data we are trying to send?
thanks!
Consider posting a letter, think of the headers as what you put on the envelope and the body as what you put in the envelope. RabbitMQ is the postal system.
Related
So I'm thinking of using RabbitMQ to send messages between all the varied apps in our organization. In the attached image is essentially the picture in my mind of how things would work.
So the message goes into the exchange, and splits out into three queues.
Payloads are always JSON text.
The consumers are long-running windows services whose only job is to sit and listen for messages destined for their particular application.When a message comes in, they look at the header to determine how this payload JSON should be interpreted, and which REST endpoint it should be sent to. e.g., "When I see a 'WORK_ORDER_COMPLETE' header I am going to parse this as a WorkOrderCompleteDto and send it as a POST to the CompletedWorkOrder WebAPI method at timelabor-api.mycompany.com. If the API returns other than 200, I reject the message and let rabbit handle it. If I get a 200 back from the API, then I ack the message to rabbit."
Then end applications are simply our internal line-of-business apps that we use for inventory, billing, etc. Those applications are then responsible for performing their respective function (decrementing inventory, creating a billing record, yadda yadda.
Does this in any way make a sensible understanding of a proper way to use Rabbit?
Conceptually, I believe you may be relying on RabbitMQ to do things that your application needs to do.
The assumption of the architecture seems to be that each message is processed by each of your consuming applications totally in a vacuum. What this means is that you don't care that a message processed successfully by Billing_App ultimately failed with Inventory_App. Maybe this is true, but in my experience, it isn't.
If the end goal is to achieve some consistent state in the overall data, you're going to need a some supervisory component orchestrating and monitoring the various operations to ensure that the state is consistent. This means, in effect, that your statement about rejecting a message back to RabbitMQ means you have a bit more thought to put into what happens when something fails.
I would focus on identifying some UML activity diagrams that describe your behavior and how it achieves the end-state, and use that as a guide to determine how the orchestration of your application needs to be designed.
I have a scenario where publishers need to send messages to a known exchange (they won't know the downstream queues directly). Most of the messages should be delivered to every bound queue (like a fanout), but some need to be delivered only to a specific bound queue. I'd like to avoid multiple exchanges as well as multiple queue bonds.
My current solution uses header routing, where a message has one of two headers, identify it as "global" or "specific to a particular sub", where the downstream queues are bound on a match-any to both of those headers. I think this will work, but I feel like there should be a simpler solution.
I tried to find an exchange plugin that would "fanout all messages except specific ones", but I couldn't find such a plugin. Outside of that, any ideas on how to implement such a routing strategy?
For what it's worth, my original solution using a "headers" exchange with queues bound using "match any" is the only one I could find, short of writing a new exchange plugin. It does work and so far seems reasonably fast (at least not measurably slower than a typical "topic" exchange--which I could find no way to apply in this scenario).
From my research on this topic, the ideal solution would be a "topic" exchange with the ability to use a RegEx or at least some form of "or-logic". I did find some information that implied that a RegEx was considered but decided against (in favor of the "dotted notation" topic format) because the latter was faster, specifically because the use a a RegEx would require that every binding be evaluated on each new message (i.e. there was no way to construct a "search shortcut").
For now, my "match-any headers exchange" solution will serve my purposes, but in the future, a "topic" exchange that allowed "or-logic" might be worth exploring. It would allow multiple topic patterns to be achieved with single-binding, without the overhead of a RegEx. But I have no experience with Erlang, nor the time to learn enough of it to write the necessary plugin. Please contact me if anyone is interested in collaborating on this.
What you describe really sounds like exchange-to-exchange binding though, as rich routing is actually one of RMQ's strengths.
You could create the entry exchange that would point to either the fanout (for generic case) or topic/direct one (for the special one), and leave all routing to RMQ. The entry-exchange could be a header exchange or a direct one, depending on what you want to put in headers:
entry-exchange -----> fanout-exchange ---*---> multiple-fanout queues
\
\---> the special exchange --> queue-for-special-usecase
TL;DR: I need to "replay" dead letter messages back into their original queues once I've fixed the consumer code that was originally causing the messages to be rejected.
I have configured the Dead Letter Exchange (DLX) for RabbitMQ and am successfully routing rejected messages to a dead letter queue. But now I want to look at the messages in the dead letter queue and try to decide what to do with each of them. Some (many?) of these messages should be replayed (requeued) to their original queues (available in the "x-death" headers) once the offending consumer code has been fixed. But how do I actually go about doing this? Should I write a one-off program that reads messages from the dead letter queue and allows me to specify a target queue to send them to? And what about searching the dead letter queue? What if I know that a message (let's say which is encoded in JSON) has a certain attribute that I want to search for and replay? For example, I fix a defect which I know will allow message with PacketId: 1234 to successfully process now. I could also write a one-off program for this I suppose.
I certainly can't be the first one to encounter these problems and I'm wondering if anyone else has already solved them. It seems like there should be some sort of Swiss Army Knife for this sort of thing. I did a pretty extensive search on Google and Stack Overflow but didn't really come up with much. The closest thing I could find were shovels but that doesn't really seem like the right tool for the job.
Should I write a one-off program that reads messages from the dead letter queue and allows me to specify a target queue to send them to?
generally speaking, yes.
you could set up a delayed re-try to resend the message back to the original queue, using a combination of the delay message exchange plugin.
but this would only automate the retries on an interval, and you may not have fixed the problem before the retries happen.
in some circumstances this is ok - like when the error is caused by an external resource being temporarily unavailable.
in your case, though, i believe your thoughts on creating an app to handle the dead letters is the best way to go, for several reasons:
you need to search through the messages, which isn't possible RMQ
this means you'll need a database to store the messages from the DLX/queue
because you're pulling the messages out of the DLX/queue, you'll need to ensure you get all the header info from the message so that you can re-publish to the correct queue when the time comes.
I certainly can't be the first one to encounter these problems and I'm wondering if anyone else has already solved them.
and you're not!
there are many solutions to this problem that all come down to the solution you've suggested.
some larger "service bus" implementations have this type of feature built in to them. i believe NServiceBus (or the SaaS version of it) has this built in, for example - though I'm not 100% sure of it.
if you want to look into this further, do some search for the term "poison message" - this is generally the term used for this situation. I've found a few things on google with a quick search, that may help you down the path:
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-January/025019.html
https://web.archive.org/web/20170809194056/http://tafakari.co.ke/2014/07/rabbitmq-poison-messages/
https://web.archive.org/web/20170809170555/http://kjnilsson.github.io/blog/2014/01/30/spread-the-poison/
hope that helps!
I am considering using AMQP for an application where delivery order is paramount.
I cannot therefore use the normal re-delivery features, as undelivered messages are re-queued out of order.
It looks like what I must do is to leave the message on the queue until it has been processed, and then specifically delete it. It is then possible that the same message is processed twice in order, but that is easy to trap and deal with.
However, I don't see how to do this. What I am looking for is some sort of peek and delete message methods, giving me direct control, but they don't seem to exist.
Am I missing something, or trying to solve the problem in the wrong way?
You cannot have peek-and-delete in AMQP. Actually, you cannot browse the messages on a queue without consuming them and Rabbit does not provide any extension to enable this.
The general response to your problem is "Think very carefully if you actually need that in-order constraint", because, for instance, with that constraint in place, you cannot have multiple consumers on a queue.
I have been solving the same problem. In my solutions I have been wrapping the messages into one single message where the outer message has been processed first and then I have processed the remaining inner messages in the wrapped order. This has some disadvantages, for example big messages (once your wrapping hierarchy contains many messages), more difficult serialization, ..., but for me the solution was suitable enough.
I am trying to get a further understanding of message buses and one question that keeps coming up in my head is "how does the message get onto the bus?". Now, I assume there is a service (WCF, etc) of some sort that receives messages and puts them onto the bus. So then the other question I have is isn't this service then likely to be a bottleneck? I assume you would architect this service so that it can be easily scaled, such as through load balancing? Or would there be another way?
Also (sorry, it was originally only supposed to be one question), where would the routing tables be held that define where messages should go; in a database? Again, wouldn't this then be a potential bottleneck?
I am trying to look at this from a non product (BizTalk etc) or framework (NServiceBus, Mass Transit, etc) perspective. As if you were going to be writing this kind of thing from scratch. I want to get my head about what you are getting and the potential issues. I guess if you use BizTalk it has the message box for the routing tables, a notorious bottleneck in the past. I also see that you have the concept of "on ramps" with the ESB part of 2009. But as I said, I would like to think beyond a product and how people see it should be architected.
Many thanks for any insight.
One thing you might want to consider is that a Service Bus is something slightly different than just a Message Bus. In order to understand the difference, we need to look at what is a service in the SOA sense.
A WCF service isn't an SOA service - as it isn't necessarily autonomous (either at runtime, where it can be blocked by other WCF services it calls, or at design time, where it may require versioning when the WCF services it calls change).
Most of the technical questions you raise (scaling, routing, etc) are first and foremost addressed by the autonomy of the service in question. Only then does an ESB begin to make sense.
I understand that this doesn't provide much in the way of guidance, but you can try reading some of the stuff I've written on this topic (for the past 3 years) on my blog and in the articles I've published. Here's a good (and recent) one that can get you started in the right direction:
http://www.udidahan.com/2009/09/29/article-eda-soa-through-the-looking-glass/
Hope that helps in some way.