Can anyone explain in which cases I need to create multiple queues (one user -> one queue name), and when one queue name for all clients with different routing keys (one user -> one routing key) and why?
A user should not be able to read messages intended for another user.
I'm using direct exchange type.
First off I am going to assume that when you say "user" you are interchangeably referring to a consumer or producer, and they aren't the same thing so I would read up on that here in rabbitmq's simplest explanation. Walking through that tutorial will definitely help solidify your understanding of rabbit a bit more overall too, which is always good.
In any case, I would recommend doing this:
Create multiple queue's, each one linked to a single consumer. The reason for doing this instead of using a single queue with multiple is discussed here but if you don't want a bunch of programmer jargon, it pretty much says that a single queue is super slow because only one message can be consumed at a time from the queue.
Also, there is a built in "default exchange" that you can use instead of setting up another direct exchange which it sounds like you're putting effort into that you might not need to, obviously I'm not sure what you are doing but I would take that into consideration... hope this helps!
Related
I am trying to use rabbitmq as a part of the notification system. I have an exchange called "notification_events" and the queues in the exchange are based on the types of events, for example, 'send_account_notification_queue' or 'send_tickets_notification_queue'. In order to send to specific user(s) I plan on binding userId to the appropriate queue as a routing key. And I'm sure the number of routing keys will grow with more users...
I read that it is bad to have thousands or millions of queues, but how about routing keys? Are there better ways of doing this? Any help is appreciated and thanks in advance for your time :)
Do you really need queueing per user? Have you considered having a single queue per type of event and using userId to notify the appropriate user? It assumes that a given user notification is fast and cannot fail just for the subset of users.
If you need more complex per-user logic (like reordering events) or dealing with a specific user not being able to receive an event then queueing system is not the right abstraction. Look for an orchestration system like temporal.io that supports per-user object with as complex logic as necessary.
See this answer that explains how Temporal solves it for a system with similar requirements.
I have the feeling I am not understanding something fundamental in AMQP/RabbitMQ, since I cannot find much help on this specific detail.
Let's assume I have a system made up of several components sending each other messages via a RabbitMQ broker. The messages can have routing keys of the form XXX.YYY. Let's further assume XXX and YYY are numbers between 000 and 999. That means there are a total of 1,000,000 different possible routing keys.
Now, not every component in my system is interested in every message. Let's say there is a component that wants all the messages in which XXX is between 300 and 500 and YYY is between 600 and 900. That means the component wants to process messages referring to 200*300 = 60,000 different routing keys. Also, the component might be restarted at any point in time and needs to be able to start processing the messages quickly after restart.
Furthermore, the routing keys the component is interested in might change at runtime.
There are several ways to approach this that I can think of:
Use topic exchanges and subscribe to each routing key. If I do this using one connection and one channel, it is awfully slow. My understanding is that bindings are created sequentially for each channel and thus creating 60,000 bindings takes a while. Adding and removing bindings is trivial, though. Would it be feasible to create more channels so that bindings can be created in parallel?
Use topic exchanges and wildcards, discard messages you're not interested in in the client. We could subscribe to *.* and receives messages for all 1,000,000 routing keys => much more load in the client. Or subscribe to all 200 relevant values of XXX.* and receive messages for 200,000 routing keys. Is this a generally applied pattern?
Use headers exchanges and set x-match to any. This feels a little hacky and it seems headers exchanges are not widely used. You also have to deal with the maximum size of the header when defining a binding. Do people do this? You only need a handful of bindings though, so re-creating the bindings after a restart is very fast. Updating the set of topics we're interested in is also not a problem: Just re-create everything.
So, I guess my question is: What's the best practice to subscribe to a large amount of topics very quickly (<5s) and still be able change routing keys dynamically at run-time?
Would it be feasible to split the component which needs the messages and the subscription into two components? One component is only responsible for keeping the subscriptions up-to-date (this would exchange-to-exchange subscriptions) and the other components receives every message from the downstream exchange.
I have a scenario where publishers need to send messages to a known exchange (they won't know the downstream queues directly). Most of the messages should be delivered to every bound queue (like a fanout), but some need to be delivered only to a specific bound queue. I'd like to avoid multiple exchanges as well as multiple queue bonds.
My current solution uses header routing, where a message has one of two headers, identify it as "global" or "specific to a particular sub", where the downstream queues are bound on a match-any to both of those headers. I think this will work, but I feel like there should be a simpler solution.
I tried to find an exchange plugin that would "fanout all messages except specific ones", but I couldn't find such a plugin. Outside of that, any ideas on how to implement such a routing strategy?
For what it's worth, my original solution using a "headers" exchange with queues bound using "match any" is the only one I could find, short of writing a new exchange plugin. It does work and so far seems reasonably fast (at least not measurably slower than a typical "topic" exchange--which I could find no way to apply in this scenario).
From my research on this topic, the ideal solution would be a "topic" exchange with the ability to use a RegEx or at least some form of "or-logic". I did find some information that implied that a RegEx was considered but decided against (in favor of the "dotted notation" topic format) because the latter was faster, specifically because the use a a RegEx would require that every binding be evaluated on each new message (i.e. there was no way to construct a "search shortcut").
For now, my "match-any headers exchange" solution will serve my purposes, but in the future, a "topic" exchange that allowed "or-logic" might be worth exploring. It would allow multiple topic patterns to be achieved with single-binding, without the overhead of a RegEx. But I have no experience with Erlang, nor the time to learn enough of it to write the necessary plugin. Please contact me if anyone is interested in collaborating on this.
What you describe really sounds like exchange-to-exchange binding though, as rich routing is actually one of RMQ's strengths.
You could create the entry exchange that would point to either the fanout (for generic case) or topic/direct one (for the special one), and leave all routing to RMQ. The entry-exchange could be a header exchange or a direct one, depending on what you want to put in headers:
entry-exchange -----> fanout-exchange ---*---> multiple-fanout queues
\
\---> the special exchange --> queue-for-special-usecase
So we will have a topic exchange that looks something like
{class}.{genus}
So we have some consumers that bind with the topic
mammal.*
(or bird.*, etc.)
Now suppose later on we want to include species information so the topic exchange now looks like this:
{class}.{genus}.{species}
Now the old consumers are broken :(
However they could have bound as
mammal.*.#
And been able to listen to whatever future information is added. However, this is something my team came up with on our own which leads me to ask:
Is this good practice?
Are there tradeoffs to this I should be aware of?
Is there an alternate way to have a producer be able to add information without breaking existing consumers, without publishing to multiple exchanges?
Typically if you have a need maximum control on queue delivery and want to do the logic in rabbit, then you should consider header exchanges.
Usually when we code up the publish we know exactly which queue it needs to go to, so whether you want to use a routing key or a boolean to do this might not make much difference depending on your application.
This brings up another design consideration to be aware of: whether you want routing logic in rabbit. Someone people prefer to just use simple routing keys and either direct or topic exchanges, focusing on flexible consumers. Its going to be hard to guess at what is best for your application obviously.
Keep in mind that your consumers will be subscribed, often statically, to the queue(s) that the exchange delivers to. Also mammal.# is the same as mammal.*.# (see: ref)
This is similar to this other question but with a bit of a twist: I read in the specification that the message-id for AMQP messages should be set by the application itself, so in theory I could use that to guarantee a certain degree of uniqueness, right?
My main question is now: In what scope is that message-id garantueed to be unique? For the messages currently enqueued inside a specific queue? Over all queues? Over the universe? :-)
And is this behaviour standardized? I plan to use RabbitMQ here, but it would be nice to have something not vendor specifc :-)
Thanks.
Another suggestion is according to the dump pipes - smart endpoints school of thought.
You could handle uniqueness in your application, using some sort of shared state.
We had the same problem when switching from Gearman to RabbitMQ. We use memcached to keep track of unique message ID's posted and consumers drop messages for which the message ID is already stored in memcache (duplicates). You could also check memcache before putting it on the queue altogether.
This frees you from using this feature in your message bus layer (so you can more easily switch between brokers, also those who do not guarantee uniqueness)
Message Id is application-specific only and may be not unique at all. You have to take care of uniqueness by yourself.