I am trying to use rabbitmq as a part of the notification system. I have an exchange called "notification_events" and the queues in the exchange are based on the types of events, for example, 'send_account_notification_queue' or 'send_tickets_notification_queue'. In order to send to specific user(s) I plan on binding userId to the appropriate queue as a routing key. And I'm sure the number of routing keys will grow with more users...
I read that it is bad to have thousands or millions of queues, but how about routing keys? Are there better ways of doing this? Any help is appreciated and thanks in advance for your time :)
Do you really need queueing per user? Have you considered having a single queue per type of event and using userId to notify the appropriate user? It assumes that a given user notification is fast and cannot fail just for the subset of users.
If you need more complex per-user logic (like reordering events) or dealing with a specific user not being able to receive an event then queueing system is not the right abstraction. Look for an orchestration system like temporal.io that supports per-user object with as complex logic as necessary.
See this answer that explains how Temporal solves it for a system with similar requirements.
Related
I have the feeling I am not understanding something fundamental in AMQP/RabbitMQ, since I cannot find much help on this specific detail.
Let's assume I have a system made up of several components sending each other messages via a RabbitMQ broker. The messages can have routing keys of the form XXX.YYY. Let's further assume XXX and YYY are numbers between 000 and 999. That means there are a total of 1,000,000 different possible routing keys.
Now, not every component in my system is interested in every message. Let's say there is a component that wants all the messages in which XXX is between 300 and 500 and YYY is between 600 and 900. That means the component wants to process messages referring to 200*300 = 60,000 different routing keys. Also, the component might be restarted at any point in time and needs to be able to start processing the messages quickly after restart.
Furthermore, the routing keys the component is interested in might change at runtime.
There are several ways to approach this that I can think of:
Use topic exchanges and subscribe to each routing key. If I do this using one connection and one channel, it is awfully slow. My understanding is that bindings are created sequentially for each channel and thus creating 60,000 bindings takes a while. Adding and removing bindings is trivial, though. Would it be feasible to create more channels so that bindings can be created in parallel?
Use topic exchanges and wildcards, discard messages you're not interested in in the client. We could subscribe to *.* and receives messages for all 1,000,000 routing keys => much more load in the client. Or subscribe to all 200 relevant values of XXX.* and receive messages for 200,000 routing keys. Is this a generally applied pattern?
Use headers exchanges and set x-match to any. This feels a little hacky and it seems headers exchanges are not widely used. You also have to deal with the maximum size of the header when defining a binding. Do people do this? You only need a handful of bindings though, so re-creating the bindings after a restart is very fast. Updating the set of topics we're interested in is also not a problem: Just re-create everything.
So, I guess my question is: What's the best practice to subscribe to a large amount of topics very quickly (<5s) and still be able change routing keys dynamically at run-time?
Would it be feasible to split the component which needs the messages and the subscription into two components? One component is only responsible for keeping the subscriptions up-to-date (this would exchange-to-exchange subscriptions) and the other components receives every message from the downstream exchange.
I have a requirement to design a notification system for multi-user(~1000 users) application, here are the high level requirements.
System event gets triggered on specific operations.
On event trigger, individual notification for all(or sometimes only for relevant) users gets generated and stored in database.
While user logs in, all unread notifications for him will be pulled and displayed in ui.
While user reads the notification, we capture the read status.
A scheduler in background evicts all the stale notifications.
This seems like a very typical use case and straight forward to implement with the database.
But my doubt is, is there any way we can replace the Database with the Queue based messaging system? The reason I think this way is because, the use case I have seems like asynchronous in nature(like events, notifications and timely eviction of messages).
While I replace the Database with Queues, the first 2 points from above fits well, but on later part I have some doubts -
In General, are queues flexible to store and query notifications based on user ids ?
Consider this scenario - Notifications gets generated and stored in the queue, and the user is not logged in, what is the best way to handle consumer messages.
a. Should the consumer constantly listen for the messages ?, If so should the messages be stored in application memory(does not seems to be good option) ?
b. Or the consumers should be created for each users dynamically on user login? Is this a regular pattern ?
Any other recommended ways ?
Thanks
Your use-case is suited to a database, not a message queue. While conceptually similar to the use case, a message queue is intended for extremely short-duration storage (i.e. to buffer data moving between running processes). Since you have no control over when users log in, these notifications will potentially be stored for minutes, hours, maybe even weeks. You need a persistent storage mechanism.
Can anyone explain in which cases I need to create multiple queues (one user -> one queue name), and when one queue name for all clients with different routing keys (one user -> one routing key) and why?
A user should not be able to read messages intended for another user.
I'm using direct exchange type.
First off I am going to assume that when you say "user" you are interchangeably referring to a consumer or producer, and they aren't the same thing so I would read up on that here in rabbitmq's simplest explanation. Walking through that tutorial will definitely help solidify your understanding of rabbit a bit more overall too, which is always good.
In any case, I would recommend doing this:
Create multiple queue's, each one linked to a single consumer. The reason for doing this instead of using a single queue with multiple is discussed here but if you don't want a bunch of programmer jargon, it pretty much says that a single queue is super slow because only one message can be consumed at a time from the queue.
Also, there is a built in "default exchange" that you can use instead of setting up another direct exchange which it sounds like you're putting effort into that you might not need to, obviously I'm not sure what you are doing but I would take that into consideration... hope this helps!
This is a new area for me so hopefully my question makes sense.
In my program I have a large number of clients which are windows services running on laptops - that are often disconnected. Occasionally they come on line and I want them to receive updates based on user profiles. There are many types of notifications that require the client to perform some work on the local application (i.e. the laptop).
I realize that I could do this with a series of restful database queries, but since there are so many clients (upwards to 10,000) and there are lots of different notification types, I was curious if perhaps this was not a problem better suited for a messaging product like RabbitMQ or even 0MQ.
But how would one set this up. (let's assume in RabbitMQ?
Would each user be assigned their own queue?
Or is it preferable to have each queue be a distinct notification type and you would use some combination of direct exchanges or filtering messages based on a routing key, where the routing key could be a username.
Since each user may potentially have a different set of notifications based on their user profile, I am thinking that each client/consumer would have a specific message for each notification sitting on a queue waiting for them to come online and process it.
Is this the right way of thinking about the problem? Thanks in advance.
It will be easier for you to balance a lot of queues than filter long ones, so it's better to use queue per consumer.
Messages can have arbitrary headers and body so it is the right place for notification types.
Since you will be using long-living queues, waiting for consumers on disk - you better use lazy queues https://www.rabbitmq.com/lazy-queues.html (it's available since version 3.6.0)
This is similar to this other question but with a bit of a twist: I read in the specification that the message-id for AMQP messages should be set by the application itself, so in theory I could use that to guarantee a certain degree of uniqueness, right?
My main question is now: In what scope is that message-id garantueed to be unique? For the messages currently enqueued inside a specific queue? Over all queues? Over the universe? :-)
And is this behaviour standardized? I plan to use RabbitMQ here, but it would be nice to have something not vendor specifc :-)
Thanks.
Another suggestion is according to the dump pipes - smart endpoints school of thought.
You could handle uniqueness in your application, using some sort of shared state.
We had the same problem when switching from Gearman to RabbitMQ. We use memcached to keep track of unique message ID's posted and consumers drop messages for which the message ID is already stored in memcache (duplicates). You could also check memcache before putting it on the queue altogether.
This frees you from using this feature in your message bus layer (so you can more easily switch between brokers, also those who do not guarantee uniqueness)
Message Id is application-specific only and may be not unique at all. You have to take care of uniqueness by yourself.