Is it possible to have multiple categorized queues in RabbitMQ ?
Our application sends bulk SMS to phone numbers. when a client requests to send a very large number of SMSes , other normal client requests have to wait a lot. Is there a solution in RabbitMQ to have multiple queues ? ( So the worker can pick from all of them ? )
Yes. it is possible to do in RabbitMQ.
Since you didn't provide enough details about your problem, I'l give you a hight level overview of the things you might be interested in:
Take a look at topic exchange. It does exactly what you described.
Take a look at consumer priority.
And random exhcnage that allows you to spread messages between queues.
Related
We are building spark based jobs. Processing each message delivered by the queue takes time. There is a need to be able to reprioritize one already sent to the queue.
I am aware there is priority queue implementation available, but not sure how to re-prioritize the existing message in the queue?
One bad workaround is to push that message again as higher priority, so that it handled on priority. Later drop the message with same content which had low or no priority when it's turns comes next.
Is there a natural way we can handle this situation or any other queues that supports scenario better?
Unfortunately there isn't. Queues are to be considered as lists of messages in flight. It is not possible to delete/update them.
Your approach of submitting a higher priority message is the only feasible solution.
RabbitMQ is a messaging system (such as the postal one), it is not a DataBase or a storage service. The storage in form of queues is a necessary feature as much as the postal service needs storage for postcards in transit. It is optimized for the purpose and does not allow to access the messages easily.
Let‘s assume I have a simple chat application where some clients can post messages to some channel X and some clients want to subscribe to all messages in X (channel-based multicasting).
Would I create a Fanout Exchange for each channel or is there a way to utilize Topic for this use-case?
Also a new client that is added to a channel X should be able to read the last message from X. How is that done?
Would it scale for millions of channels?
(Or should I rather look at MQTT?)
A RabbitMQ Topic Exchange (using pub/sub) is definitely an option to create this kind of distribution pattern... producers would ensure their messages' routing key corresponds to "Channel X", and consumers (via their queues) would bind to this Exchange with a matching pattern. This should mean you don't need to have a specific Exchange for each of your channels. But unknown whether it can scale to millions of channels.
If considering other messaging technologies, maybe take a look at Solace? (FYI, I work for them). We actually have a free Udemy dev course where you build a chat app! https://www.udemy.com/fundamentals-of-solace-development/. Solace supports MQTT natively if you want to go that route, and also has a built-in Replay capability that could be used to retrieve the last n messages on a given topic.
We are using redis message bus and handling messages using a channel. But if our application is deployed in multiple instances then the request and response is passed to all the instances. To avoid this scenario which of the below approach is better?
Create a channel for each instance of the application
Create a channel for each user
Any suggestions will be highly appreciated
The limiting factor here is the number of subscribers to the same channel. Number of channels can be large as such. So you can choose the granularity accordingly. Read more here:
https://groups.google.com/forum/#!topic/redis-db/R09u__3Jzfk
All the complexity on the end is on the PUBLISH command, that performs
an amount of work that is proportional to:
a) The number of clients receiving the message.
b) The number of clients subscribed to a pattern, even if they'll not
match the message.
This means that if you have N clients subscribed to 100000 different
channels, everything will be super fast.
If you have instead 10000 clients subscribed to the same channel,
PUBLISH commands against this channel will be slow, and take maybe a
few milliseconds (not sure about the actual time taken). Since we have
to send the same message to everybody.
Similar question asked before : How does Redis PubSub subscribe mechanism works?
I want to create a Publish-Subscribe infrastructure in which every subscriber will listen to multiple (say 100k) channels.
I think to use Redis PubSub for that purpose but I'm not sure if subscribing to thousands of channels is the best practice here.
To answer this I want to know how subscribing mechanism in Redis works in the background.
Another option is to create a channel per subscriber and put some component in between, that will get all messages and publish it to relevant channels.
Any other Idea?
Salvatore/creator of Redis has answered this here: https://groups.google.com/forum/#!topic/redis-db/R09u__3Jzfk
All the complexity on the end is on the PUBLISH command, that performs
an amount of work that is proportional to:
a) The number of clients receiving the message.
b) The number of clients subscribed to a pattern, even if they'll not
match the message.
This means that if you have N clients subscribed to 100000 different
channels, everything will be super fast.
If you have instead 10000 clients subscribed to the same channel,
PUBLISH commands against this channel will be slow, and take maybe a
few milliseconds (not sure about the actual time taken). Since we have
to send the same message to everybody.
I have a amqp connector setup that listens on a single queue for JSON messages and is working fine. The business has dropped a use case that my application now needs to listen on multiple queues in order of priority. For example having three queues:
HighQ
NormalQ
LowQ
I want the mule connector to first read from HighQ until empty, then NormalQ until empty and LowQ until empty. Restarting from HighQ after every message.
I feel like this should be standard but my google foo is failing me.
Any pointers in the right direction?
In the usecase you specified I think it would be better to go with a single queue, but posting messages with 3 priority levels.
THis way the messages are always read in the order of their priority with the highest prioritymessage are always read first.
So you can make the message producers to post the messages onto the queue with 3 priority levels(say 9 for high , 4 for normal, 0 for low).
You inbound JMS endpoint will read all the messages with priority of 9 first. Then it will read all the messages with priotiy of 4 and then the messages with priority of 0.
Sample JMS Outbound posting messages with priority.
<jms:outbound-endpoint queue="StudioOUT" connector-ref="MyAppJMS" doc:name="JMS">
<set-property propertyName="Priority" value="9"/>
</jms:outbound-endpoint>
I hope this should address your scenario.
More on priority of JMS.
http://www.christianposta.com/blog/?p=289
Dealing with message priority is really something that your broker should handle. Dealing with this yourself can be tricky and cumbersome.
Processing queues sequentially in order to simulate priority seems like a bad idea. Lets say you've processed all messages from the high priority queue and start processing the normal priority queue.
While processing the normal priority queue new messages are coming in on the high priority queue. These high priority messages will be sitting there until both the normal and low priority queues are entirely processed.
You could probably improve your mechanism to handle situations like this a bit better, but it will be hard to make it bullet proof. You really don't want to deal with stuff like this yourself.
The JMS api has the concept of 'message priority' built in, but that's of little use to U if you aren't using a JMS broker.
If you're using rabbit mq then you should have a look at this stackoverflow post: rabbitmq-and-message-priority.
As Rabbit MQ queues are basically FIFO queues there's no easy way to use "real" prioritized message (such as in JMS).
There is however a plugin that claims to provide the functionality that you are looking for: rabbitmq-priority-queue.
According to the documentation the next version of RabbitMQ (3.5.0) will support prioritized queues out of the box.
If using the plugin isn't an option and if the priority of the messages is really important then I would not use the pattern you described using multiple queues. The pattern also doesn't scale very well if more priority levels are needed. I would opt to receive all the messages on a single channel (given that each message has a property that represents the priority level) and forward them to a (non amqp) new channel that handles the resequencing for you. An open-source product that could help you with this is Apache ActiveMQ but there are also other options available.