I am trying to create chat rooms app and use redis pub/sub broadcasting messages across subscribers. Each meeting room is 1 channel. The problem is I can have infinite number of channels over time.
Is there any limit for max number of channels I can have in redis?
Is there any way to auto expire/delete channels which have not received any messages for publishing?
Is there any limit for max number of channels I can have in redis?
Redis doesn't set any limit on the number of channels. Channels are saved into a hash, whose size is of type long. So in theory, the limit is 2**63 on a 64 bits machine. However, since your memory is limited, you cannot reach the limit. In a word, practically, there's no limit.
Is there any way to auto expire/delete channels which have not received any messages for publishing?
There's no way to do that. Redis deletes a channel, only when all clients of the channel have unsubscribed it, i.e. when a channel has no subscribers, Redis remove the channel automatically. If there's at least one client subscribing it, Redis keep the channel even if there's no message published to it.
Related
Could anyone tell me whats the maximum number of concurrent channels Redis pub-sub can support?. Is there any cap to the number of subscribers and publishers
Redis uses a dict, same struct as for keys, to store channel subscriptions, both per client and for all clients (keeps a per-subscription hash with a list of clients subscribed), so it is up to 2^32 channel subscriptions in total.
It uses a list to store pattern subscriptions per client, so it is theoretically limited only by the node memory available.
However, in general, you can have infinite channels. Think of a channel as a label when a message is published. Messages are never stored. When the message is published, Redis will look for the clients subscribed to that channel, and test for every pattern subscription. The channel really exists only while the message is being published.
As there are pattern subscriptions, there are unlimited 'logical' channels.
Just in events notifications we have 2^32 * databases * key event types possible 'logical' channels.
Regarding the number of subscribers and publishers, it is limited by the maxclients setting, 10,000 by default. There is no limitation for subscribers and publishers, but the maximum clients (connections) limit applies.
As indicated by #Roman, there are buffer limitations, but this refers mostly to throughput (message processing).
Pub/Sub clients have a default hard limit of 32 megabytes and a soft limit of 8 megabytes per 60 seconds.
Is that what you have been looking for?
documentation
We are using redis message bus and handling messages using a channel. But if our application is deployed in multiple instances then the request and response is passed to all the instances. To avoid this scenario which of the below approach is better?
Create a channel for each instance of the application
Create a channel for each user
Any suggestions will be highly appreciated
The limiting factor here is the number of subscribers to the same channel. Number of channels can be large as such. So you can choose the granularity accordingly. Read more here:
https://groups.google.com/forum/#!topic/redis-db/R09u__3Jzfk
All the complexity on the end is on the PUBLISH command, that performs
an amount of work that is proportional to:
a) The number of clients receiving the message.
b) The number of clients subscribed to a pattern, even if they'll not
match the message.
This means that if you have N clients subscribed to 100000 different
channels, everything will be super fast.
If you have instead 10000 clients subscribed to the same channel,
PUBLISH commands against this channel will be slow, and take maybe a
few milliseconds (not sure about the actual time taken). Since we have
to send the same message to everybody.
Similar question asked before : How does Redis PubSub subscribe mechanism works?
I have a requirement where I want to publish the data to more than 100,000 channels and the subscribers will subscribe to whatever channels they want to.
So I want to know is there a hard limit to the number of channels that can be available on Redis?
If I want to benchmark the performance of Redis with all there channels and the subscribers to the channels is there any available tool that I can use?
So I want to know is there a hard limit to the number of channels that can be available on Redis?
In practice, there's NO limit. You can have as many channels as possible, until the memory is full.
I want to create a Publish-Subscribe infrastructure in which every subscriber will listen to multiple (say 100k) channels.
I think to use Redis PubSub for that purpose but I'm not sure if subscribing to thousands of channels is the best practice here.
To answer this I want to know how subscribing mechanism in Redis works in the background.
Another option is to create a channel per subscriber and put some component in between, that will get all messages and publish it to relevant channels.
Any other Idea?
Salvatore/creator of Redis has answered this here: https://groups.google.com/forum/#!topic/redis-db/R09u__3Jzfk
All the complexity on the end is on the PUBLISH command, that performs
an amount of work that is proportional to:
a) The number of clients receiving the message.
b) The number of clients subscribed to a pattern, even if they'll not
match the message.
This means that if you have N clients subscribed to 100000 different
channels, everything will be super fast.
If you have instead 10000 clients subscribed to the same channel,
PUBLISH commands against this channel will be slow, and take maybe a
few milliseconds (not sure about the actual time taken). Since we have
to send the same message to everybody.
I need to implement sequential message processing on multiple consumers, but only one message per time on the queue. I have a lot of queues, but all of them are sequential and I need multiple consumers support for load balancing and redundancy. Anybody can tell whether it is real or not to limit number of unacknowledged message to 1 per queue?
Anybody can tell whether it is real or not to limit number of unacknowledged message to 1 per queue?
this isn't possible with multiple consumers. you can limit the number of unacknowledged messages using prefetch limit for a single channel, but not across multiple channels / consumers. it is tied to the channel of the consumer, not the queue.
the only way you can achieve this is with a single consumer and a single queue, using prefetch.
even then, you have no guarantee that the messages will arrive in the queue in the correct order.
(this is a fundamental difficulty with distributed systems of any kind, not a rabbitmq limitation)
look at the Message Sequencer and Resequencer patterns to try and put the messages back in order.
but even then, you're going to run into difficulty.
you'll also want to read up on idempotency so you don't re-process a message that has already been processed.
You should be able to configure your consumer to consume only X message(s) at time and same for your channel. Take a look at QOS or Consumer Prefetch
https://www.rabbitmq.com/consumer-prefetch.html
Here is an example, where multi-consumers will acknoledge only one message and channel allow only one message to be acknoledged (whatever how much consumers a plugged on it)
Channel channel = ...;
Consumer consumer1 = ...;
Consumer consumer2 = ...;
channel.basicQos(1, false); // Per consumer limit
channel.basicQos(1, true); // Per channel limit
channel.basicConsume("my-queue1", false, consumer1);
channel.basicConsume("my-queue2", false, consumer2);
Here, a consumer can acknoledge only one message each time, and the channel can only have one unacknoledged message. You didn't mention which language you use so you'll problably have to adapt this example.