I have an app in which for each received message it publishes N in the same channel. I'm wondering whether it'd be better to have two channel, a Publish channel and a Receive channel.
What would it be the best practice?
Best practice is to have one channel per consumer/ queue. I would say that would extend to publishing on a different channel as consuming.
To add to the above answer, channels are essentially cost-free. A "channel" in RabbitMQ is just an integer that is tacked on to the raw packet indicating what channel it is for. The underlying TCP stream is still the same. There is a negligible additional amount of resources consumed by keeping track of all the different channels on the client and server side, but this is really only a problem if you are creating close to the maximum number of channels.
Related
Let‘s assume I have a simple chat application where some clients can post messages to some channel X and some clients want to subscribe to all messages in X (channel-based multicasting).
Would I create a Fanout Exchange for each channel or is there a way to utilize Topic for this use-case?
Also a new client that is added to a channel X should be able to read the last message from X. How is that done?
Would it scale for millions of channels?
(Or should I rather look at MQTT?)
A RabbitMQ Topic Exchange (using pub/sub) is definitely an option to create this kind of distribution pattern... producers would ensure their messages' routing key corresponds to "Channel X", and consumers (via their queues) would bind to this Exchange with a matching pattern. This should mean you don't need to have a specific Exchange for each of your channels. But unknown whether it can scale to millions of channels.
If considering other messaging technologies, maybe take a look at Solace? (FYI, I work for them). We actually have a free Udemy dev course where you build a chat app! https://www.udemy.com/fundamentals-of-solace-development/. Solace supports MQTT natively if you want to go that route, and also has a built-in Replay capability that could be used to retrieve the last n messages on a given topic.
We are using redis message bus and handling messages using a channel. But if our application is deployed in multiple instances then the request and response is passed to all the instances. To avoid this scenario which of the below approach is better?
Create a channel for each instance of the application
Create a channel for each user
Any suggestions will be highly appreciated
The limiting factor here is the number of subscribers to the same channel. Number of channels can be large as such. So you can choose the granularity accordingly. Read more here:
https://groups.google.com/forum/#!topic/redis-db/R09u__3Jzfk
All the complexity on the end is on the PUBLISH command, that performs
an amount of work that is proportional to:
a) The number of clients receiving the message.
b) The number of clients subscribed to a pattern, even if they'll not
match the message.
This means that if you have N clients subscribed to 100000 different
channels, everything will be super fast.
If you have instead 10000 clients subscribed to the same channel,
PUBLISH commands against this channel will be slow, and take maybe a
few milliseconds (not sure about the actual time taken). Since we have
to send the same message to everybody.
Similar question asked before : How does Redis PubSub subscribe mechanism works?
I want to create a Publish-Subscribe infrastructure in which every subscriber will listen to multiple (say 100k) channels.
I think to use Redis PubSub for that purpose but I'm not sure if subscribing to thousands of channels is the best practice here.
To answer this I want to know how subscribing mechanism in Redis works in the background.
Another option is to create a channel per subscriber and put some component in between, that will get all messages and publish it to relevant channels.
Any other Idea?
Salvatore/creator of Redis has answered this here: https://groups.google.com/forum/#!topic/redis-db/R09u__3Jzfk
All the complexity on the end is on the PUBLISH command, that performs
an amount of work that is proportional to:
a) The number of clients receiving the message.
b) The number of clients subscribed to a pattern, even if they'll not
match the message.
This means that if you have N clients subscribed to 100000 different
channels, everything will be super fast.
If you have instead 10000 clients subscribed to the same channel,
PUBLISH commands against this channel will be slow, and take maybe a
few milliseconds (not sure about the actual time taken). Since we have
to send the same message to everybody.
I am using amqplib in Node.js, and I am not clear about the best practices in my code.
Basically, my current code calls the amqp.connect() when the Node server starts up, and then uses a different channel for each producer and each consumer, never actually closing any of them. I'd like to know if that makes any sense, or should I create the channel, publish and close it every time I want to publish a message. And what about the connection? Is that a "good practice" to connect once, and then keep it open for the lifetime of my server?
On the Consumer side - can I use a single connection and a single channel to listen on multiple queues?
Thank you for any clarifications
In general, it's not a good practice to open and close connections and channels per message. Connections are long lived and it takes resources to keep opening and closing them. For channels, they share the TCP connection with the connection so they are more lightweight, but they will still consume memory and definitely should not be left open after done using them.
It is recommended to have a channel per thread, and a channel per consumer. But for publishing it is totally ok to use the same channel. But keep in mind that depending on the operations, the protocol might kill the channel in certain situations (e.g. queue existence check), so prepare for that. There is also soft (configurable) and hard (usually 65535) limits on the maximum number of channels on many of the client implementations.
So to sum up, depending on your use case use one to a few connections, open channels when you need them and share them when it makes sense, but remember to close them when done.
The rabbitmq documentation explains the nature of the connections and channels (end of the document). And the accepted answer on this question has good information on the subject.
I've been looking at using Redis Pub/Sub as a replacement to RabbitMQ.
From my understanding Redis's pub/sub holds a persistent connection to each of the subscribers, and if the connection is terminated, all future messages will be lost and dropped on the floor.
One possible solution is to use a list (and blocking wait) to store all the message and pub/sub as just a notification mechanism. I think this gets me most of the way there, but I still have some concerns about the failure cases.
what happens when a subscriber dies, and comes back online, how should it process all it's pending messages?
when a malformed message comes though the system, how do you handle those exceptions? DeadLetter Queue?
is there a standard practice to implementing a retry policy?
When a subscriber (consumer) dies, your list will continue to grow until the client returns. Your producer could trim the list (from either side) once it reaches a specific limit, but that is something you would need to handle at the application level. If you include a timestamp within each message, your consumer can then act on the age of a message, assuming you have application logic you want to enforce on message age.
I'm not sure how a malformed message would enter the system, as the connection to Redis is usually TCP with the its integrity assurances. But if this happens, perhaps due to a bug in message encoding at the producer layer, you could provide a general mechanism for handling errors by keeping a queue-per-producer that received consumer's exception messages.
Retry policies will depend greatly on your application needs. If you need 100% assurance that a message has been received and processed, then you should consider using Redis transactions (MULTI/EXEC) to wrap the work done by a consumer, so you can ensure that a client doesn't remove a message unless it has completed its work. If you need explicit acknowlegement, then you could use an explicit ACK message on a queue dedicated to the producer process(es).
Without knowing more about your application needs, it's hard to know how to choose wisely. Generally, if your messages require full ACID protection, then you probably also need to use redis transactions. If your messages are only meaningful when they are timely, then transactions may not be needed. It sounds as though you can't tolerate dropped messages, so your approach of using a list is good. If you need to implement a priority queue for your messages, you can use the sorted set (the Z-commands) to store your messages, using their priority as the score value, along with a polling consumer.
If you want a pub/sub system where subscribers won't lose messages when they die, consider using Redis Streams instead of Redis Pub/sub.
Redis Streams have their own architecture and pros/cons to Redis Pub/sub. With Redis Streams, a subscriber can issue the command:
the last message I received was X, now give me the next message;
if there is no new message, then wait for one to arrive.
Antirez's article linked above is a good intro to Redis streams with more info.
What I did is use a sorted set using the timestamp as the score and the key to the data as the member value. I use the score from the last item to retrieve the next few ones and then get the keys. Once the work is done I wrap both the zrem and the del in a MULTI/EXEC transaction.
Essentially what Edward said, but with the twist of storing the keys in the sorted set, as my messages can be pretty big.
Hope this helps!