I am working on redis SMQ persistence. My questions here is, While publisher publishing the messages, consumer has stopped suddenly. When consumer connects again, is it possible to subscribe messages from where it has stopped?
No - Redis' Pub/Sub has no persistence, and once a message has been published, it is sent only to the connected subscribed clients. Afterwards, the message is gone forever.
With standard Pub/Sub you can use Lua scripts to persist your message. You need to check whether you have a listener on channel or not. If not then storing your message with channel key on redis . When the subscriber cames back it checks if there is anything for him based on channel key. Second option is to use Redis Stream. Check this gist.
Plz use 2 redis connections: 1 pubsub, second - LPOP/RPOP
Related
I'm looking at turning a monolith application into a microservice-oriented application and in doing so will need a robust messaging system for interprocesses-communication. The idea is for the microserviceprocesses to be run on a cluster of servers for HA, with requests to be processed to be added on a message queue that all the applications can access. I'm looking at using Redis as both a KV-store for transient data and also as a message broker using the ServiceStack framework for .Net but I worry that the concept of eventual consistency applied by Redis will make processing of the requests unreliable. This is how I understand Redis to function in regards to Mq:
Client 1 posts a request to a queue on node 1
Node 1 will inform all listeners on that queue using pub/sub of the existence of
the request and will also push the requests to node 2 asynchronously.
The listeners on node 1 will pull the request from the node, only 1 of them will obtain it as should be. An update of the removal of the request is sent to node 2 asynchronously but will take some time to arrive.
The initial request is received by node 2 (assuming a bit of a delay in RTT) which will go ahead and inform listeners connected to it using pub/sub. Before the update from node 1 is received regarding the removal of the request from the queue a listener on node 2 may also pull the request. The result being that two listeners ended up processing the same request, which would cause havoc in our system.
Is there anything in Redis or the implementation of ServiceStack Redis Mq that would prevent the scenario described to occur? Or is there something else regarding replication in Redis that I have misunderstood? Or should I abandon the Redis/SS approach for Mq and use something like RabbitMQ instead that I have understood to be ACID-compliant?
It's not possible for the same message to be processed twice in Redis MQ as the message worker pops the message off the Redis List backed MQ and all Redis operations are atomic so no other message worker will have access to the messages that have been removed from the List.
ServiceStack.Redis (which Redis MQ uses) only supports Redis Sentinel for HA which despite Redis supporting multiple replicas they only contain a read only view of the master dataset, so all write operations like List add/remove operations can only happen on the single master instance.
One notable difference from using Redis MQ instead of specific purpose MQ like Rabbit MQ is that Redis doesn't support ACK's, so if the message worker process that pops the message off the MQ crashes then it's message is lost, as opposed to Rabbit MQ where if the stateful connection of an un Ack'd message dies the message is restored by the RabbitMQ server back to the MQ.
I'm using Redis Pub/Sub implementation to exchange messages between two projects. I have a few channels subscribing the same queue. When both publisher and subscriber are running, everything goes well. When I have only the publisher working(and a lot of messages are published), I would expect that when the subscriber starts, it would read all the messages that were enqueued previously. But what happens is that Redis does not keep the messages if there is no subscriber. Is there any configuration I could use to keep the messages until a subscriber dequeue them?
Redis currently doesn't behave like an MQTT broker with his "retain" flag.
If the subscription occur after the message has been published, It will be missed for the subscriber and lost forever.
I have an exercise about Redis Pubsub like the following:
In case the publisher publish a message but the subscriber has not received the server has crashed. How to subscriber receive that message when restart server?.
Please help me, thank you !
In this case, the message is gone forever.
Redis only has limited support for PUBSUB scenario. Besides your case, if the connection between Redis and the client is lost, the client will also lose all published messages.
If you need more reliable PUBSUB tools, you should try other stuffs, e.g. Kafka, RabbitMQ.
I found this answer to determine how to broadcast a message from a controller to a channel:
How to broadcast a message from a Phoenix Controller to a Channel?
What I need however is to publish a message to a channel from a mix task or potentially a Toniq worker. In this case the process may or may not be on the same machine.
Now if I understand correctly channels use either some in memory pub/sub or can be backed by other distributed queues.
If this is correct how could I have the channels backed by redis and have my worker/task publish to redis that would trigger a broadcast to the connected sockets?
The idea is:
I have N WCF services which connected and subscribed to the same Redis message channel. These services use this channel to exchange messages to sync some caches and other data.
How each service can ignore its own messages? I.e. how to publish to all but me?
It looks like Redis PUB/SUB doesn't support such filtration. So, the solution is to use set of individual channels for every publisher and common channel for subscription synchronization between them. Here is an golang example of no-echo chat application.