I am working on a node JS app that connect to redis server and subscribe to a channel to get the messages.
There is a bit of confusion, should we really enable "key space notifications" on redis config to get the events in client
The same scenario I have tried using rdis cli, with which i see "key space notifications" are not enabled at the same time I have subscribed to a channel with a pattern, so when ever I publish a message from the other client, I am able to capture that event in subscribed client.
Is the "key space notifications" mandatory , but the POC says other way.
Does any one know what should be the right approach here, subscribing to channel is suffice to get messages, and its nothing to do with "key-space-notifications" ??
From Redis Keyspace Notifications
Keyspace notifications allow clients to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way.
Examples of events that can be received are:
All the commands affecting a given key.
All the keys receiving an LPUSH operation.
All the keys expiring in the database 0.
Events are delivered using the normal Pub/Sub layer of Redis, so clients implementing Pub/Sub are able to use this feature without modifications.
So, if you need just pub/sub, there is no need of extra configuration regarding Keyspace Notifications
Related
I have few questions, I could not find the answers from Redis tutorial
1) how can I view/check values of a Redis PubSub channel? Monitor command is there to debug Redis, but I want to check what previously has pushed to channel.
2) What is the exact differece between channel and a queue?
3) how can I monitor Redis cluster in a free web based application?
1) You cannot view/check values that were published on a channel in the past. You can think of pubsub as fire and forget. Redis publishes a messages on a channel to clients which have subscribed to it, but does not persist the message for future reference. You can only monitor the messages published in realtime
2) Channel is a reference used by Redis to know which clients have subscribed to received messages published on that channel.
Queue is a data structure which stores values, these values can be accessed in future in FIFO order. So if you are using a queue for messaging, the messages will remain in the queue until you explicitly delete them
3) IMO there isn't any great free monitoring tool for Redis out there. See some available options here
As a side, regarding questions 1) and 2) : in case you are looking for reliable messaging, check out Redis Streams.
We have a requirement that we need to get a notification on changes to a Redis data structure. Based on my research I found out that I can use Redis key space notifications for doing the same. However, Redis key space notifications send the events to Redis pub/sub channel which is fire and forget i.e once the clients lose the connections all the events till the connection is up again are lost.
Redis streams solve this problem. Also I want to use consumer group feature of Redis streams. So is there any way that Redis key space notifications can be pushed to Redis streams instead of Redis pub/sub channel?
The only way this can be done afaik with the current - Redis v5.0.3 - is to use the Modules API to develop a module that registers for keyspace notifications, processes them and adds the relevant messages into the Stream.
With RedisGears it's pretty simple to register a listener that will automatically write each event to a Stream.
e.g. the following register.py script will write for each HSET or HMSET call on person:* key prefix an event to mystream Stream.
register.py:
GearsBuilder() \
.foreach(lambda x: execute('XADD', "mystream", '*', *sum([[k,v] for k,v in x.items()],[]))) \
.register(prefix="person:*", eventTypes=['HSET', 'HMSET'])
To run it all you need to do is call:
$ gears-cli run register.py
I have a use case in which I want to enable notification only for a certain set of keys, so that when those keys expire I can get a notification from redis.
I have followed this answer to implement this.
I have set parameter notify-keyspace-events to "Ex"
To accomplish this I am adding keys that I want notification for in DB-0 and the other keys in DB-1. But I am recieveing notification for both the DBs. Is there any way to just get notification from a particular DB?
According to redis documentation :
"Redis can notify Pub/Sub clients about events happening in the key space.
This feature is documented at http://redis.io/topics/notifications
For instance if keyspace events notification is enabled, and a client
performs a DEL operation on key "foo" stored in the Database 0, two
messages will be published via Pub/Sub:
PUBLISH keyspace#0:foo del
PUBLISH keyevent#0:del foo
"
But I am receiving notification from both DB-0 and DB-1.
PS : I know I can filter keys in my application, but I store too many expiring keys in redis and sending notification for all the expiring will increase load on my redis server.
I think you subscribed to a pattern that matches all DBs' notification message, e.g. PSUBSCRIBE __key*__:*.
In fact, you can specify the db index in the subscribed pattern: PSUBSCRIBE __keyspace#0__:* and PSUBSCRIBE __keyevent#0__:*. This way, you'll only received notification of DB0.
We have some Redis keys with a given TTL that we would like to subscribe to and take action upon once the TTL expires (a la job scheduler).
This works well in a single-host environment, and when you subscribe in ServiceStack, using its Redis client, to '__keyspace#0__:expired', that service will pick it and take action. That's fantastic...
... until you have a high-availability topology set up, with more than one API instance in that cluster. Then every single host appears to be picking up on that message and potentially doing things with it.
I know keyspace notifications don't work exactly the same as traditional pub/sub or messaging-layer events, but is there a way to perform some kind of acknowledgement on these kinds of events, so that, at the end of the day, only one host will carry on with the task?
Otherwise, is there a way to delay a message publishing?
Thanks!
As describe in https://redis.io/topics/notifications
very node of a Redis cluster generates events about its own subset of the keyspace as described above. However, unlike regular Pub/Sub communication in a cluster, events' notifications are not broadcasted to all nodes. Put differently, keyspace events are node-specific. This means that to receive all keyspace events of a cluster, clients need to subscribe to each of the nodes.
So client should create separate connection to each node to get redis keyspace notification.
My understanding of your question: You need an event based unicast notification whenever a key is expired.
This solution will be helpful to you if above assumption is correct. It's kind of crude solution but works!
Solution:
You need to put(may be using a service/thread) the expired keys in the Redis List/queue. Then blocking B*POP operation from the client instances on this list/queue will give you what you want!
How does it work?
Let's assume, a single background thread will continuously push the expired keys into a redis list/queue. The cluster of API instances will be calling blocking pop on this list/queue.
Since, blocking pop operation on each item of redis list will be consumed by only one client, only one API instance will the get the notification of expired key!!!
Ref:
List pop operation: https://redis.io/commands/lpop
Similar problem with pub/sub: Competing Consumer on Redis Pub/Sub supported?
How long will messages published to a Redis channel stay there ? Also, is there is a way to configure the max. lifetime of a message per channel ? Is there a way to control the channel size or does the channel continues to store messages as long as the Redis server has free memory ?
Redis pub/sub doesn't persist published messages. What you are looking for seems more like a message queue which can be implemented using a combination of pub/sub and lists. For more information see pattern sections in RPOPLPUSH command.