(The documentation is now updated with this information.) As of December, 2019, the Redis documentation on keyspace notifications (https://redis.io/topics/notifications) includes a lengthy list of Redis commands and the keyspace notifications that these commands fire. There are no stream operations (XADD, XDEL, XTRIM etc.) in the list. Is that because these operations do not cause keyspace notifications to fire, or is it because the list has not been updated to reflect the relatively new stream commands?
And if XADD, XDEL and XTRIM, specifically, do generate keyspace notifications, what are they?
Yes - all stream operations generate keyspace notifications. To enable the category either use the 't' flag, or the all-inclusive 'A', for the following:
XADD generates an xadd event, possibly followed an xtrim event when used with the MAXLEN subcommand.
XDEL generates a single xdel event even when multiple entries are are deleted.
XGROUP CREATE generates an xgroup-create event.
XGROUP DELCONSUMER generates an xgroup-delconsumer event.
XGROUP DESTROY generates an xgroup-destroy event.
XGROUP SETID generates an xgroup-setid event.
XSETID generates an xsetid event.
XTRIM generates an xtrim event.
Note: I've taken this as a cue to update the docs with the missing info - thanks for noticing and asking #Dan Menes :)
Related
The keyspace notifications have been essential for a recent web api I've been
developing.
We have redis setup in azure. The api mostly works, we use notifications to figure out if data on memory cache needs to be updated.
Right now, we want to handle notifying the flush event to clear local memory cache if redis database is flushed. But we can not get the flushdb event by Keyspace notification.
And the keyspace events is enable as "AKE". "AKE" string means all the events.
PS: we can get notification with 'set' event like '__keyevent#2__:set'
The subscription code is like below.
subscriber.Subscribe(
"*",
(channel, value) =>
{
// Some codes here
});
Just as the other answer mentioned, there's no such notification.
After all, Keyspace Notification is a notification for events on a single key. Each notification is associated with a key. For keyspace event, the key name is part of the channel name. For keyevent event, the key name is the message.
PUBLISH __keyspace#0__:key_name comamnd
PUBLISH __keyevent#0__:command key_name
Each command that sending a notification must have a key as argument. e.g. del key, set key val. However, the flushdb command has no key as argument. The command doesn't affect a single key. Instead, it removes all keys in the database. So there's no such notification for it. Otherwise, what do you expect from the channel? All keys that have been removed? It's not a good idea.
However, you can simulate an event for flushdb
set a special key, e.g. flushdb-event: set flushdb-event 0
subscribe on the corresponding channel: subscribe __keyspace#0__:flushdb-event
set the special key before you call flushdb: set flushdb-event 1
In this way, you can get the simulated flushdb notification.
According to Redis Documentation, there is no notification for Flushdb.
I think you have a couple of options.
You could call the INFO command periodically and check for a change in the number of flushdb or flushall calls. Here is the output from INFO that I am referring to...
Commandstats
cmdstat_flushall:calls=2,usec=112,usec_per_call=56.00
cmdstat_flushdb:calls=1,usec=110,usec_per_call=52.00
You could run the MONITOR command and parse the output. Note that this is not really a good option - it has heavy performance impact on the server side and would require a lot of processing on the client side.
We have a requirement that we need to get a notification on changes to a Redis data structure. Based on my research I found out that I can use Redis key space notifications for doing the same. However, Redis key space notifications send the events to Redis pub/sub channel which is fire and forget i.e once the clients lose the connections all the events till the connection is up again are lost.
Redis streams solve this problem. Also I want to use consumer group feature of Redis streams. So is there any way that Redis key space notifications can be pushed to Redis streams instead of Redis pub/sub channel?
The only way this can be done afaik with the current - Redis v5.0.3 - is to use the Modules API to develop a module that registers for keyspace notifications, processes them and adds the relevant messages into the Stream.
With RedisGears it's pretty simple to register a listener that will automatically write each event to a Stream.
e.g. the following register.py script will write for each HSET or HMSET call on person:* key prefix an event to mystream Stream.
register.py:
GearsBuilder() \
.foreach(lambda x: execute('XADD', "mystream", '*', *sum([[k,v] for k,v in x.items()],[]))) \
.register(prefix="person:*", eventTypes=['HSET', 'HMSET'])
To run it all you need to do is call:
$ gears-cli run register.py
I'm implementing redis Keyspace notifications in my application which is having 10 instances on our production environment.
My pubsub listens for expired event in map1 and decrements in map2 based on that.
This works fine on my local machine. My issue is that when I deploy my application with multiple instances , I think all instances will read expired event and all will decrement the key whereas I want to restrict that only 1 instance should decrement.
Is there any way to achieve this ?
Your listeners will have to coordinate the decrement somehow. You can do that with some sort of locking, but a simpler way perhaps would be embed a notion of version/timestamp into this logic. Here's what I had in mind.
What if you include a timestamp in your "map2"? An expired event has it's own timestamp, so you can have the listeners check-and-set against that (tip: I'd use Lua for the CAS). This will prevent race-like conditions and multiple decrements in one go.
Note: Redis PubSub is amazing, but note that your current solution does not ensure the decrement in "map2" in case a message is lost. In the very near future, Redis will offer the Stream data type, that is much more suitable for that type of job. Specifically, the Stream Consumer Groups functionality is IMO just what you need here for replacing keyspace notifications.
We have some Redis keys with a given TTL that we would like to subscribe to and take action upon once the TTL expires (a la job scheduler).
This works well in a single-host environment, and when you subscribe in ServiceStack, using its Redis client, to '__keyspace#0__:expired', that service will pick it and take action. That's fantastic...
... until you have a high-availability topology set up, with more than one API instance in that cluster. Then every single host appears to be picking up on that message and potentially doing things with it.
I know keyspace notifications don't work exactly the same as traditional pub/sub or messaging-layer events, but is there a way to perform some kind of acknowledgement on these kinds of events, so that, at the end of the day, only one host will carry on with the task?
Otherwise, is there a way to delay a message publishing?
Thanks!
As describe in https://redis.io/topics/notifications
very node of a Redis cluster generates events about its own subset of the keyspace as described above. However, unlike regular Pub/Sub communication in a cluster, events' notifications are not broadcasted to all nodes. Put differently, keyspace events are node-specific. This means that to receive all keyspace events of a cluster, clients need to subscribe to each of the nodes.
So client should create separate connection to each node to get redis keyspace notification.
My understanding of your question: You need an event based unicast notification whenever a key is expired.
This solution will be helpful to you if above assumption is correct. It's kind of crude solution but works!
Solution:
You need to put(may be using a service/thread) the expired keys in the Redis List/queue. Then blocking B*POP operation from the client instances on this list/queue will give you what you want!
How does it work?
Let's assume, a single background thread will continuously push the expired keys into a redis list/queue. The cluster of API instances will be calling blocking pop on this list/queue.
Since, blocking pop operation on each item of redis list will be consumed by only one client, only one API instance will the get the notification of expired key!!!
Ref:
List pop operation: https://redis.io/commands/lpop
Similar problem with pub/sub: Competing Consumer on Redis Pub/Sub supported?
I have a program that utilizes a redis key with an expire time set. I want to detect when there is a new entry to the data set. I can tell when there is a removal by listening for the "expired" event, but the "set" and "expire" events are fired every time the key is set, even if it's already in the database.
Is there a keyspace event for a NEW key appearing?
There's no keyspace configuration that detects that a key was overwritten vs. newly added.
If you are primarily using the SET command, you may be able to take advantage of the NX option and publish a custom event based on the result. Obviously this isn't an ideal approach, but it's something.
https://redis.io/commands/set
Example of a custom event:
PUBLISH __keyevent#0__:new_data_entry new_key
Details on that here: https://redis.io/topics/notifications#type-of-events
Hope that helps.