Lets say a Redis node is configured with an eviction policy of LRU. Whenever an key-eviction takes place, is there way to know which keys are evicted?
Using this information, I would like to build a system where evicted keys are streamed to another database.
Did you take a look at Redis keyspace notifications. https://redis.io/docs/manual/keyspace-notifications/
With the RedisGears module you can react to these keyspace notifications.
In RedisGears 2 you can register your consumer, and then push information from there to the stream. But this is happening after the eviction has happened.
//Second argument is to set a prefix for keys you want to filter on
redis.register_notifications_consumer("eviction_consumer", "", function(client, data) {
//Do checks here
client.call('XADD', data.key, ...);
});
There is an extensive get started in the RedisGears repository.
Related
The keyspace notifications have been essential for a recent web api I've been
developing.
We have redis setup in azure. The api mostly works, we use notifications to figure out if data on memory cache needs to be updated.
Right now, we want to handle notifying the flush event to clear local memory cache if redis database is flushed. But we can not get the flushdb event by Keyspace notification.
And the keyspace events is enable as "AKE". "AKE" string means all the events.
PS: we can get notification with 'set' event like '__keyevent#2__:set'
The subscription code is like below.
subscriber.Subscribe(
"*",
(channel, value) =>
{
// Some codes here
});
Just as the other answer mentioned, there's no such notification.
After all, Keyspace Notification is a notification for events on a single key. Each notification is associated with a key. For keyspace event, the key name is part of the channel name. For keyevent event, the key name is the message.
PUBLISH __keyspace#0__:key_name comamnd
PUBLISH __keyevent#0__:command key_name
Each command that sending a notification must have a key as argument. e.g. del key, set key val. However, the flushdb command has no key as argument. The command doesn't affect a single key. Instead, it removes all keys in the database. So there's no such notification for it. Otherwise, what do you expect from the channel? All keys that have been removed? It's not a good idea.
However, you can simulate an event for flushdb
set a special key, e.g. flushdb-event: set flushdb-event 0
subscribe on the corresponding channel: subscribe __keyspace#0__:flushdb-event
set the special key before you call flushdb: set flushdb-event 1
In this way, you can get the simulated flushdb notification.
According to Redis Documentation, there is no notification for Flushdb.
I think you have a couple of options.
You could call the INFO command periodically and check for a change in the number of flushdb or flushall calls. Here is the output from INFO that I am referring to...
Commandstats
cmdstat_flushall:calls=2,usec=112,usec_per_call=56.00
cmdstat_flushdb:calls=1,usec=110,usec_per_call=52.00
You could run the MONITOR command and parse the output. Note that this is not really a good option - it has heavy performance impact on the server side and would require a lot of processing on the client side.
We have a requirement that we need to get a notification on changes to a Redis data structure. Based on my research I found out that I can use Redis key space notifications for doing the same. However, Redis key space notifications send the events to Redis pub/sub channel which is fire and forget i.e once the clients lose the connections all the events till the connection is up again are lost.
Redis streams solve this problem. Also I want to use consumer group feature of Redis streams. So is there any way that Redis key space notifications can be pushed to Redis streams instead of Redis pub/sub channel?
The only way this can be done afaik with the current - Redis v5.0.3 - is to use the Modules API to develop a module that registers for keyspace notifications, processes them and adds the relevant messages into the Stream.
With RedisGears it's pretty simple to register a listener that will automatically write each event to a Stream.
e.g. the following register.py script will write for each HSET or HMSET call on person:* key prefix an event to mystream Stream.
register.py:
GearsBuilder() \
.foreach(lambda x: execute('XADD', "mystream", '*', *sum([[k,v] for k,v in x.items()],[]))) \
.register(prefix="person:*", eventTypes=['HSET', 'HMSET'])
To run it all you need to do is call:
$ gears-cli run register.py
I have a use case in which I want to enable notification only for a certain set of keys, so that when those keys expire I can get a notification from redis.
I have followed this answer to implement this.
I have set parameter notify-keyspace-events to "Ex"
To accomplish this I am adding keys that I want notification for in DB-0 and the other keys in DB-1. But I am recieveing notification for both the DBs. Is there any way to just get notification from a particular DB?
According to redis documentation :
"Redis can notify Pub/Sub clients about events happening in the key space.
This feature is documented at http://redis.io/topics/notifications
For instance if keyspace events notification is enabled, and a client
performs a DEL operation on key "foo" stored in the Database 0, two
messages will be published via Pub/Sub:
PUBLISH keyspace#0:foo del
PUBLISH keyevent#0:del foo
"
But I am receiving notification from both DB-0 and DB-1.
PS : I know I can filter keys in my application, but I store too many expiring keys in redis and sending notification for all the expiring will increase load on my redis server.
I think you subscribed to a pattern that matches all DBs' notification message, e.g. PSUBSCRIBE __key*__:*.
In fact, you can specify the db index in the subscribed pattern: PSUBSCRIBE __keyspace#0__:* and PSUBSCRIBE __keyevent#0__:*. This way, you'll only received notification of DB0.
I'm implementing redis Keyspace notifications in my application which is having 10 instances on our production environment.
My pubsub listens for expired event in map1 and decrements in map2 based on that.
This works fine on my local machine. My issue is that when I deploy my application with multiple instances , I think all instances will read expired event and all will decrement the key whereas I want to restrict that only 1 instance should decrement.
Is there any way to achieve this ?
Your listeners will have to coordinate the decrement somehow. You can do that with some sort of locking, but a simpler way perhaps would be embed a notion of version/timestamp into this logic. Here's what I had in mind.
What if you include a timestamp in your "map2"? An expired event has it's own timestamp, so you can have the listeners check-and-set against that (tip: I'd use Lua for the CAS). This will prevent race-like conditions and multiple decrements in one go.
Note: Redis PubSub is amazing, but note that your current solution does not ensure the decrement in "map2" in case a message is lost. In the very near future, Redis will offer the Stream data type, that is much more suitable for that type of job. Specifically, the Stream Consumer Groups functionality is IMO just what you need here for replacing keyspace notifications.
We have some Redis keys with a given TTL that we would like to subscribe to and take action upon once the TTL expires (a la job scheduler).
This works well in a single-host environment, and when you subscribe in ServiceStack, using its Redis client, to '__keyspace#0__:expired', that service will pick it and take action. That's fantastic...
... until you have a high-availability topology set up, with more than one API instance in that cluster. Then every single host appears to be picking up on that message and potentially doing things with it.
I know keyspace notifications don't work exactly the same as traditional pub/sub or messaging-layer events, but is there a way to perform some kind of acknowledgement on these kinds of events, so that, at the end of the day, only one host will carry on with the task?
Otherwise, is there a way to delay a message publishing?
Thanks!
As describe in https://redis.io/topics/notifications
very node of a Redis cluster generates events about its own subset of the keyspace as described above. However, unlike regular Pub/Sub communication in a cluster, events' notifications are not broadcasted to all nodes. Put differently, keyspace events are node-specific. This means that to receive all keyspace events of a cluster, clients need to subscribe to each of the nodes.
So client should create separate connection to each node to get redis keyspace notification.
My understanding of your question: You need an event based unicast notification whenever a key is expired.
This solution will be helpful to you if above assumption is correct. It's kind of crude solution but works!
Solution:
You need to put(may be using a service/thread) the expired keys in the Redis List/queue. Then blocking B*POP operation from the client instances on this list/queue will give you what you want!
How does it work?
Let's assume, a single background thread will continuously push the expired keys into a redis list/queue. The cluster of API instances will be calling blocking pop on this list/queue.
Since, blocking pop operation on each item of redis list will be consumed by only one client, only one API instance will the get the notification of expired key!!!
Ref:
List pop operation: https://redis.io/commands/lpop
Similar problem with pub/sub: Competing Consumer on Redis Pub/Sub supported?