The documentation about Redis Keyspace Notifications http://redis.io/topics/notifications says near its end, that a key with timeout is removed from the database
"When the key is accessed by a command and is found to be expired."
..
Question: Is it enough to retrieve the very key, e.g. via KEYS *, or do I have to access the content the key refers to?
Background: The second process I omitted (the .. above) is a probabilistic process, and the real deletion of an expired key may be delayed, and thus the delivery of the EXPIRED event. I want to ensure the notification is given to a subscriber, so just accessing the keys would be easiest.
Redis implements a logic of periodic checking of keys for expiry and picks a number (100) of keys and checks them for their expiry.
What I understand is that your concerned with the fact that with above logic there would exist events which belong to expired keys which have not been deleted.
To avoid such a case checking the keys just for existence would delete them. Cost of REDIS calls should be kept in mind and hence a LUA script or bulk command should be designed which is invoked periodically and iterates a list of keys and run EXISTS command on them and cause automatic delete if they are expired.
To test this you would need a large dataset.
Related
I have a database system which processes read operations against logical objects that are identified by OIDs (object identifiers). Example is 1.2.4.3.544. There are users identified by GUIDs. API gets results from DB and puts them into Redis. Key example is
SomePrefix_<oid>1.2.4.3.544</oid>_somedetails_<user>1f0c6cfe-ee9d-472c-b320-190df55f9527</user>
There are about a couple of hundreds unique OIDs in the system and about a hundred registered users. Also keys could not have user part for anonymous requests. Number of keys could vary from 80K to.. I suppose 500k.
I need to provide per-OID and per-user statistics to the UI and implement the possibility to delete-per-OID (the same for user). So the task is splitted into two. First version I had implemented was unsuccessful - I used .Keys("*") method in c# app which turns into Redis SCAN * to get all keys to the app to run through them, collect and distinct OIDs/users and show them on the UI. This operation was taking too much time to extract all keys to the app so I switched to another approach - on every save/delete key I incremented/decremented counters stored in Redis in another DB. Counters are simply integers with keys. This was almost okay but I got a requirement to set TTL for every cached request, so I faced with a dilemma how to store/track my statistics to keep it up-to-date with actually stored keys. Options I think
A) run LUA script to scan all keys and get all unique OIDS and unique users with counts, and return them to app. For deletion option - run SCAN inside LUA script and DEL all keys matched with pattern. Pros - no need for separate stats tracking. Cons - need to scan for all keys on every call. I have Zabbix dashboard to show statistics requested through my app and it could be painful to scan keys on every N seconds. These are cons.
B) In separate Redis DB store keys with sets where key is OID (or for users - key is GUID) and inside set store keys to cached requests. But how I could delete them when keys to cached requests are expired? Can I somehow link a value stored in a set with a key from another DB and make this value disappear from a set when key expires?
I'm using StackExchange.Redis IDatabase.StringSetAsync in an event subscriber to deduplicate events and I want this operation to fail if the key used all ready exist.
I do check if the key exist. But I want to make sure a race condition where two subscriber instances both read at the same time and find that no key exist and then both try to set the key.
I have read about "watch" in Redis. But this concept seems to be abstracted away in StackExchange.Redis implementation.
I am interested in using Redis to store a customer's session on the server side for authorization. Basically, when a customer logs in a hash will be stored in a Redis cluster and the key returned to the client. On each request, the client will pass the key as a header and the service will check that the hash still exists in Redis, if it doesn't then an error message will be returned. This key will expire after X minutes resulting in any requests using that key to fail. However, I have been reading online that some people experienced issues because of the way the expiration is replicated to slaves. Slaves only expire a key when they receive a del command from the master so if a "get" is made on a slave before this command, the value at that key will be returned.
https://github.com/antirez/redis/issues/187
Does this issue still exist? It seems like a big issue to me and would create a bit of a security hole. Maybe not a big deal for stale data but when using for authorization it is a big deal
A) no, not really — since 2014, a GET of an expired key will return "not found" on a slave even if the slave hasn't yet received a DEL from the replication stream. The outstanding issue has to do with EXISTS being inconsistent with GET, which only matters if you rely on the output of the EXISTS command.
B) Completely independent of this issue, the possibility of replication lag always exists. The security of your app shouldn't depend on the premise that replicas are always up-to-date.
What will happen if key expires when some process is reading the key? Does redis allows to read the key in this scenario? what will be its behavior/return value if key is deleted when process is reading the key?
Redis is single-threaded, so only one request is being processed at a time. If you're able to read it, the read will complete before the delete/expiration is performed. Any expiration, deletion, or alteration operations will happen sequentially after.
I'm using MSETNX (http://redis.io/commands/msetnx) as a locking system, whereby all keys are locked only if no locks already exist.
If a machine holding a lock dies, that lock will be stuck locked - this is a problem.
My ideal answer would be that all keys expire in 15 seconds by default, so even if a machine dies it's held locks will auto-reset in a short time. This way I don't have to call expire on every key I set.
Is this possible in any way?
To build a reliable lock that is high available please check this document: http://redis.io/topics/distlock
The algorithm is still in beta but was stress-tested in a few sessions and is likely to be far more reliable than a single-instance approach anyway.
There are reference implementations for a few languages (linked in the doc).
Redis doesn't have a built-in way to do MSETNX and expire all keys together atomically. Nor can you set a default expiry tube for keys.
You could consider instead:
1. Using a WATCH/MULTI/EXEC block that wraps multiple 'SET key value EX 15 NX', or
2. Doing this using a Lua server-side script.