Problem:
I want to set a TTL on a key (to avoid it lasting forever) but I do NOT want to have that specific key to be evicted.
When I am setting the TTL I know when it will be safe to expire that cache, but it is NOT safe to expire the cache before this time, and eviction presents the risk of having this cache expire to early.
Context:
I am using Redis to cache an object in multiple languages, if the underlying data changes however I want to remove all associated caches from Redis.
The way I went around and sorted this problem was to create a SET on Redis that contains a reference to the keys in every language. My concern is that if that SET is evicted - I loose the reference to the other keys, and risk having them persist on the cache when they shouldn't.
What I am looking for
A Redis command that looks something like
PLEASE_DO_NOT_EVICT key
while not preventing that key from expiring after the TTL runs out.
Thanks very much for taking your time to reading and answering!
While I could use wildcard matching to find all of the associated keys, this is WAY slower than SMEMBERS, and I am doing this in an environment where every MS counts, as these objects are created and deleted very frequently, so this query happens very often.
Not having a TTL in these objects means they start building up in memory which is undersirable. And they do tend to stop being referenced after a while
Having a no eviction policy seems risky, and I would very much want to
When creating:
SADD 'object:id:group', 'object:id:spanish'
SETEX 'object:id:spanish', 'Este es el object en espaniol', 100
EXPIRE 'object:id:group', 100
When expiring the group because the object changed:
SMEMBERS 'object:id:group'
=> 'object:id:spanish', 'object:id:english'
DELETE 'object:id:spanish', 'object:id:english'
DELETE 'object:id:group'
You can set the maxmemory-policy to its default value of "noeviction". In this mode, no keys are evicted.
Related
Is there an advantage to set a default value for an entry that will be heavily queried in Redis or will querying for the unset key take the same time?
Given the keys are stored in a distributed hash, it will have to check that the key is not in the bucket before returning on a miss, which may be a bit slower than finding and stopping at a hit. Is the bucket sorted of linear? Does anything else make it slower either way?
Redis is setup in a cluster and has many million entries in this case.
I'm assuming you're just talking about strings & hashes here here (so the only operations you care about are set/get, maybe hget/hset) - From Redis' perspective, a cache hit and cache miss have the same time complexity, if anything, a cache miss will be faster because redis will not have to transfer any data back over the socket to your app.
Can't seem to find an answer and benchmarks are really tale-telling.
How does Redis handle itself during peak load/usage?
The question comes from knowing CPU usage may hit 100% of its logical core, or memory may be over used.
What happens in these cases?
In general, Redis isn't CPU heavy, and will act like any other application when CPU usage is high, but this largely depends on your Redis version.
Prior to Redis 4.0, it was entirely single-threaded, and long running operations would block (like background saves, DEL's with large objects, etc.) Since 4.0, most of these types of operations are pushed to the background. With saves to disk with the bgsave command, Redis now forks itself and does the save in the child, leaving the parent open to accept changes. 6.0 changed a few things like the del command now acts as unlink, and pushes the actual delete to a thread. There were some plans to add more multi-threading to Redis version 7.0, but it appears this was pushed to 7.2.
The biggest concern however is reaching max system memory, or the maxmemory directive in Redis' config. When this happens, Redis' eviction policy comes into play (set by the maxmemory-policy directive).
Here are the available eviction policies and what they do:
noeviction: New values aren’t saved when memory limit is reached. When a database uses replication, this applies to the primary database
allkeys-lru: Keeps most recently used keys; removes least recently used (LRU) keys
allkeys-lfu: Keeps frequently used keys; removes least frequently used (LFU) keys
volatile-lru: Removes least recently used keys with the expire field set to true.
volatile-lfu: Removes least frequently used keys with the expire field set to true.
allkeys-random: Randomly removes keys to make space for the new data added.
volatile-random: Randomly removes keys with expire field set to true.
volatile-ttl: Removes least frequently used keys with expire field set to true and the shortest remaining time-to-live (TTL) value.
The default maxmemory-policy is noeviction from Redis version 3.0 up through 7.0. In version 2.8 and earlier, the default is volatile-lru.
You can read the Key Eviction docs for more.
Is there anyway to create a Redis database where keys HAVE TO expire after a certain time? I know I can expire an individual key using the EXPIRE command but since I am expiring every key after a certain time anyways, it would be nice to have this behavior specified in the Redis config file.
No, Redis (up to and including v3.2) does not provide the means for automatically setting the TTL of newly-created keys. You have to set it explicitly for each key you create.
If I have several hashes on Redis, each with keys that expire in 24 hours, if memory runs out while using an eviction policy like allkeys-lru, will Redis remove an entire hash or single keys?
Redis only supports expiration at the key's level. Therefore, once set with a TTL, your key that contains a hashmap will be expired entirely (all child fields will be gone with it). The same goes for the other Redis data types (e.g. Sets and Lists).
If you use Hashes for storing key names that need to expire, simply set the TTL for each such key name individually instead of for the Hash's key.
I'm using MSETNX (http://redis.io/commands/msetnx) as a locking system, whereby all keys are locked only if no locks already exist.
If a machine holding a lock dies, that lock will be stuck locked - this is a problem.
My ideal answer would be that all keys expire in 15 seconds by default, so even if a machine dies it's held locks will auto-reset in a short time. This way I don't have to call expire on every key I set.
Is this possible in any way?
To build a reliable lock that is high available please check this document: http://redis.io/topics/distlock
The algorithm is still in beta but was stress-tested in a few sessions and is likely to be far more reliable than a single-instance approach anyway.
There are reference implementations for a few languages (linked in the doc).
Redis doesn't have a built-in way to do MSETNX and expire all keys together atomically. Nor can you set a default expiry tube for keys.
You could consider instead:
1. Using a WATCH/MULTI/EXEC block that wraps multiple 'SET key value EX 15 NX', or
2. Doing this using a Lua server-side script.