How does Redis implement the expiration of keys? From here I learnt that Redis stores the time at which the key will expire, but how exactly is this implemented?
In short - for each redis object, there is an expiration time. Unless you set the object to expire, that time is "never".
Now, the expiration mechanism itself is semi-lazy. Lazy expiration means that you don't actually expire the objects until they are read. When reading an object, we check its expiration timestamp, and if it's in the past, we return nothing, and delete the object while we're at it. But the problem is that if a key is never touched, it just takes up memory for no reason.
So Redis adds a second layer of random active expiration. It just reads random keys all the time, and when an expired key is touched it is deleted based on the lazy mechanism. This does not affect the expire behavior, it just adds "garbage collection" of expired keys.
Of course the actual implementation is more complicated than this, but this is the main idea.
You can read more about it here: http://redis.io/commands/expire
And the source code for the active expiration cycle can be found here: https://github.com/antirez/redis/blob/a92921da135e38eedd89138e15fe9fd1ffdd9b48/src/expire.c#L98
For the Redis 6, from the release notes:
The expiration cycle has been rewritten in Redis 6.0 to allow for much
faster expirations that more closely match the time-to-live (TTL)
property. Redis 6 expiration will no longer be based on random
sampling but will take keys sorted by expire time in a radix tree
— Redis 6 release notes
For full release notes here.
From reference:
Redis keys are expired in two ways: a passive way, and an active
way.
A key is passively expired simply when some client tries to access it,
and the key is found to be timed out.
Of course this is not enough as there are expired keys that will never
be accessed again. These keys should be expired anyway, so
periodically Redis tests a few keys at random among keys with an
expire set. All the keys that are already expired are deleted from the
keyspace.
Specifically this is what Redis does 10 times per second:
Test 20 random keys from the set of keys with an associated expire.
Delete all the keys found expired. If more than 25% of keys were
expired, start again from step 1. This is a trivial probabilistic
algorithm, basically the assumption is that our sample is
representative of the whole key space, and we continue to expire until
the percentage of keys that are likely to be expired is under 25%
This means that at any given moment the maximum amount of keys already
expired that are using memory is at max equal to max amount of write
operations per second divided by 4.
How Redis expires keys
How expires are handled in the replication link and AOF file
Related
According to Redis documentaiton, EXPIREAT has the 'exact same effect and semantic as EXPIRE'.
EXPIRE invalidates itself when a key is retrieved via GET. There is no mention in the EXPIREAT documentation of a similar behaviour, but it would not make sense for EXPIREAT to no longer expire upon key retrieval.
Due to the vague nature of the documentation I am unable to tell what the actual behaviour of EXPIREAT is in this scenario.
Actually expiring itself when the key is retrieved is not the only way, according to documentation
Redis keys are expired in two ways: a passive way, and an active way.
A key is passively expired simply when some client tries to access it, and the key is found to be timed out.
Of course this is not enough as there are expired keys that will never be accessed again. These keys should be expired anyway, so periodically Redis tests a few keys at random among keys with an expire set. All the keys that are already expired are deleted from the keyspace.
For your question;
There is a generic expire function in the Redis codebase and in the comments section it says
This is the generic command implementation for EXPIRE, PEXPIRE, EXPIREAT and PEXPIREAT.
"my optimistic guess" is that these different expire( or *at) commands invoke this function with different parameters by making a conversion. So EXPIREAT command also has the same behavior as EXPIRE and the keys are expired with the same two ways.
I'm working with Spring Data Redis and I'm a little bit confused about the utility of phantom key. Below some questions about that :
What's the goal of Phantom Key for Spring Data Redis?
When should I save it or not? What are the impacts?
Is there best practices on this subject?
Thank advance for your feedback.
When the expiration is set to a positive value, the corresponding
EXPIRE command is run. In addition to persisting the original, a
phantom copy is persisted in Redis and set to expire five minutes
after the original one. This is done to enable the Repository support
to publish RedisKeyExpiredEvent, holding the expired value in Spring’s
ApplicationEventPublisher whenever a key expires, even though the
original values have already been removed. Expiry events are received
on all connected applications that use Spring Data Redis repositories.
By default, the key expiry listener is disabled when initializing the
application. The startup mode can be adjusted in
#EnableRedisRepositories or RedisKeyValueAdapter to start the listener
with the application or upon the first insert of an entity with a TTL.
See EnableKeyspaceEvents for possible values.
The RedisKeyExpiredEvent holds a copy of the expired domain object as
well as the key.
For more information see https://docs.spring.io/spring-data/data-redis/docs/current/reference/html/#redis.repositories.expirations
I’m wondering if anyone here can help me confirm how key expiry would work in a particular use case. Our case is that we are using Redis to deduplicate messages in a message delivery system based on a unique identifier used as the key. We are using SETEX to set a TTL and have used GET calls in the past to check expiry, but I’m wondering if an EXISTS call instead of a GET would work just the same. I see in the documentation that Redis will passively expire keys “simply when some client tries to access it, and the key is found to be timed out”, which is why we can rely on GET calls to only return a value if the TTL hasn’t passed, but would the same happen for an EXISTS call? To put that another way, will EXISTS calls passively expire a key if it’s past the TTL and return 0, or is it possible that an EXISTS call will not take TTL into consideration, not passively expire a key that we’re calling EXISTS for, and possibly return a 1 even though the key’s TTL has expired?
Yes Redis should passively expire keys on Exists
I have a question while I'm studying about Redis with docs.
Here is the description in official Redis document about expiration :
https://redis.io/commands/expire
Specifically this is what Redis does 10 times per second:
1. Test 20 random keys from the set of keys with an associated expire.
2. Delete all the keys found expired.
3. If more than 25% of keys were expired, start again from step 1.
After I read this, I have a question about 25% of keys above.
What is the keys meaning in 25% of keys exactly?
The keys are the all of keys in Redis? or 20 random keys in description above?
I think the keys are looks like the all of keys in Redis.
I couldn't find any description about this in document.
Thank you.
By default, a key in Redis does not have a time to live (ttl) set. The 20 random keys in this algorithm are sampled from the set of all keys where an explicit ttl has been set. If more than 5 of the keys in this sample have expired then it's likely that the portion of expired keys in Redis is high, and so the algorithm repeats to try to drive the portion of expired keys down below the threshold.
It's important also to note that keys that have lived beyond their ttl will be expired automatically on access (meaning if the user tries to request them). So a heuristic algorithm like this simply needs to ensure that Redis doesn't pile up a bunch of expired keys, while making sure that Redis is not spending too many resources cleaning up expired keys.
See the source code:
https://github.com/antirez/redis/blob/unstable/src/expire.c
I have set an expiration value to a key in redis, and want to get the opportunity to run a piece of code before the key will be deleted by redis. Is it possible, and if so how...?
Thanks
My solution was to create a new key, with the same name as the one I wanted to hook, only I added a prefix for it indicating it's a key for timeouts usage ("TO") - something like:
set key1 data1
set TO_key1 ""
expire TO_key1 20
In the example above, as soon as "TO_key1" will expire, it will notify my program and I'll get the opportunity to run my code before I will manually delete "key1".
I found this link very useful for creating the listener for redis: Redis Key expire notification with Jedis
This isn't possible with standard OS Redis... yet. There is a way, however, to do something similar without too much hassle. If you stop using Redis' expiry (at least for those keys that you're interesting in "hooking") and manage expiry "manually" in your code, you can do anything you want before/during/after the expiry event.
Since Redis offers key-level expiry out of the box, people are usually content with it. In some cases, i.e.g. expiring members in a Set, the only way to go is the manual approach but that approach is still valid for regular keys when you need finer control.