When calling expire or pexpire in redis, is time rounded up? - redis

I am calling expire for an existing redis key. Let's say I am passing 5 for the value. If the key already exists and is 4.75 seconds away from expiring, does it stay at 4.75 seconds or is it rounded back up to 5 seconds?
I can use pexpire to get more granularity, but there is still a rounding problem with partial milliseconds - unless milliseconds is the smallest granularity in redis...
If it helps, here is my rate limit script, which takes a key, an amount to increment and a millisecond rate limit window, which keeps decrementing until the key drops out, at which point the next call adds the key and sets a fresh expire time. The new incremented value is then returned.
local f,k,a,b,c c=ARGV[2] f=redis.call k=KEYS[1] a=f('incrby',k,ARGV[1]) b=f('pttl',k) f('pexpire',k,math.min(b<0 and c or b,c)) return a
UPDATE
New rate limit script that does not have partial time issue, it only sets expire if the key does not have an expire set at all:
local f,k,a,b f=redis.call k=KEYS[1] a=f('incrby',k,ARGV[1]) b=f('pttl',k) if b<0 then f('pexpire',k,ARGV[2]) end return a

Does it stay at 4.75 seconds or is it rounded back up to 5 seconds?
It is back to full 5 seconds TTL.
unless milliseconds is the smallest granularity in redis...
It is milliseconds, for version 2.6 or greater
See Expire accuracy
In Redis 2.4 the expire might not be pin-point accurate, and it could
be between zero to one seconds out. Since Redis 2.6 the expire error
is from 0 to 1 milliseconds.
And
Keys expiring information is stored as absolute Unix timestamps (in milliseconds in case of Redis version 2.6 or greater).
You can play with some Lua scripts if you want to verify
EVAL "local result = {'Time at start', 0, 'Expires in (ms)', 0, 'Time at end', 0} \n result[2] = redis.call('TIME') \n redis.call('EXPIRE', KEYS[1], ARGV[1]) \n result[4] = redis.call('PTTL', KEYS[1]) \n result[6] = redis.call('TIME') \n return result" 1 myKey 5
Friendly view of the script:
local result = {'Time at start', 0, 'Expires in (ms)', 0, 'Time at end', 0}
result[2] = redis.call('TIME')
redis.call('EXPIRE', KEYS[1], ARGV[1])
result[4] = redis.call('PTTL', KEYS[1])
result[6] = redis.call('TIME')
return result

Related

mktime() returns an incorrect value right after entering DST

The following snippet code is from rtc.c in busybox-1.22.1.
In my case the utc is always 0, so this function is just doing a conversion from struct tm to time_t.
time_t FAST_FUNC rtc_tm2time(struct tm *ptm, int utc)
{
//fprintf(stdout, "ptm->tm_hour: %d\n", ptm->tm_hour);
char *oldtz = oldtz; /* for compiler */
time_t t;
if (utc) {
oldtz = getenv("TZ");
putenv((char*)"TZ=UTC0");
tzset();
}
t = mktime(ptm); //problem here
//struct tm* temp = localtime(&t);
//fprintf(stdout, "temp->tm_hour: %d\n", temp->tm_hour);
if (utc) {
unsetenv("TZ");
if (oldtz)
{
putenv(oldtz - 3);
}
tzset();
}
return t;
}
Also, there is a file /etc/TZ displaying timezone and DST information.
~ # cat /etc/TZ
LMT0:00LMT-1:00,M8.5.1/10,M12.5.1/10
Then, I set system time to 2021/8/30, 9:59:30 (30 seconds earlier than DST start date), and sync to hwclock.
date -s 2021.08.30-09:59:30 >/dev/null 2>/dev/null //set system time
hwclock -w //sync RTC to system time
Entering hwclock continuously while observing the output on CLI.
~ # hwclock
ptm->tm_hour : 9
temp->tm_hour : 9
Mon Aug 30 09:59:58 2021 0.000000 seconds
~ # hwclock
ptm->tm_hour : 10
temp->tm_hour : 11 //why not 10?
Mon Aug 30 11:00:00 2021 0.000000 seconds
Why the return value from mktime is added by 1 when entering DST? Shouldn't it be affected by DST?
According to the mktime() man pages, mktime() is allowed to update the tm_isdst value. The starting value may cause the mktime() algorithm to branch differently:
The value specified in the tm_isdst field informs mktime() whether or not
daylight saving time (DST) is in effect for the time supplied in the tm
structure: a positive value means DST is in effect; zero means that DST is not in
effect; and a negative value means that mktime() should (use timezone information
and system databases to) attempt to determine whether DST is in effect at the
specified time.
and then update the value of tm_isdst accordingly:
tm_isdst is set (regardless of its initial value) to a positive value or to 0,
respectively, to indicate whether DST is or is not in effect at the specified time.
In other words, I'd check the tm_isdst value before and after the mktime() call.
One way I've dealt with this in the past is to
call mktime() with a known value of tm_isdst (e.g. zero or one)
call localtime() on the returned time_t value, then check tm_isdst on the struct tm pointer localtime() returns.
if the tm_isdst has been changed from the prior known value, change the original struct tm to use the new tm_isdt value then call mktime() again with it before trusting the time_t it returns.
It's definitely less efficient but it's possible to know when the DST change occurs by expecting it and checking for it.
Another option would be to set tm_isdst to -1 before calling mktime() and trusting its lookup of timezone and set tm_isdst appropriately.

How to store in Redis sorted set with server-side timestamp as score?

I want to use a sorted set to store objects using the redis-server timestamp as score.
I know I can use Redis Streams with * id, but Redis Streams have limitations, including I cannot edit the objects, I cannot use rank or lexicographical sorting, I cannot really delete objects in the middle, unions or intersects, etc.
I want to do this atomically, and use the redis-server timestamp so I can use multiple clients to ZADD without worrying about clock-sync.
How to do this?
The solution is to use a Lua script:
local time = redis.call('TIME')
local ts = time[1]..string.format('%06d', time[2])
return redis.call('ZADD', KEYS[1], ts, ARGV[1])
Here we use Redis TIME command. The command returns:
unix time in seconds
microseconds
So we can concatenate these two and use a microsecond-timestamp. We need to zero-pad the microseconds part.
Since sorted sets are good with integer values up to 2^53, our timestamp is safe all the way up to the year 2255.
This is Redis-Cluster-safe as we store in one key. To use multiple keys, make sure to land them on the same node using hash tags if you want to compare timestamps.
You can modify the script to use lower than microsecond resolution.
Here the EVAL command, simple pass key and value as arguments, no need to create the sorted set before hand:
EVAL "local time = redis.call('TIME') local ts = time[1]..string.format('%06d', time[2]) return redis.call('ZADD', KEYS[1], ts, ARGV[1])" 1 ssetKey myVal
As always, you may want to load the script and use EVALSHA.
> SCRIPT LOAD "local time = redis.call('TIME') local ts = time[1]..string.format('%06d', time[2]) return redis.call('ZADD', KEYS[1], ts, ARGV[1])"
"81e366e422d0b09c9b395b5dfe03c03c3b7b3bf7"
> EVALSHA 81e366e422d0b09c9b395b5dfe03c03c3b7b3bf7 1 ssetKey myNewVal
(integer) 1
A note on Redis version. If you are using:
Redis Version before 3.2: sorry, you cannot use TIME (non-deterministic command) and then write with ZADD.
Redis Version greater than 3.2 but < 5.0: Add redis.replicate_commands() on top of the script. See Scripts as pure functions
Redis 5.0 an up: you are good.

How to expire set of redis in lua every day?

I am banning spider using lua script in openresty(1.13.6.2). It will probe the illegal ip and add to black list.
res , err = cache:sadd('black-list', ngx.var.remote_addr)
Now I want to make black list expire in 12:00 every day(Release ban ip). How to expire set daily?What is the proper way?This is the ban forbidden code:
is_black ,err = cache:sismember('black_list', ngx.var.remote_addr)
if is_black == 1 then
ngx.exit(ngx.HTTP_FORBIDDEN)
goto label
end
As an alternative: you can use string redis data type for saving IPs instead of set (for convenience in searching you can use some common part in the key, for example black_list; redis key can be divided by semicolon):
# redis cli commands
set black_list:127.0.0.1 127.0.0.1 EX 3600 # expires in 1 hour
set black_list:127.0.0.2 127.0.0.2 EX 60 # expires in 1 minute
set black_list:127.0.0.3 127.0.0.3 EX 60 # expires in 1 minute
In this case you can calculate (12:00 - now) and set TTL (time to live) for each IP record and they will be expired automatically in needed time without any additional actions from code.

stop tcl procedure from running more regularly than once every 2 minutes, regardless of how often it is called

I have a piece of code that has a procedure, I only want the proc to trigger, at most every 2 minutes. If it has triggered within the last 2 min, then it should just exit. Currently it has a small delay before it executes (after 5000), but what this seems to do as well, is queue any other execution requests that have occurred in the quiesce time (5 seconds) and then just pummel the queued commands out in a flurry of activity.
Obviously this is missing a significant portion, but
I have considered doing something with variables like:
#gets current time since epoch as var0
set timer0 clock format [clock seconds] -format %s
#gets current time since epoch as var1
set timer1 clock format [clock seconds] -format %s
#calculates elapsed time since epoch in seconds, converts seconds to minutes
set split [ expr (($timer0 - $timer1)/60)/60 ]
if { $split > 2 } {
my_proc $maybe_1var $maybe_2vars $maybe_3vars
} else {
exit super gracefully
}
I can provide snippets of my current code if you like. It is just not nearly as elegant as I imagine it could be, and I am not sure if there are better ways of doing this in Tcl.
One possible solution:
set lastExecuted ""
proc foo {} {
global lastExecuted
set currentTimestamp [clock scan seconds]
if {$lastExecuted != "" && ($currentTimestamp - $lastExecuted) > 120} {
exit
}
set lastExecuted $currentTimestamp
# continue execution
}

How to understand redis-cli's result vs redis-benchmark's result

First, I am new to Redis.
So, I measure latency with redis-cli:
$ redis-cli --latency
min: 0, max: 31, avg: 0.55 (5216 samples)^C
OK, on average I get response in 0.55 milliseconds. From this I assume that using only one connection in 1 second I can get: 1000ms / 0.55ms = 1800 requests per second.
Then on the same computer I run redis-benchmark using only one connection and get more than 6000 requests per second:
$ redis-benchmark -q -n 100000 -c 1 -P 1
PING_INLINE: 5953.80 requests per second
PING_BULK: 6189.65 requests per second
So having measured latency I expected to get around 2000 request per seconds at best. However I got 6000 request per second. I cannot find explanation for it. Am I correct when I calculate: 1000ms / 0.55ms = 1800 requests per second?
Yes, your maths are correct.
IMO, the discrepancy comes from scheduling artifacts (i.e. to the behavior of the operating system scheduler or the network loopback).
redis-cli latency is implemented by a loop which only sends a PING command before waiting for 10 ms. Let's try an experiment and compare the result of redis-cli --latency with the 10 ms wait state and without.
In order to be accurate, we first make sure the client and server are always scheduled on deterministic CPU cores. Note: it is generally a good idea to do it for benchmarking purpose on NUMA boxes. Also, make sure the frequency of the CPUs is blocked to a given value (i.e. no power mode management).
# Starting Redis
numactl -C 2 src/redis-server redis.conf
# Running benchmark
numactl -C 4 src/redis-benchmark -n 100000 -c 1 -q -P 1 -t PING
PING_INLINE: 26336.58 requests per second
PING_BULK: 27166.53 requests per second
Now let's look at the latency (with the 10 ms wait state):
numactl -C 4 src/redis-cli --latency
min: 0, max: 1, avg: 0.17761 (2376 samples)
It seems too high compared to the throughput result of redis-benchmark.
Then, we alter the source code of redis-cli.c to remove the wait state, and we recompile. The code has also been modified to display more accurate figures (but less frequently, because there is no wait state anymore).
Here is the diff against redis 3.0.5:
1123,1128c1123
< avg = ((double) tot)/((double)count);
< }
< if ( count % 1024 == 0 ) {
< printf("\x1b[0G\x1b[2Kmin: %lld, max: %lld, avg: %.5f (%lld samples)",
< min, max, avg, count);
< fflush(stdout);
---
> avg = (double) tot/count;
1129a1125,1127
> printf("\x1b[0G\x1b[2Kmin: %lld, max: %lld, avg: %.2f (%lld samples)",
> min, max, avg, count);
> fflush(stdout);
1135a1134
> usleep(LATENCY_SAMPLE_RATE * 1000);
Note that this patch should not be used against a real system, since it will make the redis-client --latency feature expensive and intrusive for the performance of the server. Its purpose is just to illustrate my point for the current discussion.
Here we go again:
numactl -C 4 src/redis-cli --latency
min: 0, max: 1, avg: 0.03605 (745280 samples)
Surprise! The average latency is now much lower. Furthermore, 1000/0.03605=27739.25, which is completely in line with the result of redis-benchmark.
Morality: the more the client loop is scheduled by the OS, the lower the average latency. It is wise to trust redis-benchmark over redis-cli --latency if your Redis clients are active enough. And anyway keep in mind the average latency does not mean much for the performance of a system (i.e. you should also look at the latency distribution, the high percentiles, etc. ..)