What's the best way to get the sum of SCORES in a Redis sorted set?
The only option I think is iterating the sorted set and computing the sum client side.
Available since Redis v2.6 is the most awesome ability to execute Lua scripts on the Redis server. This renders the challenge of summing up a Sorted Set's scores to trivial:
local sum=0
local z=redis.call('ZRANGE', KEYS[1], 0, -1, 'WITHSCORES')
for i=2, #z, 2 do
sum=sum+z[i]
end
return sum
Runtime example:
~$ redis-cli zadd z 1 a 2 b 3 c 4 d 5 e
(integer) 5
~$ redis-cli eval "local sum=0 local z=redis.call('ZRANGE', KEYS[1], 0, -1, 'WITHSCORES') for i=2, #z, 2 do sum=sum+z[i] end return sum" 1 z
(integer) 15
If the sets are small, and you don't need killer performance, I would just iterate (zrange/zrangebyscore) and sum the values client side.
If, on the other hand, you are talking about many thousands - millions of items, you can always keep a reference set with running totals for each user and increment/decrement them as the gifts are sent.
So when you do your ZINCR 123:gifts 1 "3|345", you could do a seperate ZINCR command, which could be something like this:
ZINCR received-gifts 1 <user_id>
Then, to get the # of gifts received for a given user, you just need to run a ZSCORE:
ZSCORE received-gifts <user_id>
Here is a little lua script that maintains the zset score total as you go, in a counter with key postfixed with '.ss'. You can use it instead of ZADD.
local delta = 0
for i=1,#ARGV,2 do
local oldScore = redis.call('zscore', KEYS[1], ARGV[i+1])
if oldScore == false then
oldScore = 0
end
delta = delta - oldScore + ARGV[i]
end
local val = redis.call('zadd', KEYS[1], unpack(ARGV))
redis.call('INCRBY', KEYS[1]..'.ss', delta)
Related
I'm trying to calculate the response times between messages stored in Redis. But I have no clue how to do this.
First I have to store the time-stream of chat_messages like this
ZADD conversation:CONVERSATION_ID:chat_messages TIMESTAMP CHAT_MESSAGE_ID
ZADD conversation:9:chat_messages 1511533205001 2583
ZADD conversation:9:chat_messages 1511533207057 732016
Afterward, I need the application to be able to calculate the time difference between timestamps using Redis because I need the extra speed of doing it without another (potentially slower) technology.
Is there a way of achieving this using plain Redis or Lua?
The timestamp you are using seems to be in milliseconds, so you only need to substract and convert to your desired units.
You can get the score using ZSCORE for each message. Or use one of the ZRANGE methods to get multiple messages at once: ZRANGEBYSCORE ... WITHSCORES.
You can use a Lua script to get the time difference:
local t1 = redis.call('ZSCORE', KEYS[1], ARGV[1])
local t2 = redis.call('ZSCORE', KEYS[1], ARGV[2])
return tonumber(t2) - tonumber(t1)
Here the full EVAL command:
EVAL "local t1 = redis.call('ZSCORE', KEYS[1], ARGV[1]) local t2 = redis.call('ZSCORE', KEYS[1], ARGV[2]) return tonumber(t2) - tonumber(t1)" 1 conversation:9:chat_messages 2583 732016
I was needed to implement set of items with individual expiration, so I used zsetwith score of expiration timestamp.
Now I want to get random item from range of not expired items, or at least from all items in set.
How can I do it?
Can I get min and max rank of range and random rank in between of it via LUA scripting?
Redis version: 5.0.2
I solve this via following script:
-- KEYS[1] - set key
-- ARGV[1] - seed timestamp
local count = redis.call('ZCARD', KEYS[1])
if count ~= 0 then
math.randomseed(ARGV[1])
local rank = math.random(0, count - 1)
local range = redis.call('ZRANGE', KEYS[1], rank, rank)
return range[1]
else
return ''
end
And because I search among all items I do sanitization from expired items every n seconds.
Can change:
ARGV[1] -> os.time()
If I have 5 members with scores as follows
a - 1
b - 2
c - 3
d - 3
e - 5
ZRANK of c returns 2, ZRANK of d returns 3
Is there a way to get same rank for same scores?
Example: ZRANK c = 2, d = 2, e = 3
If yes, then how to implement that in spring-data-redis?
Any real solution needs to fit the requirements, which are kind of missing in the original question. My 1st answer had assumed a small dataset, but this approach does not scale as dense ranking is done (e.g. via Lua) in O(N) at least.
So, assuming that there are a lot of users with scores, the direction that for_stack suggested is better, in which multiple data structures are combined. I believe this is the gist of his last remark.
To store users' scores you can use a Hash. While conceptually you can use a single key to store a Hash of all users scores, in practice you'd want to hash the Hash so it will scale. To keep this example simple, I'll ignore Hash scaling.
This is how you'd add (update) a user's score in Lua:
local hscores_key = KEYS[1]
local user = ARGV[1]
local increment = ARGV[2]
local new_score = redis.call('HINCRBY', hscores_key, user, increment)
Next, we want to track the current count of users per discrete score value so we keep another hash for that:
local old_score = new_score - increment
local hcounts_key = KEYS[2]
local old_count = redis.call('HINCRBY', hcounts_key, old_score, -1)
local new_count = redis.call('HINCRBY', hcounts_key, new_score, 1)
Now, the last thing we need to maintain is the per score rank, with a sorted set. Every new score is added as a member in the zset, and scores that have no more users are removed:
local zdranks_key = KEYS[3]
if new_count == 1 then
redis.call('ZADD', zdranks_key, new_score, new_score)
end
if old_count == 0 then
redis.call('ZREM', zdranks_key, old_score)
end
This 3-piece-script's complexity is O(logN) due to the use of the Sorted Set, but note that N is the number of discrete score values, not the users in the system. Getting a user's dense ranking is done via another, shorter and simpler script:
local hscores_key = KEYS[1]
local zdranks_key = KEYS[2]
local user = ARGV[1]
local score = redis.call('HGET', hscores_key, user)
return redis.call('ZRANK', zdranks_key, score)
You can achieve the goal with two Sorted Set: one for member to score mapping, and one for score to rank mapping.
Add
Add items to member to score mapping: ZADD mem_2_score 1 a 2 b 3 c 3 d 5 e
Add the scores to score to rank mapping: ZADD score_2_rank 1 1 2 2 3 3 5 5
Search
Get score first: ZSCORE mem_2_score c, this should return the score, i.e. 3.
Get the rank for the score: ZRANK score_2_rank 3, this should return the dense ranking, i.e. 2.
In order to run it atomically, wrap the Add, and Search operations into 2 Lua scripts.
Then there's this Pull Request - https://github.com/antirez/redis/pull/2011 - which is dead, but appears to make dense rankings on the fly. The original issue/feature request (https://github.com/antirez/redis/issues/943) got some interest so perhaps it is worth reviving it /cc #antirez :)
The rank is unique in a sorted set, and elements with the same score are ordered (ranked) lexically.
There is no Redis command that does this "dense ranking"
You could, however, use a Lua script that fetches a range from a sorted set, and reduces it to your requested form. This could work on small data sets, but you'd have to devise something more complex for to scale.
unsigned long zslGetRank(zskiplist *zsl, double score, sds ele) {
zskiplistNode *x;
unsigned long rank = 0;
int i;
x = zsl->header;
for (i = zsl->level-1; i >= 0; i--) {
while (x->level[i].forward &&
(x->level[i].forward->score < score ||
(x->level[i].forward->score == score &&
sdscmp(x->level[i].forward->ele,ele) <= 0))) {
rank += x->level[i].span;
x = x->level[i].forward;
}
/* x might be equal to zsl->header, so test if obj is non-NULL */
if (x->ele && x->score == score && sdscmp(x->ele,ele) == 0) {
return rank;
}
}
return 0;
}
https://github.com/redis/redis/blob/b375f5919ea7458ecf453cbe58f05a6085a954f0/src/t_zset.c#L475
This is the piece of code redis uses to compute the rank in sorted sets. Right now ,it just gives rank based on the position in the Skiplist (which is sorted based on scores).
What does the skiplistnode variable "span" mean in redis.h? (what is span ?)
I'm a rookie in using Redis, and recently have a problem when I'm thinking the solution to handle the case of high concurrency in out system, I want to use Redis, everybody know access from In-Memory is great faster than IO. Redis sorted set is the possible tool for use to do it, we want to a fixed size of sorted set to contain the user's mobile number, I Googled/Baidued a lot, didn't find any meaning message, so can anybody tell me How to specify Redis Sorted Set a fixed size? And set should tell me any add operation to sorted set is success or not?
Thanks
I don't think you can specify a size, that you will have to check it yourself.
You can use ZCARD
ZCARD KEY
Or you could just remove the first element ZREMRANGEBYRANK
[ZREMRANGEBYRANK][2] [KEY] 0 -[YOURSIZE]
This can be done with a lua script by checking the set size (ZCARD) before adding the element (ZADD). Complexity is O(log(|setsize|)) :
-- KEYS[1] the sorted set
-- ARGV[1] the score
-- ARGV[2] the member
-- ARGV[3] the max size of the sorted set
-- Returns number of elements added
--
if redis.call('ZCARD', KEYS[1]) < tonumber(ARGV[3]) then
return redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2])
else
return 0
end
You can try it out if you save the above snippet as test.lua file and then run it a few times, e.g
redis-cli --eval test.lua myset , 1 A 3
redis-cli --eval test.lua myset , 2 B 3
redis-cli --eval test.lua myset , 3 C 3
redis-cli --eval test.lua myset , 4 D 3
redis-cli --eval test.lua myset , 5 E 3
Verify
127.0.0.1:6379> ZRANGE myset 0 100
1) "A"
2) "B"
3) "C"
I got several sorted sets with a common prefix (itemmovements:) in Redis.
I know we can use ZCOUNT to get the number of items for a single (sorted set) key like this:
127.0.0.1:6379> zcount itemmovements:8 0 1000000000
(integer) 23
(I am able to do this, since I know the range of the item scores.)
How to run this in a loop for all keys prefixed itemmovements:?
Taking hint from How to atomically delete keys matching a pattern using Redis I tried this:
127.0.0.1:6379> EVAL "return redis.call('zcount', unpack(redis.call('keys', ARGV[1])), 0, 1000000000)" 0 itemmovements:*
(integer) 150
but as you can see it just returns a single number (which happens to be the size of itemmovements:0, the first value returned by keys).
I realized I did not understand what that lua code in EVAL was doing. The code below works fine:
eval "local a = {}; for _,k in ipairs(redis.call('keys', 'itemmovements:*')) do table.insert(a, k); table.insert(a, redis.call('zcount', k, 0, 1000000000)); end; return a" 0