Is it possible (or is there another idiomatic way) to ZADD a member with a score of the highest existing score plus one?
For example:
> FLUSHDB
> ZADD key 1 one
> ZADD key * mem
> ZSCORE key mem
1) "2"
You can use zrevrange to get the largest score, and then call zadd to set the score to be largest + 1. There's no built-in way to do that. However, you can write a Lua script to do the job:
local key = KEYS[1]
local field = ARGV[1]
local score = ARGV[2]
if score then
-- client specifies a score, use it
redis.call('zadd', key, score, field)
else
-- get the largest score in the sorted set
local largest = redis.call('zrevrange', key, 0, 0, 'withscores')
score = largest[2]
if score then
-- update the score
score = score + 1
else
-- the sorted set is empty, set a default score
score = 0
end
redis.call('zadd', key, score, field)
end
Try it: ./src/redis-cli --eval t.lua key , field
Related
I am new to Redis, and now I need to update the a sorted set if the key exists in another sorted set.
I think it may be clearer to explain by an example, lets say that there are two sorted sets like the following:
set_1
{key_1:val_1, key_2:val_2, key_3:val_3}
set_2
{key_1:val_new_1, key_3:val_new_3, key_4:val_new_4}
Now I am trying to update the first set if the key exists in the second set, so the result should be:
set_1
{key_1:val_new_1, key_2:val_2, key_3:val_new_3}
I have been reading the Redis documentation for a while, and it seems using he SET command with XX option may help:
The SET command supports a set of options that modify its behavior:
XX -- Only set the key if it already exist.
But is it possible to avoid running this on each entry in the first set? Maybe using something like zunionstore?
The SET command only works for regular keys, not for sorted sets.
In sorted sets, you have score-member pairs, so the key-value pair nomenclature of your example is a little confusing. I'll assume key_1, key_2, key_3, ... are members and val_1, val_2, ... are scores.
Let's create the sorted sets as follows to review the solution:
> ZADD set_1 1 key_1 2 key_2 3 key_3
(integer) 3
> ZADD set_2 1001 key_1 1003 key_3 1004 key_4
(integer) 3
The default AGGREGATE is SUM, it's what we will use all across.
We'll create two sorted sets with the intersection of both, one with the scores of set_1 and one with the scores of set_2.
> ZINTERSTORE intersect_set_1 2 set_1 set_2 WEIGHTS 1 0
(integer) 2
> ZINTERSTORE intersect_set_2 2 set_1 set_2 WEIGHTS 0 1
(integer) 2
Now, we create a middle-step set for set_1, where we set the score to zero for those that are in set_2 as well:
> ZUNIONSTORE pre_set_1 2 set_1 intersect_set_1 WEIGHTS 1 -1
(integer) 3
Now we are ready to update set_1, doing a union of:
pre_set_1: all set_1 but with those also in set_2 set to zero score.
intersect_set_2: the intersection of set_1 and set_2, with the scores of set_2.
Here is the final command:
> ZUNIONSTORE set_1 2 pre_set_1 intersect_set_2
(integer) 3
Let's see the result:
> ZRANGE set_1 0 -1 WITHSCORES
1) "key_2"
2) "2"
3) "key_1"
4) "1001"
5) "key_3"
6) "1003"
Don't forget to clean up:
> UNLINK pre_set_1 intersect_set_1 intersect_set_2
This solution is not optimal as it uses multiple middle-steps, there is a risk for members added to the original sets in between and it uses more memory than necessary.
The optimal solution would be a Lua script:
local set2 = redis.call('ZRANGE', KEYS[1], '0', '-1', 'WITHSCORES')
local set2length = table.getn(set2)
for i=1,set2length,2 do redis.call('ZADD', KEYS[2], 'XX', set2[i+1], set2[i]) end
return set2length/2
This loops through set_2, updating set_1. Note the use of XX in the ZADD command, to only update if it exists.
Use as:
EVAL "local set2 = redis.call('ZRANGE', KEYS[1], '0', '-1', 'WITHSCORES') \n local set2length = table.getn(set2) \n for i=1,set2length,2 do print(1) redis.call('ZADD', KEYS[2], 'XX', set2[i+1], set2[i]) end \n return set2length/2" 2 set_2 set_1
The Lua script is atomic due to the single-threaded nature of Redis.
I was needed to implement set of items with individual expiration, so I used zsetwith score of expiration timestamp.
Now I want to get random item from range of not expired items, or at least from all items in set.
How can I do it?
Can I get min and max rank of range and random rank in between of it via LUA scripting?
Redis version: 5.0.2
I solve this via following script:
-- KEYS[1] - set key
-- ARGV[1] - seed timestamp
local count = redis.call('ZCARD', KEYS[1])
if count ~= 0 then
math.randomseed(ARGV[1])
local rank = math.random(0, count - 1)
local range = redis.call('ZRANGE', KEYS[1], rank, rank)
return range[1]
else
return ''
end
And because I search among all items I do sanitization from expired items every n seconds.
Can change:
ARGV[1] -> os.time()
If I have 5 members with scores as follows
a - 1
b - 2
c - 3
d - 3
e - 5
ZRANK of c returns 2, ZRANK of d returns 3
Is there a way to get same rank for same scores?
Example: ZRANK c = 2, d = 2, e = 3
If yes, then how to implement that in spring-data-redis?
Any real solution needs to fit the requirements, which are kind of missing in the original question. My 1st answer had assumed a small dataset, but this approach does not scale as dense ranking is done (e.g. via Lua) in O(N) at least.
So, assuming that there are a lot of users with scores, the direction that for_stack suggested is better, in which multiple data structures are combined. I believe this is the gist of his last remark.
To store users' scores you can use a Hash. While conceptually you can use a single key to store a Hash of all users scores, in practice you'd want to hash the Hash so it will scale. To keep this example simple, I'll ignore Hash scaling.
This is how you'd add (update) a user's score in Lua:
local hscores_key = KEYS[1]
local user = ARGV[1]
local increment = ARGV[2]
local new_score = redis.call('HINCRBY', hscores_key, user, increment)
Next, we want to track the current count of users per discrete score value so we keep another hash for that:
local old_score = new_score - increment
local hcounts_key = KEYS[2]
local old_count = redis.call('HINCRBY', hcounts_key, old_score, -1)
local new_count = redis.call('HINCRBY', hcounts_key, new_score, 1)
Now, the last thing we need to maintain is the per score rank, with a sorted set. Every new score is added as a member in the zset, and scores that have no more users are removed:
local zdranks_key = KEYS[3]
if new_count == 1 then
redis.call('ZADD', zdranks_key, new_score, new_score)
end
if old_count == 0 then
redis.call('ZREM', zdranks_key, old_score)
end
This 3-piece-script's complexity is O(logN) due to the use of the Sorted Set, but note that N is the number of discrete score values, not the users in the system. Getting a user's dense ranking is done via another, shorter and simpler script:
local hscores_key = KEYS[1]
local zdranks_key = KEYS[2]
local user = ARGV[1]
local score = redis.call('HGET', hscores_key, user)
return redis.call('ZRANK', zdranks_key, score)
You can achieve the goal with two Sorted Set: one for member to score mapping, and one for score to rank mapping.
Add
Add items to member to score mapping: ZADD mem_2_score 1 a 2 b 3 c 3 d 5 e
Add the scores to score to rank mapping: ZADD score_2_rank 1 1 2 2 3 3 5 5
Search
Get score first: ZSCORE mem_2_score c, this should return the score, i.e. 3.
Get the rank for the score: ZRANK score_2_rank 3, this should return the dense ranking, i.e. 2.
In order to run it atomically, wrap the Add, and Search operations into 2 Lua scripts.
Then there's this Pull Request - https://github.com/antirez/redis/pull/2011 - which is dead, but appears to make dense rankings on the fly. The original issue/feature request (https://github.com/antirez/redis/issues/943) got some interest so perhaps it is worth reviving it /cc #antirez :)
The rank is unique in a sorted set, and elements with the same score are ordered (ranked) lexically.
There is no Redis command that does this "dense ranking"
You could, however, use a Lua script that fetches a range from a sorted set, and reduces it to your requested form. This could work on small data sets, but you'd have to devise something more complex for to scale.
unsigned long zslGetRank(zskiplist *zsl, double score, sds ele) {
zskiplistNode *x;
unsigned long rank = 0;
int i;
x = zsl->header;
for (i = zsl->level-1; i >= 0; i--) {
while (x->level[i].forward &&
(x->level[i].forward->score < score ||
(x->level[i].forward->score == score &&
sdscmp(x->level[i].forward->ele,ele) <= 0))) {
rank += x->level[i].span;
x = x->level[i].forward;
}
/* x might be equal to zsl->header, so test if obj is non-NULL */
if (x->ele && x->score == score && sdscmp(x->ele,ele) == 0) {
return rank;
}
}
return 0;
}
https://github.com/redis/redis/blob/b375f5919ea7458ecf453cbe58f05a6085a954f0/src/t_zset.c#L475
This is the piece of code redis uses to compute the rank in sorted sets. Right now ,it just gives rank based on the position in the Skiplist (which is sorted based on scores).
What does the skiplistnode variable "span" mean in redis.h? (what is span ?)
I have an ordered set in Redis (I am actually using a python client https://github.com/andymccurdy/redis-py), for example:
zadd myset 1 key1
zadd myset 1 key2
zadd myset 1 key3
zadd myset 0 key4
Note that 3 keys have the same score.
Using ZRANGE, i would like to get the top 2 entries (i.e lowest scores). "key4" will always be the first result as it has a lower value, but I would like the second return value to be randomly selected between the ties: key1,key2,key3. ZRANGE actually returns the keys in the order they are indexed: "keys1" is always my second result:
zrange myset 0 -1 WITHSCORES
1) "key4"
2) "0"
3) "key1"
4) "1"
5) "key2"
6) "1"
7) "key3"
8) "1"
any idea?
thanks,
J.
As kindly requested by Linus G Thiel, here are more details about my usecase:
I would like to use zsets to perform a simple ranking system. I have a list of items, for each one a score representing the relevance of the item. For the cold start of my system, most of the scores will be identical (i.e 0), and I would like to randomly select among the items having the same score. Otherwise I will always return the exact same lexicographic ordering, which will introduce a bias in the system.
The solution you propose, using one specific set for each duplicated score value will work. I will give it a try.
Thanks,
What's the best way to get the sum of SCORES in a Redis sorted set?
The only option I think is iterating the sorted set and computing the sum client side.
Available since Redis v2.6 is the most awesome ability to execute Lua scripts on the Redis server. This renders the challenge of summing up a Sorted Set's scores to trivial:
local sum=0
local z=redis.call('ZRANGE', KEYS[1], 0, -1, 'WITHSCORES')
for i=2, #z, 2 do
sum=sum+z[i]
end
return sum
Runtime example:
~$ redis-cli zadd z 1 a 2 b 3 c 4 d 5 e
(integer) 5
~$ redis-cli eval "local sum=0 local z=redis.call('ZRANGE', KEYS[1], 0, -1, 'WITHSCORES') for i=2, #z, 2 do sum=sum+z[i] end return sum" 1 z
(integer) 15
If the sets are small, and you don't need killer performance, I would just iterate (zrange/zrangebyscore) and sum the values client side.
If, on the other hand, you are talking about many thousands - millions of items, you can always keep a reference set with running totals for each user and increment/decrement them as the gifts are sent.
So when you do your ZINCR 123:gifts 1 "3|345", you could do a seperate ZINCR command, which could be something like this:
ZINCR received-gifts 1 <user_id>
Then, to get the # of gifts received for a given user, you just need to run a ZSCORE:
ZSCORE received-gifts <user_id>
Here is a little lua script that maintains the zset score total as you go, in a counter with key postfixed with '.ss'. You can use it instead of ZADD.
local delta = 0
for i=1,#ARGV,2 do
local oldScore = redis.call('zscore', KEYS[1], ARGV[i+1])
if oldScore == false then
oldScore = 0
end
delta = delta - oldScore + ARGV[i]
end
local val = redis.call('zadd', KEYS[1], unpack(ARGV))
redis.call('INCRBY', KEYS[1]..'.ss', delta)