What's the best way to keep a large list (e.g. 10K items) in Redis, where I also want to efficiently retrieve items by key.
It seems Redis has no data structure equivalent to Java's OrderedHashMap, which accomplishes this, so maybe it's necessary to maintain a set and a list and ensure they stay in sync.
Use a sorted set;
Add some bookmarks; use current time for score to sort chronologically:
> zadd bookmarks 123 "bk1"
> zadd bookmarks 456 "bk2"
> zadd bookmarks 789 "bk3"
> zadd bookmarks 999 "bk4"
To get a bookmark, you need the index first:
> zrank bookmarks "bk3"
> "3"
...then pull the bookmark by index:
> zrevrange bookmarks 3 3
> "bk3"
If you don't want to use timestamps, you can sort bookmarks lexicographically using "1" for score:
> zadd bookmarks 1 "link_xyz"
> zadd bookmarks 1 "link_abc"
> zadd bookmarks 1 "link_foo"
> zrange bookmarks 0 -1
1) "link_abc"
2) "link_foo"
3) "link_xyz"
The index lookup is O(log(n)), add to that O(log(n)+1) to pull a single member by index; better than O(n) for lists.
Also, if you add the same bookmark twice, redis automatically replaces the previous member, so you avoid duplicates.
Hope it helps,
Related
In the redis documentation, there is this example of ZRANGEBYLEX
ZADD myindex 0 0056:0028.44:90
ZADD myindex 0 0034:0011.00:832
ZRANGEBYLEX myindex [0056:0010.00 [0056:0030.00
1) "0056:0028.44:90"
It is very straightforward.
However if I want to apply the same technique to the following example,
127.0.0.1:6379> zadd feedbacks 1 feedback1 2 feedback2 3 feedback3 1 feedback4
(integer) 4
127.0.0.1:6379> ZRANGEBYLEX feedbacks [feed [feed
(empty list or set)
I get an empty set.
I would expect to see the four values (feedback1 to feedback4)
Why ZRANGEBYLEX failed on my test sample?
It fails because they have different scores. ZRANGEBYLEX works only on same-score subsets.
See https://redis.io/commands/ZRANGEBYLEX
When all the elements in a sorted set are inserted with the same
score, in order to force lexicographical ordering, this command
returns all the elements in the sorted set at key with a value between
min and max.
If the elements in the sorted set have different scores, the returned
elements are unspecified.
Sorted sets have the property of being lexicographically ordered within same-score subsets. This gives them a second use-case, a lexicographically sorted set, but in this case, you add all elements with the same score.
So, you have to choose how to use your sorted set:
Sorted by score (with same-score sorted lex, for predictable order)
Sorted lexicographically, all elements given the same score
You cannot have both. You'd need two sorted sets then.
I have an index for each user which represents their current score.
I want to find all users with the same score.
Provided that the input is as follows, how can I retrieve only users 1 and 2?
ZADD users 0 1:10
ZADD users 0 2:10
ZADD users 0 3:5
I would expect to do something like mentioned in the docs but I cannot use the key as my index since it is too large (for the post I changed to 1,2,3 so it is not an issue), which is why I set it to 0.
I thought the command could be something like ZRANGEBYLEX users [0:10 (:10 but that returns all 3.
1) "1:10"
2) "2:10"
3) "3:5"
I was able to come up with a solution for this problem, which can further refine as:
Retrieve all users with a given score or retrieve a single score for a single user.
What I determined was that using the namespace users is not a sufficient descriptor for both the score and user id. When using 2 different keys, it is clearly possible to get either a range or a single value. The downside is twice the writes I had hoped, but that is overall not a lot of overhead for me and the only way I found to properly retrieve both score and user.
ZADD score:users 0 10:1
ZADD score:users 0 10:2
ZADD score:users 0 5:3
ZADD users:score 0 1:10
ZADD users:score 0 2:10
ZADD users:score 0 3:5
ZRANGEBYLEX users:score [1: [1:\xff # "1:10"
ZRANGEBYLEX score:users [10: [10:\xff # "10:1" "10:2"
How do I get most weighted elements from a sorted set, but excluding those found in another set(or list or hash).
>zadd all 1 one
>zadd all 2 two
>zadd all 3 three
>sadd disabled 2
>sdiff all disabled
(error) WRONGTYPE Operation against a key holding the wrong kind of value
Is my only option is to get elements from the sorted set one-by-one and compare to the list of "disabled" items? Wouldn't that be very slow because of so many transactions to a server?
What is the approach here?
Note: I assume you've meant sadd disabled two
As you've found out, SDIFF does not operate on sorted sets - that is because defining the difference between sorted sets isn't trivial.
What you could do is first create a temporary set with ZUNIONSTORE and set the intersect's scores to 0. Then do a range excluding the 0, e.g.:
127.0.0.1:6379> ZADD all 1 one 2 two 3 three
(integer) 3
127.0.0.1:6379> SADD disabled two
(integer) 1
127.0.0.1:6379> ZUNIONSTORE tmp 2 all disabled WEIGHTS 1 0 AGGREGATE MIN
(integer) 3
127.0.0.1:6379> ZREVRANGEBYSCORE tmp +inf 1 WITHSCORES
1) "three"
2) "3"
3) "one"
4) "1"
How do I get the value of multiple keys from redis using a sorted set?
zadd Users 0 David
zadd Users 5 John
zadd Users 15 Linda
zrevrange Users 0 -1 withscores
This will have two users in it.
How can I retrieve the users with key 'David' and 'Linda' in one query?
You can use Redis MGET
redis> MGET key1 key2 nonexisting
1) "Hello"
2) "World"
3) (nil)
More here
http://redis.io/commands/mget
There are multiple ways to do it without introducing a new command in Redis.
For instance, you can fill a temporary set with the names you are interested in, then calculate the intersection between the temporary set and the zset:
multi
sadd tmp David Linda ... and more ...
zinterstore res 2 tmp Users weights 0 1
zrange res 0 -1 withscores
del tmp res
exec
With pipelining, this will only generate one roundtrip and you can fill an arbitrary number of input parameters in tmp.
With Redis 2.6, you can also wrap these lines into a server-side Lua script to finally get a command accepting an input list and returning the result you want:
eval "redis.call( 'sadd', 'tmp', unpack(KEYS) );
redis.call( 'zinterstore', 'res', 2, 'tmp', 'Users', 'weights', 0, 1 );
local res = redis.call( 'zrange', 'res', 0, -1, 'withscores' );
redis.call( 'del', 'res', 'tmp' ) ;
return res
" 2 David Linda
You can safely assume no new command will be added to Redis if it can easily been implemented using scripting.
One uses a sorted set because you want to deal with items that are sorted. What you are asking for is to not use a sorted set as a sorted set. If you don't care about sort order, then perhaps a sorted set is not what you are looking for. You already can retrieve multiple keys, but not arbitrary ones.
If your primary goal is to retrieve multiple arbitrary keys, use a hash and hmget. If your primary need is to access a sorted set, use sorted set and either go the scripting route or pipeline a series of zscore calls.
You cannot get this with one command. The closest you can do to get it in one response:
MULTI
ZSCORE Users David
ZSCORE Users Linda
EXEC
EDIT: Alternatively, you can maintain a parallel hash with users' scores, and query it with
HMGET UserScores David Linda
Assume I have a set (or sorted set or list if that would be better) A of 100 to 1000 strings.
Then I have a sorted set B of many more strings, say one million.
Now C should be the intersection of A and B (of the strings of course).
I want to have every tuple (X, SCORE_OF_X_IN_B) where X is in C.
Any Idea?
I got two ideas:
Interstore
store A a sorted set with every score being 0
interstore to D
get every item of D
delete D
Simple loop in client
loop over A in my client programm
get zscore for every string
While 1. has way too much overhead on the redis side (Has to write for example. The redis page states quite a high time complexity, too http://redis.io/commands/zinterstore), 2. would have |A| database connections and won't be a good choice.
Maybe I could write a redis/lua script which will work like zscore but with an arbitrary number of strings, but I'm not sure if my hoster allows scripts...
So I just wanted to ask SO, if there is an elegant and fast solution available without scripting!
There is a simple solution to your problem: ZINTERSTORE will work with a SET and a ZSET. Try:
redis> sadd foo a
(integer) 1
redis> zadd bar 1 a
(integer) 1
redis> zadd bar 2 b
(integer) 1
redis> zinterstore baz 2 foo bar AGGREGATE MAX
(integer) 1
redis> zrange baz 0 -1 withscores
1) "a"
2) "1"
Edit: I added AGGREGATE MAX above, since redis will give each member of the (non-sorted) set foo a default score of 1, and SUM that with whatever score it has in the (sorted) set bar.