I have a a list in redis containing a sequence of Ids. Each id is unique for a single object which I am storing as a JSON string on a separate key.
So I have something like:
redis> LRANGE mylist 0 -1
1) "one"
2) "two"
3) "three"
And I have separate keys mylist:one, mylist:two, mylist:three.
I am saving the ids to a list in order to build a simple FIFO queue on my application.
What is the most efficient way to get all the ids in mylist and their matching values from each individual key? Is there a better way to go about it?
The most efficient way is probably to use the SORT command:
# Populate list
rpush mylist one two three
set mylist:one 1
set mylist:two 2
set mylist:three 3
# Retrieve all items with their corresponding values
sort mylist by nosort get # get mylist:*
1) "one"
2) "1"
3) "two"
4) "2"
5) "three"
6) "3"
Related
I have this set of keys and values that I need to eventually sort by the number of keys' occurences. I'm aware that Redis isn't suppose to work like this, but hoping there may be some smart workaround*.
Schema requirements:
Allow each key to occur any number of times.
Have each value expire after a certain amount of time (and the key with it).
Keep each full pair unique.
Known constraints:
No inbuilt way to expire values, just keys.
Keys can't be duplicated even when they have different values (or can they?)
Using sets or other methods doesn't allow easy counting either (tried that too...)
So apparently the requirements can only be met by grouping both key and value in the Redis key (while assigning them with null/random values and ttls), like this...
Input keys:
"apples:123"
"oranges:123"
"bananas:456"
"apples:456"
"oranges:789"
"apples:789"
[then maybe another hundred or so such pairs]
Expected output:
apples, oranges, bananas
[or apples(3), oranges(2), bananas(1) – but I'll then ditch the numbers anyway.]
* while it can be done in app's logic, I think it loses in efficiency as it needs to get all data at once and cycle through each item, when all I need is a rather limited subset.
So right now I'd have to do it like this (node.js)...
client.keys('*').then(response => {
let occurences = {}
response.forEach(function (pair){
let fruit = pair.split(':')[0]
occurences[fruit] = (occurences[fruit] || 0) + 1
})
let topfruits = Object.keys(occurences).sort((a, b) => occurences[a] - occurences[b]).reverse().slice(0, 3)
console.log(topfruits)
})
// (client.scan in production, which makes it more complicated and doesn't help that much for this use case)
...migrating from a SQL query that does it in one line:
let topfruits = 'SELECT fruit, number, count (fruit) AS occurences FROM fruits GROUP BY fruit ORDER BY occurences DESC LIMIT 3'
This is a great Redisearch Aggregation problem
you can have multiple rows of fruits and counts
They can be expired by using TTLs (EXPIRE command)
These can all be unique (I used order# but it could be a UUID or some other generated informatio)
127.0.0.1:6379> FT.CREATE fruitIndex ON HASH PREFIX 1 fruit_order: SCHEMA fruit TEXT quantity NUMERIC
OK
127.0.0.1:6379> HSET fruit_order:100 fruit bananas quantity 2
(integer) 2
127.0.0.1:6379> HSET fruit_order:101 fruit bananas quantity 200
(integer) 2
127.0.0.1:6379> HSET fruit_order:103 fruit apples quantity 12
(integer) 2
127.0.0.1:6379> FT.AGGREGATE fruitIndex "*" GROUPBY 1 #fruit REDUCE SUM 1 quantity as totals SORTBY 2 #totals DESC
1) (integer) 3
2) 1) "fruit"
2) "bananas"
3) "totals"
4) "202"
3) 1) "fruit"
2) "oranges"
3) "totals"
4) "25"
4) 1) "fruit"
2) "apples"
3) "totals"
4) "12"
127.0.0.1:6379> EXPIRE fruit_order:101 5
(integer) 1
## Wait 5 seconds and re-run the query and you can see that order drop out
127.0.0.1:6379> FT.AGGREGATE fruitIndex "*" GROUPBY 1 #fruit REDUCE SUM 1 quantity as totals SORTBY 2 #totals DESC
1) (integer) 3
2) 1) "fruit"
2) "oranges"
3) "totals"
4) "25"
3) 1) "fruit"
2) "apples"
3) "totals"
4) "12"
4) 1) "fruit"
2) "bananas"
3) "totals"
4) "2"
I have a redis hash store that looks like Item:<id>, with attribute name. I want to filter the hash store by a prefix for name attribute.
What I'm trying to do is store the name (lowercased) in a separate Z-set called Item:::name while setting the score to 0. By doing this, I'm successfully able to get the desired result using ZRANGEBYLEX however I'm unable to map the results back to the original Items. How should I go about implementing something like this?
I've seen multiple autocomplete examples for Redis which require the same functionality but without linking the words back to an actual Item (hash in this case)
In sorted sets the member can't be duplicated, it has to be unique. So different users with the same name will cause problem.
My suggestion requires application layer coding to parse response array and executing hash commands (it will be like secondary indexes);
127.0.0.1:6379> HSET user:1 name jack
(integer) 1
127.0.0.1:6379> HSET user:2 name john
(integer) 1
127.0.0.1:6379> HSET user:3 name keanu
(integer) 1
127.0.0.1:6379> HSET user:4 name jack
(integer) 1
127.0.0.1:6379> ZADD item:names 0 jack::user:1 0 john::user:2 0 keanu::user:3 0 jack::user:4
(integer) 4
127.0.0.1:6379> ZRANGE item:names 0 -1 WITHSCORES
1) "jack::user:1"
2) "0"
3) "jack::user:4"
4) "0"
5) "john::user:2"
6) "0"
7) "keanu::user:3"
8) "0"
127.0.0.1:6379> ZRANGEBYLEX item:names [jack [jo
1) "jack::user:1"
2) "jack::user:4"
At the end you will have name::hash-key formatted array elements. At application layer if you separate each element to two substrings by using ::(any other string such as !!! or || etc) you will have user:1 and user:4.
127.0.0.1:6379> HGETALL user:1
1) "name"
2) "jack"
127.0.0.1:6379> HGETALL user:4
1) "name"
2) "jack"
127.0.0.1:6379>
I have a large set of ids (around 100000) which I want to store in redis.
I am looking for the most optimal way through which I can check if a given list of ids, what are the ids that are part of my set.
If I use a redis set, I can use SISMEMBER to check if a id is part of my set, but in this case I want to check if, given a list of ids, which one is part of my set.
Example:
redis> SADD myset "1"
(integer) 1
redis> SADD myset "2"
(integer) 2
redis> MYCOMMAND myset "[1,2,4,5]"
(list) 1, 2
Does anything of this sort exist already ?
thanks !
How do I get most weighted elements from a sorted set, but excluding those found in another set(or list or hash).
>zadd all 1 one
>zadd all 2 two
>zadd all 3 three
>sadd disabled 2
>sdiff all disabled
(error) WRONGTYPE Operation against a key holding the wrong kind of value
Is my only option is to get elements from the sorted set one-by-one and compare to the list of "disabled" items? Wouldn't that be very slow because of so many transactions to a server?
What is the approach here?
Note: I assume you've meant sadd disabled two
As you've found out, SDIFF does not operate on sorted sets - that is because defining the difference between sorted sets isn't trivial.
What you could do is first create a temporary set with ZUNIONSTORE and set the intersect's scores to 0. Then do a range excluding the 0, e.g.:
127.0.0.1:6379> ZADD all 1 one 2 two 3 three
(integer) 3
127.0.0.1:6379> SADD disabled two
(integer) 1
127.0.0.1:6379> ZUNIONSTORE tmp 2 all disabled WEIGHTS 1 0 AGGREGATE MIN
(integer) 3
127.0.0.1:6379> ZREVRANGEBYSCORE tmp +inf 1 WITHSCORES
1) "three"
2) "3"
3) "one"
4) "1"
I have an ordered set in Redis (I am actually using a python client https://github.com/andymccurdy/redis-py), for example:
zadd myset 1 key1
zadd myset 1 key2
zadd myset 1 key3
zadd myset 0 key4
Note that 3 keys have the same score.
Using ZRANGE, i would like to get the top 2 entries (i.e lowest scores). "key4" will always be the first result as it has a lower value, but I would like the second return value to be randomly selected between the ties: key1,key2,key3. ZRANGE actually returns the keys in the order they are indexed: "keys1" is always my second result:
zrange myset 0 -1 WITHSCORES
1) "key4"
2) "0"
3) "key1"
4) "1"
5) "key2"
6) "1"
7) "key3"
8) "1"
any idea?
thanks,
J.
As kindly requested by Linus G Thiel, here are more details about my usecase:
I would like to use zsets to perform a simple ranking system. I have a list of items, for each one a score representing the relevance of the item. For the cold start of my system, most of the scores will be identical (i.e 0), and I would like to randomly select among the items having the same score. Otherwise I will always return the exact same lexicographic ordering, which will introduce a bias in the system.
The solution you propose, using one specific set for each duplicated score value will work. I will give it a try.
Thanks,