I have a large set of ids (around 100000) which I want to store in redis.
I am looking for the most optimal way through which I can check if a given list of ids, what are the ids that are part of my set.
If I use a redis set, I can use SISMEMBER to check if a id is part of my set, but in this case I want to check if, given a list of ids, which one is part of my set.
Example:
redis> SADD myset "1"
(integer) 1
redis> SADD myset "2"
(integer) 2
redis> MYCOMMAND myset "[1,2,4,5]"
(list) 1, 2
Does anything of this sort exist already ?
thanks !
Related
I'm tracking members in multiple Sorted Sets in Redis as a way to do multi-column indexing on the members. As an example, let's say I have two Sorted Sets, lastseen (which is epoch time) and points, and I store usernames as members in these Sorted Sets.
I'm wanting to use ZREVRANGEBYSCORE against lastseen to get a list of users who were recently seen, but I only want a user to be included in the results if its also present in the points Sorted Set.
I looked at using ZINTERSTORE to help me do this, but the available AGGREGATE options are not conducive to my scenario. If one of the AGGREGATE options was to use the score from a specific set provided to the ZINTERSTORE command, then it would work, but that's not an option.
---EDIT---
Example:
redis> ZADD lastseen 12345 "foo"
redis> ZADD lastseen 12346 "bar"
redis> ZADD lastseen 12347 "sucka"
redis> ZADD points 5 "foo"
redis> ZADD points 9 "bar"
Now, if I run:
redis> ZREVRANGEBYSCORE lastseen +inf -inf
1) "sucka"
2) "bar"
3) "foo"
What I'm looking for is a way to only get "foo" and "bar" returned by the call to ZREVRANGEBYSCORE for lastseen since "sucka" isn't resent in the points Sorted Set.
I can do:
redis> ZINTERSTORE out 2 lastseen points
redis> ZRANGE out 0 -1 WITHSCORES
1) "foo"
2) "12350"
3) "bar"
4) "12355"
As you can see, the ZINTERSTORE does cut out the "sucka" member, but the scores for the other two members get aggregated using SUM. There's two other aggregation options (MIN and MAX) available for use with ZINTERSTORE. In this case, using MAX would give me what I'm looking for, but that wouldn't be the case if a user's points were greater than the epoch value for the last time they were seen.
You can get what you're asking by setting the WEIGHT of points to 0 and of lastseen to 1 when running ZINTERSTORE so that only the weight of lastseen is considered:
redis> ZINTERSTORE out 2 lastseen points WEIGHTS 1 0
(integer) 2
Then, depending on what ordering you're going for, you can either:
redis> ZRANGE out 0 -1 WITHSCORES
1) "foo"
2) "12345"
3) "bar"
4) "12346"
or:
redis> ZREVRANGE out 0 -1 WITHSCORES
1) "bar"
2) "12346"
3) "foo"
4) "12345"
Here is the hash set I have
HSET MySet 111222333 Tom
HSET MySet 444555666 Julia
HSET MySet 777888999 Paul
You can think about the set field as a phone number, and the SET value as a person's name.
I need to get all records that contains "23" and "89" inside the phone number.
It's possible doing 2 requests and merge them on the server:
HSCAN MySet 0 MATCH *23*
HSCAN MySet 0 MATCH *89*
Can we do the same thing using one expression? Like this
HSCAN MySet 0 Match *23* OR *89*
OR
HSCAN MySet 0 Match *23|89*
Not directly - Redis does glob-style pattern matching and that's not really supported.
What you could do is use a little bit of Lua magic to craft your own efficient filtering, similarly to the example in this answer: https://stackoverflow.com/a/29945372/3160475
How do I get most weighted elements from a sorted set, but excluding those found in another set(or list or hash).
>zadd all 1 one
>zadd all 2 two
>zadd all 3 three
>sadd disabled 2
>sdiff all disabled
(error) WRONGTYPE Operation against a key holding the wrong kind of value
Is my only option is to get elements from the sorted set one-by-one and compare to the list of "disabled" items? Wouldn't that be very slow because of so many transactions to a server?
What is the approach here?
Note: I assume you've meant sadd disabled two
As you've found out, SDIFF does not operate on sorted sets - that is because defining the difference between sorted sets isn't trivial.
What you could do is first create a temporary set with ZUNIONSTORE and set the intersect's scores to 0. Then do a range excluding the 0, e.g.:
127.0.0.1:6379> ZADD all 1 one 2 two 3 three
(integer) 3
127.0.0.1:6379> SADD disabled two
(integer) 1
127.0.0.1:6379> ZUNIONSTORE tmp 2 all disabled WEIGHTS 1 0 AGGREGATE MIN
(integer) 3
127.0.0.1:6379> ZREVRANGEBYSCORE tmp +inf 1 WITHSCORES
1) "three"
2) "3"
3) "one"
4) "1"
Assume I have a set (or sorted set or list if that would be better) A of 100 to 1000 strings.
Then I have a sorted set B of many more strings, say one million.
Now C should be the intersection of A and B (of the strings of course).
I want to have every tuple (X, SCORE_OF_X_IN_B) where X is in C.
Any Idea?
I got two ideas:
Interstore
store A a sorted set with every score being 0
interstore to D
get every item of D
delete D
Simple loop in client
loop over A in my client programm
get zscore for every string
While 1. has way too much overhead on the redis side (Has to write for example. The redis page states quite a high time complexity, too http://redis.io/commands/zinterstore), 2. would have |A| database connections and won't be a good choice.
Maybe I could write a redis/lua script which will work like zscore but with an arbitrary number of strings, but I'm not sure if my hoster allows scripts...
So I just wanted to ask SO, if there is an elegant and fast solution available without scripting!
There is a simple solution to your problem: ZINTERSTORE will work with a SET and a ZSET. Try:
redis> sadd foo a
(integer) 1
redis> zadd bar 1 a
(integer) 1
redis> zadd bar 2 b
(integer) 1
redis> zinterstore baz 2 foo bar AGGREGATE MAX
(integer) 1
redis> zrange baz 0 -1 withscores
1) "a"
2) "1"
Edit: I added AGGREGATE MAX above, since redis will give each member of the (non-sorted) set foo a default score of 1, and SUM that with whatever score it has in the (sorted) set bar.
I'm planning to start using hashes insead of regular keys. But I can't find any information about multi get for hash-keys in Redis wiki. Is this kind of command is supported by Redis?
Thank you.
You can query hashes or any keys in pipeline, i.e. in one request to your redis instance. Actual implementation depends on your client, but with redis-py it'd look like this:
pipe = conn.pipeline()
pipe.hgetall('foo')
pipe.hgetall('bar')
pipe.hgetall('zar')
hash1, hash2, hash3 = pipe.execute()
Client will issue one request with 3 commands. This is the same technique that is used to add multiple values to a set at once.
Read more at http://redis.io/topics/pipelining
No MHGETALL but you can Lua it:
local r = {}
for _, v in pairs(KEYS) do
r[#r+1] = redis.call('HGETALL', v)
end
return r
If SORT let you use multiple GETs with the -> syntax, and all your hashes had the same fields, you could get them in a bulk reply by putting their names into a set and sorting that.
SORT names_of_hashes GET *->field1 *->field2 *->field3 *->etc
But it doesn't look like you can do that with the hash access. Plus you'd have to turn the return list back into hashes yourself.
UPDATE: Redis seems to let you fetch multiple fields if you name your hashes nicely:
redis> hset hash:1 name fish
(integer) 1
redis> hset hash:2 name donkey
(integer) 1
redis> hset hash:3 name horse
(integer) 1
redis> hset hash:1 type fish
(integer) 1
redis> hset hash:2 type mammal
(integer) 1
redis> hset hash:3 type mammal
(integer) 1
redis> sadd animals 1
(integer) 1
redis> sadd animals 2
(integer) 1
redis> sadd animals 3
(integer) 1
redis> sort animals get # get hash:*->name get hash:*->type
1. "1"
2. "fish"
3. "fish"
4. "2"
5. "donkey"
6. "mammal"
7. "3"
8. "horse"
9. "mammal"
There is no command to do it on one shot, but there is a way to do it "nicely", using a list (or sorted set) where you would store you hashKeys, and then retrieve them as bulk using multi.
In PHP:
$redis->zAdd("myHashzSet", 1, "myHashKey:1");
$redis->zAdd("myHashzSet", 2, "myHashKey:2");
$redis->zAdd("myHashzSet", 3, "myHashKey:3");
$members = $redis->zRange("myHashzSet", 0, -1);
$redis->multi();
foreach($members as $hashKey) {
$redis->hGetAll($hashKey);
}
$results = $redis->exec();
I recommand using a sorted set, where you use the score as an ID for your hash, it allows to take advantages of all score based command.
Redis has a HMGET command, which returns the values of several hash keys with one command.