redis how to combine multiple commands in 1 query - redis

How to Query
red ferrari with limited edition
with topSpeed between 200 To 210 And
Price between 190 to 205
DATA
HMSET cars:1 make ferrari Price 199 limited yes color red topSpeed 202
HMSET cars:2 make porsche Price 555 limited no color yellow topSpeed 500
SADD make:ferrari 1
SADD color:red 1
SADD limited:yes 1
ZADD Price 199 1
ZADD topSpeed 202 1
SADD make:porsche 2
SADD color:yellow 2
SADD limited:no 2
ZADD Price 555 2
ZADD topSpeed 500 2
I Tried & Don't know how to add multiple range for price and topSpeed both?
multi.ZINTERSTORE('tempTom',4,
'color:red',
'make:ferrari',
'limited:yes',
'topSpeed'
);
multi.ZRANGEBYSCORE('tempTom' , 202 ,205) //range for topSpeed
//so how to add range for Price also ?
Output
[1,[]]
What am i doing wrong , how to query with multiple commands 1 after another ?

AFAIK - This is pretty much impossible to do with 1 query in Redis. You are trying to use a key-vale store as a relational database. There is a way to do it in Redis but with 2 queries. However, you can wrap it as a single transaction in MULTI / EXEC and thus effectively making it 1 query.
For example:
Create 2 more sets, one for topSpeed and another for price. Then just perform a SINTER between those two (first query). Then use that result to query your car (second query).
Explanation:
# Inserting cars top speed
ZADD car:top-speed:210 "ferrari"
ZADD car:top-speed:300 "porsche"
# Inserting cars price (e.g. both cars have the same price)
ZADD car:price:190 "ferrari" "porsche"
# Using SINTER to get all cars with top speed of 210 and a price of 190
SINTER car:top-speed:210 car:price:300 // output "ferrari"
Use the output to query your car set and get all the other details (Don't forget the MULTI / EXEC).
You can add more 'filters' by simply adding more sets and performing the intersection on all of them. You would however, have to populate all these sets whenever you add a new car. But this is normal in Redis and does not suffer big performance issues. You can check here for another similar example if my explanation is not clear.
Hope this helps

Perhaps EVAL is what you are looking for? (perhaps not)..
It uses LUA scripts to execute a batch of commands in one call (and optionally return the result).
It is KIND OF like a stored procedure for REDIS. The uploaded LUA gets cached so you don't need to load it over and over.
The examples I see aren't EXACTLY what you are looking for but demonstrate multi-part queries. You need to read down into them to see the meat before you give up on EVAL (I almost did).
Object Queries with Redis
A Speed Guide To Redis Lua Scripting
EVAL (from the redis docs)
The only reason I know about this is because a coworker here insists on using redis as a relational database. It is not unlike using a hammer as a screwdriver.

Related

How to get subset of SMEMBERS result? Or should I use SortedSet with the same value for both score & member?

I am new to Redis. For example, if I have the following schema:
INCR id:product
SET product:<id:product> value
SADD color:red <id:product>
(Aside: I am not sure how to express a variable in Redis. I will just use <id:product> as the primary key value. In production, I will use golang client for this job)
To query products which have red color, I can do:
SMEMBERS color:red
But the problem is I just want to display 10 of them in the first page, and then next 10 in the second page and so on. How to let Redis return only part of them by specifying offset and limit arguments?
What do redis experts normally do for this case? Return all IDs even if I just want 10 of them? Is that efficient? What if it has millions of values in the set, but I only want 10?
Edited 1
Incidentally, I use sets instead of lists and sorted sets because I will need to do SINTER jobs for other queries.
For example:
SADD type:foo <id:product>
SINTER color:red type:foo
And then I will have pagination problem again. Because I actually just want to find 10 of the intersection at a time. (eg: if the intersection returns millions of keys, but actually I just want 10 of them at a time for pagination).
Edited 2
Should I use a sorted set instead? I am not sure if this is the expert choice or not. Something like:
ZADD color:red <id:product> <id:product>
ZADD type:foo <id:product> <id:product>
ZRANGE color:red 0 9 // for the first page of red color products
ZINTERSTORE out 2 color:red type:foo AGGREGATE MIN
ZRANGE out 0 9 // for the first page of red color and type foo products
I have no ideas if the above way is suggested or not.
What will happen if multiple clients are creating the same out sorted set?
Is that meaningful to use the same value for both score and member?
Using sorted sets is the standard way to do pagination in Redis.
The documentation of ZINTERSTORE says that: "If destination already exists, it is overwritten."
Therefore, you shouldn't use "out" as the destination key name. You should instead use a unique or sufficiently random key name and then delete it when you're done.
I'm not sure what you mean by "meaningful". It's a fine choice if that's the order you want them to be in.

Creating a new set from a range of a Sorted Set in Redis

I have a number of sorted sets that are used as secondary indexes on my system and user queries could hit a number of them.
ZADD scoreSet 1 "fred"
ZADD scoreSet 5 "bob"
ZADD scoreSet 2 "spodrick"
ZADD ageSet 25 "fred"
ZADD ageSet 29 "bob"
ZADD ageSet 38 "spodrick"
To use these indexes to get all users under 30 with a score >2
ZRANGEBYSCORE scoreSet (2 +inf
(store these in my application code)
ZRANGEBYSCORE ageSet -inf (30
(store these in my application code)
(Perform Set intersection in my application code)
but this means I have copied all the data from redis to my app server to perform the intersections, is there a more efficient way to do this where I am not transporting all the matching ranges across the network and instead doing the intersections in Rediss?
What I would like is
ZRANGEBYSCORESTORE tempSet1 scoreSet (2 +inf
ZRANGEBYSCORESTORE tempSet2 ageSet -inf (30
SINTER tempSet1 tempSet2
Where ZRANGEBYSCORESTORE performs the ZRANGEBYSCORE operation and stores the results in a new set instead of returning them. That way Redis does all the heavy lifting and only sends me the actual intersection I am interested in.
How could I do this, since I can do ZINTERSTORE and ZUNIONSTORE not having RANGESTORE seems like I am missing something.
That's a question with many possible answers. Here're my top two:
Use the power of Lua to avoid moving data to the client and have all the work done server-side. You can work around the lack of RANGESTORE this way.
Maintain a sorted set where the scores are made of both attributes, using bit-interleaving. In the Redis-verse there exists an experiment by antirez called redimension which does exactly that. The links are https://www.reddit.com/r/redis/comments/3qjlkk/redimension_ruby_library_implementing/ and my port of the above to Redis lua https://www.reddit.com/r/redis/comments/3s0h73/luaredimension_redis_multidimensional_query/. You can read more about the approach at Multi dimensional indexes.

Storing game profiles for high scores / leaderboards in Redis

Hey I am currently working on a game that has over 1 Million users, I've had a leaderboard for some time ranking people by their kills/deaths/kill streak and more using MySQL, however doing all these SQL calls doesn't seem to be good so I've started playing around with Redis.
So here are the options I have in mind.
Store all the data with ZADD, for example, then grab it all and use the ZRANGE and ZRANK to find the ranks
put player_kills player1 100
put player_deaths player_2 200
However the problem I have with this is that for example, what if I only wanted to go over players that have over 10 kills ? Should I start storing user data in hashes and sort from there ? Please let me know if you have any examples or ideas I could use.
To use the Sorted Set data structure, you'll need to call Redis' ZADD command as follows (not that the score precedes the member):
ZADD player_kills 100 player1
Once you have all kills recorded, you can use the ZRANGEBYSCORE command to query and in your case:
ZRANGEBYSCORE player_kills 10 +inf
If you only want players with over 10 kills, just use zrangebyscore, like so:
ZRANGEBYSCORE player_kills 10 20

Creating a workable Redis store with several filters

I am working on a system to display information about real estate. It runs in angular with the data stored as a json file on the server, which is updated once a day.
I have filters on number of bedrooms, bathrooms, price and a free text field for the address. It's all very snappy, but the problem is the load time of the app. This is why I am looking at Redis. Trouble is, I just can't get my head round how to get data with several different filters running.
Let's say I have some data like this: (missing off lots of fields for simplicity)
id beds price
0 3 270000
1 2 130000
2 4 420000
etc...
I am thinking I could set up three sets, one to hold the whole dataset, one to create an index on bedrooms and another for price:
beds id
2 1
3 0
4 2
and the same for price:
price id
130000 1
270000 0
420000 2
Then I was thinking I could use SINTER to return the overlapping sets.
Let's say I looking for a house with more than 2 bedrooms that is less than 300000.
From the bedrooms set I get IDs 0,2 for beds > 2.
From the prices set I get IDs 0,1 for price < 300000
So the common id is 0, which I would then lookup in the main dataset.
It all sounds good in theory, but being a Redis newbie, I have no clue how to go about achieving it!
Any advice would be gratefully received!
You're on the right track; sets + sorted sets is the right answer.
Two sources for all of the information that you could ever want:
Chapter 7 of my book, Redis in Action - http://bitly.com/redis-in-action
My Python/Redis object mapper - https://github.com/josiahcarlson/rom (it uses ideas directly from chapter 7 of my book to implement sql-like indices)
Both of those resources use Python as the programming language, though chapter 7 has been translated into Java: https://github.com/josiahcarlson/redis-in-action/ (go to the java path to see the code).
... That said, a normal relational database (especially one with built-in Geo handling like Postgres) should handle this data with ease. Have you considered a relational database?

Multi-parameter match-finder with Redis

I need to create an match-finder system for some data set, as follows:
There is a set of objects, each identified by a string ObjectID.
Each object has exactly N properties Pi. Each property value is a string.
Database example for N = 3 (in real life N = 8).
ObjectID: P1 P2 P3
--------------------------------
APPLE: RED ROUND FRUIT
ORANGE: ORANGE ROUND FRUIT
CARROT: RED LONG VEGETABLE
The system must return sets of ObjectIDs, matching given query on object properties. In query user must specify all property values. Alternatively, for some or all properties in query user may specify a "wildcard" *, meaning that any property value would match criteria.
Example queries:
P1 P2 P3 => Result
------------------------------------
* ROUND FRUIT => APPLE, ORANGE
RED LONG VEGETABLE => CARROT
RED * * => CARROT, APPLE
All of this is trivially done with SQL.
The question is: is there a neat way to do that with Redis?
Note that I'm interested in Redis-based solutions specifically, for self-education purposes; other DBs are offtopic for this particular question.
Update: Trivial solution with explicit ObjectID lists for each Pi and application-side filtering does not look neat enough to me :-)
What you are trying to do here is an inverted index.
For each column, have it map to a "set". Then, you can intersect the sets to get the result.
So, APPLE: RED ROUND FRUIT would map to the following inserts:
SADD p1:RED APPLE
SADD p2:ROUND APPLE
SADD p3:FRUIT APPLE
Then, let's say I want to query for * ROUND FRUIT, I would do:
SINTER p2:ROUND p3:FRUIT
This command is taking the intersection of the items in the p2:ROUND set and the p3:FRUIT set. This will return all the items that are ROUND and FRUIT, not caring what p1 is.
Some other examples:
SMEMBERS p1:GREEN
SINTER p1:RED p2:ROUND p3:FRUIT
SUNION p1:RED p1:GREEN
My above answer is going to use some computation power because the intersection operation is O(N*M). Here is a way of doing it that is more memory intensive, but will have faster retrieval because it effectively precomputes the indexes.
For each combination of properties, make a key that stores a set:
So, APPLE: RED ROUND FRUIT would map to the following inserts:
SADD RED:ROUND:FRUIT APPLE
SADD :ROUND:FRUIT APPLE
SADD RED::FRUIT APPLE
SADD RED:ROUND: APPLE
SADD RED:: APPLE
SADD :ROUND: APPLE
SADD ::FRUIT APPLE
SADD ::: APPLE
Then, to query, you simply access the respective key. For example, * ROUND FRUIT would simply be
SMEMBERS :ROUND:FRUIT
Obviously, this doesn't scale well at all in terms of memory when you have many dimensions, but it will be extremely snappy to retrieve results.