Calculate number of matching properties in Redis - redis

Question

I'd like to see if a migration of a dataset from PostgreSQL to Redis has a positive influence on a particular search query. Unfortunately, I don't really know how to organize the keys and values.
What I want is a that users are able to supply a list of properties and the application delivers a list of items in an ascending order of properties that have not been entered.
For example:
Item #1 { prop1, prop2, prop4, prop7 } Query: "prop1 prop3 prop4 prop5"
Item #2 { prop7, prop8 } Result: Item #3
Item #3 { prop2, prop3, prop5 } Item #1
Item #2
What I have come up with so far:
#!/usr/bin/python
properties = (1, 3, 4, 5)
items = ["Properties:%s:items" % id for property in properties ]
redis.zunionstore("query:related_items", items)
redis.zinterstore("query:result", { "Items:all": 1, "query:related_items": -1 })
This builds a sorted set of Items (all with a score of 1) that are connected with the user-entered Propertys. Afterwards, an intersection with the sorted set of all Items (where each value's score is the number of Propertys) is calculated. The weights are set to create a score of 0 if all Propertys of an Item are supplied in the query.
As the number of Items is about 600.000 entries this query takes approximately 4-6 seconds. Is there a better way to accomplish this?

Answer 1

I imagine you're looking for a Python solution, but the Ohm library for Ruby is my favorite of the Redis-based database analogues. Considering the similarities between Python and Ruby and Ohm's exceptional documentation, you might find some inspiration.

Answer 2

EDIT: Used real properties as stated in comments.
I think I did it (once more). I used PHPRedis.
I used sorted sets too, but I inverted your schema: Each zset represents an ingredient, and each recipe id is a member of that zset. So every zset has the same number of members, i.e., every recipe on the application. Every recipe uses or not a ingredient. That defines the score.
Loading is somewhat expensive, but query is done under 3s, for a sample with 12 ingredients and 600.000 recipes. (you've got a lot of them for sure!).
Loading
Pseudo-code:
For every ingredient i on the system
For every recipe j on the system
If recipe j uses the ingredient i Then
score = 1
INCR recipe:j:ing_count //Will help sorting
RPUSH recipe:j:ing_list i //For listing all ingredients in recipe
Else
score = 0
End If
ZADD ing:i score j
End For
End For
Code:
#!/usr/bin/php
<?
### Total of ingredients
define('NUM_OF_ING',12);
### Total of recipes
define('NUM_OF_RECIPES',600000);
$redis = new \Redis();
$redis->connect('localhost');
for ($ing=1; $ing<=NUM_OF_ING; $ing++) {
for ($recipe=1; $recipe<=NUM_OF_RECIPES; $recipe++) {
$score = rand() % 2;
if ($score == 1) {
$redis->incr("recipe:$recipe:ing_count");
$redis->rpush("recipe:$recipe:ing_list", $ing);
}
$redis->zAdd("ing:$ing", $score, $recipe);
}
}
echo "Done.\n";
?>
Querying
Before I paste the PHP code and measured running time, let me do some observations:
Sorting is done based on the number of ingredients used (sum of the zsets in query). If two recipes use all ingredients that are in query, then the tie-break is done by the number of additional ingredients that one recipe has. More ingredients, higher position.
The sum is handled by ZINTERSTORE. The zset with sums is stored in result.
Then a SORT command looks in the count key for each recipe, tailoring the order with this additional constraint.
Code:
#!/usr/bin/php
<?
$redis = new \Redis();
$redis->connect('localhost');
//properties in query
$query = array('ing:2', 'ing:4', 'ing:5');
$weights = array(1, 1, 1);
//intersection
$redis->zInter('result', $query, $weights, 'sum');
//sorting
echo "Result:\n";
var_dump($redis->sort('result', array('by'=>'recipe:*:ing_count', 'sort'=>'desc', 'limit'=>array(0,10))));
echo "End.\n";
?>
Output and running time:
niloct#Impulse-Ubuntu:~$ time ./final2.php
Result:
array(10) {
[0]=>
string(4) "5230"
[1]=>
string(5) "79549"
[2]=>
string(4) "2871"
[3]=>
string(3) "336"
[4]=>
string(6) "109279"
[5]=>
string(4) "5352"
[6]=>
string(5) "16868"
[7]=>
string(3) "690"
[8]=>
string(4) "3174"
[9]=>
string(4) "8795"
}
End.
real 0m2.930s
user 0m0.016s
sys 0m0.004s
niloct#Impulse-Ubuntu:~$ redis-cli lrange recipe:5230:ing_list 0 -1
1) "12"
2) "11"
3) "10"
4) "9"
5) "8"
6) "7"
7) "6"
8) "5"
9) "4"
10) "3"
11) "2"
12) "1"
Hope that helps.
PS: Can you post your performance measures after trying this ?