Cursor Pagination with RedisJSON - redis

I am using Redis to store JSON data with a model that has Id and Name fields. My goal is to implement cursor pagination over this data using the Name as the cursor. I realize this isn't unique and I'll need to do some additional work to resolve the issues presented by that using that particular field as the cursor. However I am struggling to find a way to even begin to implement a basic cursor pagination with Redis commands. Redis has options for various string operations using the "Search" module but these are limited to search within a given string, nothing with comparisons. Am I barking up the wrong tree here and cursor pagination is not possible? Or is there something fundamental I am missing in my design?

Redis has options for various string operations using the "Search" module but these are limited to search within a given string, nothing with comparisons.
Can you please give an example for a required comparison not within a given string?
The Search query can do some comparisons.
For Cursor, FT.CURSOR can be used, for example:
127.0.0.1:6379> JSON.SET key1 $ '{"Id": 100, "Name": "foo", "data": "val1"}'
OK
127.0.0.1:6379> JSON.SET key2 $ '{"Id": 200, "Name": "bar", "data": "val2"}'
OK
127.0.0.1:6379> JSON.SET key3 $ '{"Id": 300, "Name": "foo", "data": "val3"}'
OK
127.0.0.1:6379> FT.CREATE idx ON JSON SCHEMA '$.Id' as id NUMERIC '$.Name' as name TAG
OK
127.0.0.1:6379> FT.AGGREGATE idx '#name:{foo}' LOAD 2 #id #name WITHCURSOR COUNT 1
1) 1) (integer) 1
2) 1) "id"
2) "300"
3) "name"
4) "foo"
2) (integer) 17284697
127.0.0.1:6379> FT.CURSOR READ idx 17284697 COUNT 1
1) 1) (integer) 1
2) 1) "id"
2) "100"
3) "name"
4) "foo"
2) (integer) 17284697
127.0.0.1:6379> FT.CURSOR READ idx 17284697 COUNT 1
1) 1) (integer) 0
2) (integer) 0
127.0.0.1:6379>

Related

Redis - Sort and filter hash store using string attribute

I have a redis hash store that looks like Item:<id>, with attribute name. I want to filter the hash store by a prefix for name attribute.
What I'm trying to do is store the name (lowercased) in a separate Z-set called Item:::name while setting the score to 0. By doing this, I'm successfully able to get the desired result using ZRANGEBYLEX however I'm unable to map the results back to the original Items. How should I go about implementing something like this?
I've seen multiple autocomplete examples for Redis which require the same functionality but without linking the words back to an actual Item (hash in this case)
In sorted sets the member can't be duplicated, it has to be unique. So different users with the same name will cause problem.
My suggestion requires application layer coding to parse response array and executing hash commands (it will be like secondary indexes);
127.0.0.1:6379> HSET user:1 name jack
(integer) 1
127.0.0.1:6379> HSET user:2 name john
(integer) 1
127.0.0.1:6379> HSET user:3 name keanu
(integer) 1
127.0.0.1:6379> HSET user:4 name jack
(integer) 1
127.0.0.1:6379> ZADD item:names 0 jack::user:1 0 john::user:2 0 keanu::user:3 0 jack::user:4
(integer) 4
127.0.0.1:6379> ZRANGE item:names 0 -1 WITHSCORES
1) "jack::user:1"
2) "0"
3) "jack::user:4"
4) "0"
5) "john::user:2"
6) "0"
7) "keanu::user:3"
8) "0"
127.0.0.1:6379> ZRANGEBYLEX item:names [jack [jo
1) "jack::user:1"
2) "jack::user:4"
At the end you will have name::hash-key formatted array elements. At application layer if you separate each element to two substrings by using ::(any other string such as !!! or || etc) you will have user:1 and user:4.
127.0.0.1:6379> HGETALL user:1
1) "name"
2) "jack"
127.0.0.1:6379> HGETALL user:4
1) "name"
2) "jack"
127.0.0.1:6379>

Can we add Value in Redis List with Expiration Time in C#?

I am using Redis queue and adding the data using ListLeftPush and reading data using ListRightPop. It works fine I am able to get the data. But what if data has not popped out? Can we delete old data? or Can we add Value in Redis List with Expiration Time?
How to add time limit for each value while using ListLeftPush command in C#?
It is not possible to add expire time for individual value for the sake of keeping redis simple and fast.
you can only add expire time for individual keys i.e in ur case it is for whole list.
No redis doesn't support that. Expiration is available only for the top level keys. The closes data type/solution for your case would be sorted sets.
You put your expiration time(timestamp) as score while adding to sorted set(ZADD)
Instead of LPOP you use ZPOPMAX to get "to be last expired" element.
Periodically you may use ZREMRANGEBYSCORE to remove expired elements.
For the demonstration i used smaller numbers as expiration dates.
127.0.0.1:6379> ZADD myset 15 "a"
(integer) 0
127.0.0.1:6379> ZADD myset 25 "b"
(integer) 0
127.0.0.1:6379> ZADD myset 35 "c" 45 "d" 55 "e"
(integer) 0
127.0.0.1:6379> ZRANGE myset 0 -1 WITHSCORES
1) "a"
2) "15"
3) "b"
4) "25"
5) "c"
6) "35"
7) "d"
8) "45"
9) "e"
10) "55"
127.0.0.1:6379> ZPOPMAX myset
1) "e"
2) "55"
127.0.0.1:6379> ZREMRANGEBYSCORE myset -1 15
(integer) 1
127.0.0.1:6379> ZRANGE myset 0 -1
1) "b"
2) "c"
3) "d"
127.0.0.1:6379>

Unique value in redis list/set

I want to make a list of existing products in redis but I want to check if the product name already exists first (duplicate checking).
My list currently accepts duplicates, so: Can anyone help me to show how to add unique value in list?
Instead of using a list, use a set. Sets are containers for unique objects. Each object can only appear once in a set. Take a look at the set-related commands here: http://redis.io/commands/#set.
And an example using redis-cli (we attempt to add "Product One" twice, but it only appears once in the list of products):
$ redis-cli
127.0.0.1:6379> sadd products "Product One"
(integer) 1
127.0.0.1:6379> sadd products "Product Two"
(integer) 1
127.0.0.1:6379> sadd products "Product Three"
(integer) 1
127.0.0.1:6379> sadd products "Product One"
(integer) 0
127.0.0.1:6379> smembers products
1) "Product Three"
2) "Product One"
3) "Product Two"
127.0.0.1:6379>
Why not just call Redis.lrem before? So if it finds any occurences of the item, removes them, otherwise will do nothing. Something like this:
def push_item_to_the_list(LIST_KEY, item)
Redis.lrem(LIST_KEY, 0, item)
Redis.lpush(LIST_KEY, item)
end
This is my (reckless) solution to keep Redis List unique.
(implementation in Ruby)
def push_item_to_the_list(LIST_KEY, item)
insert_status = Redis.linsert(LIST_KEY, 'before', item, item)
if insert_status == -1
Redis.lpush(LIST_KEY, item)
else
Redis.lrem(LIST_KEY, 1, item)
end
end
Each time when you want to push or insert item to your list, check if LINSERT command will be able to put this item just after the very same item (this is the only way I know to check if the item is already in the redis list or not).
If LINSERT will return status -1, it means that it was not able to find item in your list - everything is ok (you can push it or insert it now).
If LINSERT will return other value (size of the list in other case) - it means that it was able to find item already and it was able to insert another item, just after the previous one. It means that you have (at least one) duplication of your item. You can delete one of them now.
In case you need to maintain the order and the uniqueness you can use a sorted set
127.0.0.1:6379> zadd products 1 "Product One"
(integer) 1
127.0.0.1:6379> zadd products 2 "Product Two"
(integer) 1
127.0.0.1:6379> zadd products 3 "Product Tree"
(integer) 1
127.0.0.1:6379> zadd products 4 "Product Four"
(integer) 1
127.0.0.1:6379> zrange products 0 -1
1) "Product One"
2) "Product Two"
3) "Product Tree"
4) "Product Four"
I propose an option that creates a unique list, the elements of which do not lose their position. This option works much faster than lpush, lrem, rpop. At the same time, it supports functional of sadd, spop.
PHP code example:
public function addJobs(array $jobs): int
{
foreach ($jobs as $key => $job) {
$hash = hash('md4', $job);
if (0 === $this->client->hsetnx('QUEUE-HASHES', $hash, 1)) {
unset($jobs[$key]);
}
}
return $this->client->rpush('QUEUE', $jobs);
}
public function popJobs(int $count): array
{
if ($count < 1) {
throw new \InvalidArgumentException('Jobs count must be greater than zero');
}
$index = $count - 1;
$jobs = $this->client->lrange('QUEUE', 0, $index);
if (\count($jobs)) {
$this->client->ltrim('QUEUE', $count, -1);
$hashes = [];
foreach ($jobs as $job) {
$hashes[] = hash('md4', $job);
}
$this->client->hdel('QUEUE-HASHES', $hashes);
}
return $jobs;
}
redis 5 ZPOPMIN https://redis.io/commands/zpopmin
Removes and returns up to count members with the lowest scores in the sorted set stored at key.
When left unspecified, the default value for count is 1. Specifying a count value that is higher than the sorted set's cardinality will not produce an error. When returning multiple elements, the one with the lowest score will be the first, followed by the elements with greater scores.

How do I know the data type of the value of a given key?

I can't seem to find useful information about Redis commands. I want to know the data type of the value of a given key. For instance to list all the keys of my database I run the following command:
keys *
In my setup, I get the following result:
1) "username:testuser:uid"
2) "uid:1:first"
3) "uid:1:email"
4) "uid:1:hash"
5) "global:next_uid"
6) "members:email"
7) "uid:1:username"
8) "uid:1:last"
9) "uid:1:salt"
10) "uid:1:access"
11) "uid:1:company"
12) "email:testuser#gmail.com:uid"
13) "uid:1:phone_number"
How do I know what data type the key members:email contains? I tried to run get members:email but and I get the error (error) ERR Operation against a key holding the wrong kind of value
Any thoughts?
You could use the type command:
http://redis.io/commands/type
See below from docs:
redis> SET key1 "value"
"OK"
redis> LPUSH key2 "value"
(integer) 1
redis> SADD key3 "value"
(integer) 1
redis> TYPE key1
"string"
redis> TYPE key2
"list"
redis> TYPE key3
"set"
redis>

Is there MGET analog for Redis hashes?

I'm planning to start using hashes insead of regular keys. But I can't find any information about multi get for hash-keys in Redis wiki. Is this kind of command is supported by Redis?
Thank you.
You can query hashes or any keys in pipeline, i.e. in one request to your redis instance. Actual implementation depends on your client, but with redis-py it'd look like this:
pipe = conn.pipeline()
pipe.hgetall('foo')
pipe.hgetall('bar')
pipe.hgetall('zar')
hash1, hash2, hash3 = pipe.execute()
Client will issue one request with 3 commands. This is the same technique that is used to add multiple values to a set at once.
Read more at http://redis.io/topics/pipelining
No MHGETALL but you can Lua it:
local r = {}
for _, v in pairs(KEYS) do
r[#r+1] = redis.call('HGETALL', v)
end
return r
If SORT let you use multiple GETs with the -> syntax, and all your hashes had the same fields, you could get them in a bulk reply by putting their names into a set and sorting that.
SORT names_of_hashes GET *->field1 *->field2 *->field3 *->etc
But it doesn't look like you can do that with the hash access. Plus you'd have to turn the return list back into hashes yourself.
UPDATE: Redis seems to let you fetch multiple fields if you name your hashes nicely:
redis> hset hash:1 name fish
(integer) 1
redis> hset hash:2 name donkey
(integer) 1
redis> hset hash:3 name horse
(integer) 1
redis> hset hash:1 type fish
(integer) 1
redis> hset hash:2 type mammal
(integer) 1
redis> hset hash:3 type mammal
(integer) 1
redis> sadd animals 1
(integer) 1
redis> sadd animals 2
(integer) 1
redis> sadd animals 3
(integer) 1
redis> sort animals get # get hash:*->name get hash:*->type
1. "1"
2. "fish"
3. "fish"
4. "2"
5. "donkey"
6. "mammal"
7. "3"
8. "horse"
9. "mammal"
There is no command to do it on one shot, but there is a way to do it "nicely", using a list (or sorted set) where you would store you hashKeys, and then retrieve them as bulk using multi.
In PHP:
$redis->zAdd("myHashzSet", 1, "myHashKey:1");
$redis->zAdd("myHashzSet", 2, "myHashKey:2");
$redis->zAdd("myHashzSet", 3, "myHashKey:3");
$members = $redis->zRange("myHashzSet", 0, -1);
$redis->multi();
foreach($members as $hashKey) {
$redis->hGetAll($hashKey);
}
$results = $redis->exec();
I recommand using a sorted set, where you use the score as an ID for your hash, it allows to take advantages of all score based command.
Redis has a HMGET command, which returns the values of several hash keys with one command.