Why there is no ordered hashmap in Redis? - redis

Redis Data types includes sorted set and other necessary data-structures for key-value storage. But I wonder why it doesn't have any sorted map like Java's TreeMap or C++'s std::map. I think the underlying data-structure would be mostly similar of sorted set as both are supposed to be balanced binary search tree.
There must be some use-cases where we have to store key-value pair in specific order according to key. But current sorted set only serves the purpose of storing key according to score.

There must be some use-cases where we have to store key-value pair in specific order according to key
Since Redis keys are binary strings, I assume that the specific order you mentioned, is lexicographical order (specifically, keys are compared with the memcmp function). In that case, you can easily implement a C++'s std::map with SORTED SET. You can achieve this with 2 steps:
Build a std::set with Redis' Sorted Set
If 2 elements in a SORTED SET have the same score, they're ordered in lexicographical order. So in order to build a std::set, just give all members in a SORTED SET with the same score:
zadd std::set 0 c
zadd std::set 0 a
zadd std::set 0 b
// since all these members have the same score,
// the result is lexicographical ordered:
// a b c
zrange std::set 0 -1
// the following command will fail, since 'c' already exists.
zadd std::set 0 c
Since Redis 2.8, it supports some commands to operate on the lexicographical ranges, so that you can build something similar to std::set::lower_bound, or std::set::upper_bound
// something similar to lower_bound: find all members not less than b
zrangebylex std::set [b +
// something similar to upper_bound: find all members greater than b
zrangebylex std::set (b +
Map each key in the set with a value
Since you already get a std::set, then map the key with a value, you can get a std::map.
set a value_a
set b value_b
set c value_c
Combine these 2 steps together
You can wrap the whole work into a lua script to have a built-in std::map. And use it like this:
redis-cli --eval map.lua map_name , key value

Related

How to get subset of SMEMBERS result? Or should I use SortedSet with the same value for both score & member?

I am new to Redis. For example, if I have the following schema:
INCR id:product
SET product:<id:product> value
SADD color:red <id:product>
(Aside: I am not sure how to express a variable in Redis. I will just use <id:product> as the primary key value. In production, I will use golang client for this job)
To query products which have red color, I can do:
SMEMBERS color:red
But the problem is I just want to display 10 of them in the first page, and then next 10 in the second page and so on. How to let Redis return only part of them by specifying offset and limit arguments?
What do redis experts normally do for this case? Return all IDs even if I just want 10 of them? Is that efficient? What if it has millions of values in the set, but I only want 10?
Edited 1
Incidentally, I use sets instead of lists and sorted sets because I will need to do SINTER jobs for other queries.
For example:
SADD type:foo <id:product>
SINTER color:red type:foo
And then I will have pagination problem again. Because I actually just want to find 10 of the intersection at a time. (eg: if the intersection returns millions of keys, but actually I just want 10 of them at a time for pagination).
Edited 2
Should I use a sorted set instead? I am not sure if this is the expert choice or not. Something like:
ZADD color:red <id:product> <id:product>
ZADD type:foo <id:product> <id:product>
ZRANGE color:red 0 9 // for the first page of red color products
ZINTERSTORE out 2 color:red type:foo AGGREGATE MIN
ZRANGE out 0 9 // for the first page of red color and type foo products
I have no ideas if the above way is suggested or not.
What will happen if multiple clients are creating the same out sorted set?
Is that meaningful to use the same value for both score and member?
Using sorted sets is the standard way to do pagination in Redis.
The documentation of ZINTERSTORE says that: "If destination already exists, it is overwritten."
Therefore, you shouldn't use "out" as the destination key name. You should instead use a unique or sufficiently random key name and then delete it when you're done.
I'm not sure what you mean by "meaningful". It's a fine choice if that's the order you want them to be in.

Cascade deletes in Redis

On my current project I'm implementing autocompletion service on top of Redis, for it I use such approach (this article describes it more widely):
1) for storing dump of the data I have hash in which I put searchable objects as a values, for instance
HSET data 1 "{\"name\":\"Kill Bill\",\"year\":2003}"
HSET data 2 "{\"name\":\"King Kong\",\"year\":2005}"
2) for storing all possible sequences of input characters (that I generate in advance) which could be used in search I use sorted sets, like
ZADD search:index:k 0 1
ZADD search:index:ki 0 1
ZADD search:index:kil 0 1
ZADD search:index:kill 0 1
Where value stored in sorted set (in my example '1') is key for data from hash. So, for searching some data (for example where name started with 'ki') we need to make two steps:
data_keys = REDIS.zrevrange('search:index:ki', 0, -1)
matching_data = REDIS.hmget(data, *data_keys)
The issue I tried to solve - how automatically remove all data from sorted sets related to hash values when I removed it? In relational databases I can use cascade deletion for such cases, but how can I handle it in Redis?
Your design appears awkward to me, I'm unsure what you're actually trying to do with Redis and perhaps that could be the topic of another question.
That said, to address your question, Redis does offer a "cascading delete"-like behavior. Instead, if you're deleting hash "1", iterate the prefix and ZREM it from the relevant sorted sets.
Note: do not use a Lua script for this task, as it will generate key names (i.e. sorted sets by prefix) and that is against the recommendations (will not work on a cluster)

Expire geospatial items in Redis

There are proposals for sorted set item expiration in Redis (see https://groups.google.com/d/msg/redis-db/rXXMCLNkNSs/Bcbd5Ae12qQJ and https://quickleft.com/blog/how-to-create-and-expire-list-items-in-redis/), I tried the worker approach to expire geospatial indexes with ZREMRANGEBYSCORE and ZREMRANGEBYRANK commands unsuccessfully (nothing removed).
I succeded using ZREMRANGEBYLEX.
Is there a way to work with geospatial items score other than Strings?
Update:
For example, if time to live(ttl) of an item is 30sec, I add it as:
geoadd 1 -8.616021 41.154503 30
Now, suppose worker executes after 40sec, I was expecting that
zremrangebyscore 1 0 40
would do the job, but it does not,
ZREMRANGEBYLEX 1 [0 [40
does it. Why is this behavior? That means the score of a geospatial item supports only lexicographical operations?
Sorted Sets have elements (strings), and every element has a score (floating-point). Geosets use the score to encode a coordinate.
Redis doesn't expire members in a Sorted Set (or a Geoset). You have to remove them yourself if that is required.
In your case, you'll need to keep two Sorted Sets - one as your GeoSet and one for managing TTLs as scores.
For example, assuming your member is called 'foo', to add it:
ZADD ttls 30 foo
ZADD elems -8.616021 41.154503 foo
To manually expire, first find the members with a call to ZRANGEBYSCORE ttls, and then remove them from both Sets.
Tip: it is preferable to use a timestamp as score instead of seconds.

Redis - Sorted set, find item by property value

In redis I store objects in a sorted set.
In my solution, it's important to be able to run a ranged query by dates, so I store the items with the score being the timestamp of each items, for example:
# Score Value
0 1443476076 {"Id":"92","Ref":"7ADT","DTime":1443476076,"ATime":1443901554,"ExTime":0,"SPName":"7ADT33CFSAU6","StPName":"7ADT33CFSAU6"}
1 1443482969 {"Id":"11","Ref":"DAJT","DTime":1443482969,"ATime":1443901326,"ExTime":0,"SPName":"DAJTJTT4T02O","StPName":"DAJTJTT4T02O"}
However, in other situations I need to find a single item in the set based on it's ID.
I know I can't just query this data structure as if it were a nosql db, but I tried using ZSCAN, which didn't work.
ZSCAN MySet 0 MATCH Id:92 count 1
It returns; "empty list or set"
Maybe I need to serialize different?
I have serialized using Json.Net.
How, if possible, can I achieve this; using dates as score and still be able to lookup an item by it's ID?
Many thanks,
Lars
Edit:
Assume it's not possible, but any thoughts or inputs are welcome:
Ref: http://openmymind.net/2011/11/8/Redis-Zero-To-Master-In-30-Minutes-Part-1/
In Redis, data can only be queried by its key. Even if we use a hash,
we can't say get me the keys wherever the field race is equal to
sayan.
Edit 2:
I tried to do:
ZSCAN MySet 0 MATCH *87*
127.0.0.1:6379> ZSCAN MySet 0 MATCH *87*
1) "192"
2) 1) "{\"Id\":\"64\",\"Ref\":\"XQH4\",\"DTime\":1443837798,\"ATime\":1444187707,\"ExTime\":0,\"SPName\":\"XQH4BPGW47FM\",\"StPName\":\"XQH4BPGW47FM\"}"
2) "1443837798"
3) "{\"Id\":\"87\",\"Ref\":\"5CY6\",\"DTime\":1443519199,\"ATime\":1444172326,\"ExTime\":0,\"SPName\":\"5CY6DHP23RXB\",\"StPName\":\"5CY6DHP23RXB\"}"
4) "1443519199"
And it finds the desired item, but it also finds another one with an occurance of 87 in the property ATime. Having more unique, longer IDs might work this way and I would have to filter the results in code to find the one with the exact value in its property.
Still open for suggestions.
I think it's very simple.
Solution 1(Inferior, not recommended)
Your way of ZSCAN MySet 0 MATCH Id:92 count 1 didn't work out because the stored string is "{\"Id\":\"92\"... not "{\"Id:92\".... The string has been changed into another format. So try to use MATCH Id\":\"64 or something like that to match the json serialized data in redis. I'm not familiar with json.net, so the actual string leaves for you to discover.
By the way, I have to ask you did ZSCAN MySet 0 MATCH Id:92 count 1 return a cursor? I suspect you used ZSCAN in a wrong way.
Solution 2(Better, strongly recommended)
ZSCAN is good when your sorted set is not large and you know how to save network roundtrip time by Redis' Lua transaction. This still make "look up by ID" operation O(n). Therefore, a better solution is to change you data model in the following way:
change sorted set
from
# Score Value
0 1443476076 {"Id":"92","Ref":"7ADT","DTime":1443476076,"ATime":1443901554,"ExTime":0,"SPName":"7ADT33CFSAU6","StPName":"7ADT33CFSAU6"}
1 1443482969 {"Id":"11","Ref":"DAJT","DTime":1443482969,"ATime":1443901326,"ExTime":0,"SPName":"DAJTJTT4T02O","StPName":"DAJTJTT4T02O"}
to
# Score Value
0 1443476076 Id:92
1 1443482969 Id:11
Move the rest detailed data in another set of hashes type keys:
# Key field-value field-value ...
0 Id:92 Ref-7ADT DTime-1443476076 ...
1 Id:11 Ref-7ADT DTime-1443476076 ...
Then, you locate by id by doing hgetall id:92. As to ranged query by date, you need do ZRANGEBYSCORE sortedset mindate maxdate then hgetall every id one by one. You'd better use lua to wrap these commands in one and it will still be super fast!
Data in NoSql database need to be organized in a redundant way like above. This may make some usual operation involve more than one commands and roundtrip, but it can be tackled by redis's lua feature. I strongly recommend the lua feature of redis, cause it wrap commands into one network roundtrip, which are all executed on the redis-server side and is atomic and super fast!
Reply if there's anything you don't know

Redis Bit Select All Keys With Offset Value of 1

If I have a series of keys representing bitmaps in Redis, how can I get a list of all keys whose n bit value is equal to 1. For example, I have the following bitmaps:
keyname:20140418:item1: 000111...0010
keyname:20140418:item2: 000101...1010
keyname:20140418:item3: 100011...0010
I want to get the list of all items whose first value is 0 which would result in:
keyname:20140418:item1
keyname:20140418:item2
In Redis, always prepare your data in such a way that retrieval is easy and most of all scalable.
When storing the data, accompany the SET with a ZADD zerostartingbitmaps [unixtimestamp] [linkedkey]
To fix all your data and fill the Sorted Set, use SCAN and bitwise operations.
Hope this helps, TW