I have a data structure that looks like the following
data[price] = amount
so if I entered this in redis as
hset mydata 300 500
I can increment the key like so
hincrby mydata 300 500
I am using the Python redis Package. From Python I can get the minimum keys by like so
min(redis.hgetall(keyname))
But this is an operation in order of O(N) and the official documentation cautions against using this in production. The other approach I have considered is sorting the keys and getting the minimum value. Something that looks like:
redis.sort(keyName)[0] #pseudocode
But I am not sure how to do this. I'd appreciate any ideas to either sort numeric keys in hset or get minimum value of the keys
Related
I have the n number of keys in my redis server with some data. Now i want to check what all keys got created in last two months. How to check this. Is there any way to sort all cache keys in redis-cli by creation time or anything?
Redis doesn't store this information. You need to do this explicitly. There are many ways you can achieve this. Some of them are:
SET time or date or datetime string when setting key
ex:
SET key1 data
SET key1:date "12-JULY-2018"
Make data Object type - Add an explicit key of created at and then store it to Redis. Then sort it in your own application.
Create Sets/Lists of each hour/day/month and keep pushing all the keys to those lists. You can then retrieve keys for each hour/day/month. Now get data using these keys.
ex:
SET key1 data // At this point date is "12-JULY-2018"
SADD "JULY-SET" key1
Now you can get all keys of JULY by doing this:
SMEMBERS "JULY-SET"
On my current project I'm implementing autocompletion service on top of Redis, for it I use such approach (this article describes it more widely):
1) for storing dump of the data I have hash in which I put searchable objects as a values, for instance
HSET data 1 "{\"name\":\"Kill Bill\",\"year\":2003}"
HSET data 2 "{\"name\":\"King Kong\",\"year\":2005}"
2) for storing all possible sequences of input characters (that I generate in advance) which could be used in search I use sorted sets, like
ZADD search:index:k 0 1
ZADD search:index:ki 0 1
ZADD search:index:kil 0 1
ZADD search:index:kill 0 1
Where value stored in sorted set (in my example '1') is key for data from hash. So, for searching some data (for example where name started with 'ki') we need to make two steps:
data_keys = REDIS.zrevrange('search:index:ki', 0, -1)
matching_data = REDIS.hmget(data, *data_keys)
The issue I tried to solve - how automatically remove all data from sorted sets related to hash values when I removed it? In relational databases I can use cascade deletion for such cases, but how can I handle it in Redis?
Your design appears awkward to me, I'm unsure what you're actually trying to do with Redis and perhaps that could be the topic of another question.
That said, to address your question, Redis does offer a "cascading delete"-like behavior. Instead, if you're deleting hash "1", iterate the prefix and ZREM it from the relevant sorted sets.
Note: do not use a Lua script for this task, as it will generate key names (i.e. sorted sets by prefix) and that is against the recommendations (will not work on a cluster)
I have a redis table where values are stored like
key: "aa - 1" value {"price": "10"}
key: "aa - 2" value {"price": "8"}
key: "aa - 4" value {"price": "15"}
I am executing keys *aa* to get the list of keys. Is there a way to get this list ordered by the price as well?
thanks.
Redis is not very complex. It is for sure not document based, and doesn't have a json type.
From your comment I understand that you saved json within strings. There's no way to sort that.
Here is an overview of redis types: if you need sorting, you should decide for sorted sets, with price as score.
First of all this is not a table actually. You should consider your data as a set of objects which you can identify by key: you put your object with key1 and can get it back quickly using key. But your object is simply string or even byte[].
To store data in key-value storage many ways can be used. You can put whole object into the key and always get back whole object. No way to sort anything.
You may also to do some kind of vertical split on property level, and store each property or sub-object separately. Something like:
key: "aa - 1:price" value "10"
key: "aa - 1:name" value "CoolName"
Obvious drawback - it is hard to maintain in consistent way, you need many requests to get whole objects. Though GET "aa - 1:*" can be used as well
But as positive thing here, you may store different properties using different redis data structures.
For example you can store price using ZADD. Something like that:
ZADD pricesSet 10 "aa - 1"
ZADD pricesSet 4 "aa - 2"
Then you can use ZRANGEBYSCORE to take data from this key
ZRANGEBYSCORE priceSet 5 15
Means to get keys of all items with price from 5 to 15. You will get "aa - 1" and can then get it separately.
And YES, output of ZRANGEBYSCORE is sorted by price.
In redis I store objects in a sorted set.
In my solution, it's important to be able to run a ranged query by dates, so I store the items with the score being the timestamp of each items, for example:
# Score Value
0 1443476076 {"Id":"92","Ref":"7ADT","DTime":1443476076,"ATime":1443901554,"ExTime":0,"SPName":"7ADT33CFSAU6","StPName":"7ADT33CFSAU6"}
1 1443482969 {"Id":"11","Ref":"DAJT","DTime":1443482969,"ATime":1443901326,"ExTime":0,"SPName":"DAJTJTT4T02O","StPName":"DAJTJTT4T02O"}
However, in other situations I need to find a single item in the set based on it's ID.
I know I can't just query this data structure as if it were a nosql db, but I tried using ZSCAN, which didn't work.
ZSCAN MySet 0 MATCH Id:92 count 1
It returns; "empty list or set"
Maybe I need to serialize different?
I have serialized using Json.Net.
How, if possible, can I achieve this; using dates as score and still be able to lookup an item by it's ID?
Many thanks,
Lars
Edit:
Assume it's not possible, but any thoughts or inputs are welcome:
Ref: http://openmymind.net/2011/11/8/Redis-Zero-To-Master-In-30-Minutes-Part-1/
In Redis, data can only be queried by its key. Even if we use a hash,
we can't say get me the keys wherever the field race is equal to
sayan.
Edit 2:
I tried to do:
ZSCAN MySet 0 MATCH *87*
127.0.0.1:6379> ZSCAN MySet 0 MATCH *87*
1) "192"
2) 1) "{\"Id\":\"64\",\"Ref\":\"XQH4\",\"DTime\":1443837798,\"ATime\":1444187707,\"ExTime\":0,\"SPName\":\"XQH4BPGW47FM\",\"StPName\":\"XQH4BPGW47FM\"}"
2) "1443837798"
3) "{\"Id\":\"87\",\"Ref\":\"5CY6\",\"DTime\":1443519199,\"ATime\":1444172326,\"ExTime\":0,\"SPName\":\"5CY6DHP23RXB\",\"StPName\":\"5CY6DHP23RXB\"}"
4) "1443519199"
And it finds the desired item, but it also finds another one with an occurance of 87 in the property ATime. Having more unique, longer IDs might work this way and I would have to filter the results in code to find the one with the exact value in its property.
Still open for suggestions.
I think it's very simple.
Solution 1(Inferior, not recommended)
Your way of ZSCAN MySet 0 MATCH Id:92 count 1 didn't work out because the stored string is "{\"Id\":\"92\"... not "{\"Id:92\".... The string has been changed into another format. So try to use MATCH Id\":\"64 or something like that to match the json serialized data in redis. I'm not familiar with json.net, so the actual string leaves for you to discover.
By the way, I have to ask you did ZSCAN MySet 0 MATCH Id:92 count 1 return a cursor? I suspect you used ZSCAN in a wrong way.
Solution 2(Better, strongly recommended)
ZSCAN is good when your sorted set is not large and you know how to save network roundtrip time by Redis' Lua transaction. This still make "look up by ID" operation O(n). Therefore, a better solution is to change you data model in the following way:
change sorted set
from
# Score Value
0 1443476076 {"Id":"92","Ref":"7ADT","DTime":1443476076,"ATime":1443901554,"ExTime":0,"SPName":"7ADT33CFSAU6","StPName":"7ADT33CFSAU6"}
1 1443482969 {"Id":"11","Ref":"DAJT","DTime":1443482969,"ATime":1443901326,"ExTime":0,"SPName":"DAJTJTT4T02O","StPName":"DAJTJTT4T02O"}
to
# Score Value
0 1443476076 Id:92
1 1443482969 Id:11
Move the rest detailed data in another set of hashes type keys:
# Key field-value field-value ...
0 Id:92 Ref-7ADT DTime-1443476076 ...
1 Id:11 Ref-7ADT DTime-1443476076 ...
Then, you locate by id by doing hgetall id:92. As to ranged query by date, you need do ZRANGEBYSCORE sortedset mindate maxdate then hgetall every id one by one. You'd better use lua to wrap these commands in one and it will still be super fast!
Data in NoSql database need to be organized in a redundant way like above. This may make some usual operation involve more than one commands and roundtrip, but it can be tackled by redis's lua feature. I strongly recommend the lua feature of redis, cause it wrap commands into one network roundtrip, which are all executed on the redis-server side and is atomic and super fast!
Reply if there's anything you don't know
If I have a series of keys representing bitmaps in Redis, how can I get a list of all keys whose n bit value is equal to 1. For example, I have the following bitmaps:
keyname:20140418:item1: 000111...0010
keyname:20140418:item2: 000101...1010
keyname:20140418:item3: 100011...0010
I want to get the list of all items whose first value is 0 which would result in:
keyname:20140418:item1
keyname:20140418:item2
In Redis, always prepare your data in such a way that retrieval is easy and most of all scalable.
When storing the data, accompany the SET with a ZADD zerostartingbitmaps [unixtimestamp] [linkedkey]
To fix all your data and fill the Sorted Set, use SCAN and bitwise operations.
Hope this helps, TW