I am working on small application where I am using redis to hold my intermediate data. After inserting data, I need to reload my data in same order in which i have inserted.
I am using keys method to get all keys but the order of returned keys is not same as they were inserted.
You have to maintain order yourself, by keeping a separate list for inserted keys. So, instead of
SET foo, bar
you may do something like this:
SET foo, bar
RPUSH insert_order, foo
Then you can do
LRANGE insert_order, 0, 100
to get first 100 set fields.
If you want to track actual insertion (and not updates), you can use SETNX, for example. Also, you can use a sorted set instead of a list (as mentioned by #Leonid) Additionally, you can wrap the whole thing in Lua, so that the bookkeeping is hidden from the client code.
For indexing URL and getting the list by inserted order, you should use sorted set:
zadd <your_url_list_key> <inserted_time> <url>
Detail data for a single url should be stored in a different place. For example, use hash:
hset <your_url_data_key> <url> <url_data>
It's better if you don't store detail data on redis, so instead of using redis hash, you should save url detail data on mysql.
You can also md5(url) before indexing to reduce the size (then the full url value will be stored in url_data).
In my project, sorted set still works ok with about 3mil records (read & write frequently). But you should watch the hash size often, it will grow really fast.
Related
I'm currently studying Redis, and have the following case:
So what I have is a sorted set by google place id, for which all posts are sorted from recent to older.
The first page that is requested fetches posts < current timestamp.
When a cursor is sent to the backend, this cursor is a simple timestamp that indicates from where to fetch the next posts from the ZSET.
The query to retrieve posts by place id would be:
ZREVRANGEBYSCORE <gplaceId> <cur_timestamp> -INF WITHSCORES LIMIT <offset:timestamp as from where to fetch> <count:number of posts>
My question is what is the recommended way to clean up members of the ZSET.
Since I want to use Redis as a cache, I would prefer to limit the number of posts per place for example up until 50. When places get new posts when already 50 posts have been added to the set, I want to drop the last post from the set.
Of course I realise that I could make a manual check on every insert and perform operations as such, but I wonder if Redis has a recommended way of performing this cleanup. Alternatively I could create a scheduler for this, but I would prefer not having to do this.
Unfortunately Redis sorted set do not come with an out of the box feature for this. If sorted sets allowed a max size attribute with a configurable eviction strategy - you could have avoided some extra work.
See this related question:
How to specify Redis Sorted Set a fixed size?
In absence of such a feature, the two approaches you mentioned are right.
You can replace inserts with a transaction : insert, check size, delete if > 50
A thread that checks size of the set and trims it periodically
On my current project I'm implementing autocompletion service on top of Redis, for it I use such approach (this article describes it more widely):
1) for storing dump of the data I have hash in which I put searchable objects as a values, for instance
HSET data 1 "{\"name\":\"Kill Bill\",\"year\":2003}"
HSET data 2 "{\"name\":\"King Kong\",\"year\":2005}"
2) for storing all possible sequences of input characters (that I generate in advance) which could be used in search I use sorted sets, like
ZADD search:index:k 0 1
ZADD search:index:ki 0 1
ZADD search:index:kil 0 1
ZADD search:index:kill 0 1
Where value stored in sorted set (in my example '1') is key for data from hash. So, for searching some data (for example where name started with 'ki') we need to make two steps:
data_keys = REDIS.zrevrange('search:index:ki', 0, -1)
matching_data = REDIS.hmget(data, *data_keys)
The issue I tried to solve - how automatically remove all data from sorted sets related to hash values when I removed it? In relational databases I can use cascade deletion for such cases, but how can I handle it in Redis?
Your design appears awkward to me, I'm unsure what you're actually trying to do with Redis and perhaps that could be the topic of another question.
That said, to address your question, Redis does offer a "cascading delete"-like behavior. Instead, if you're deleting hash "1", iterate the prefix and ZREM it from the relevant sorted sets.
Note: do not use a Lua script for this task, as it will generate key names (i.e. sorted sets by prefix) and that is against the recommendations (will not work on a cluster)
In a Redis database I have a number of hashes corresponding to "story" objects.
I have an ordered set stories containing all keys of the above (the stories) enabling convenient retrieval of stories.
I now want to store arbitrary emoticons (ie. the Unicode characters corresponding to "smiley face" etc) with stories as "user emotions" corresponding to the emotion the story made the user feel.
I am thinking of:
creating new hashes called emotions containing single emoticons (one per emotion expressed)
creating a hash called story-emotions that enables efficient retrieval of and counting of all the emotions associated with a story
creating another new hash called user-story-emotions mapping user IDs to items in the story-emotion hash.
Typical queries will be:
retrieve all the emotions for a story for the current user
retrieve the count of each kind of emotion for the 50 latest stories
Does this sound like a sensible approach?
Very sensible, but I think I can help make it even more so.
To store the emoticons dictionary, use two Hashes. The first, lets call it emoticon-id should have a field for each emoticon expressed. The field name is the actual Unicode sequence and the value is a unique integer value starting from 0, and increasing for each new emoticon added.
Another Hash, id-emoticon, should be put in place to do the reverse mapping, i.e. from field names that are ids to actual Unicode values.
This gives you O(1) lookups for emoticons, and you should also consider caching this in your app.
To store the user-story-emotions data, look into Redis' Bitmaps. Tersely, use the emoticon id as index to toggle the presence/lack of it by that user towards that story.
Note that in order to keep things compact, you'll want popular emotions to have low ids so your bitmaps remain a small as possible.
To store the aggregative story-emotions, the Sorted Set would be a better option. Elements can be either id or actual unicode, and the score should be the current count. This will allow you to fetch the top emoticons (ZREVRANGEBYSCORE) and/or page similarly to how you're doing with the recent 50 stories (I assume you're using the stories Sorted Set for that).
Lastly, when serving the second query, use pipelining or Lua scripting when fetching the bulk of 50 story-emotion counter values in order get more throughput and better concurrency.
I am trying to insert time based records with multiple fields on the values (with TTL enabled).
For the multiple fields the best way to do it via Redis is using HSET:
HSET user:32 name "johns" timecreated "3333311232" address "somewhere"
I also try to read those values via time range:
for example return all history records (for example user 32) which was inserted in the last day:
so the best for that would be storing via ZADD using scores(this time I am losing the hash-map structure for easy retrieval):
ZADD user:32 3333311232 "name=johns,timecreated=3333311232,address=somewhere"
On the top of the things I want to add TTL for each record
Any idea how I could optimize my design?
I could split into two but that will requires two queries when reading:
ZADD user:32 3333311232 "user:32:3333311232"
HMSET user:32:3333311232 name “johns” timecreated “3333311232” address="somewhere"
than to retrieve ill need:
//some range
ZRANGEBYSCORE user:32 3333311232 333331123
result: 1389772850
now to get all information: HGETALL user:32:1389772850
What do you think?
Thank you,
ray.
The two methods you describe are the two common approaches. If you store the entire object in the ZSET, you would typically store it as a JSON string. If you don't need "random" access to the object, that's a valid approach.
I usually go for the other approach; a ZSET combined with hashes. the two queries are not a big deal. You could even abstract it away with a Lua script; see EVAL.
Regarding the TTL, while you cannot expire individual ZSET values, you could expire the hash, and use keyspace notifications to listen for the expired event, and remove the corresponding value from the ZSET.
Let me know if you need some more specifics.
Is it possible to iterate over all stored redis keys sorted by their value?
stored values:
SET foo:a 1
SET foo:b 20
SET foo:c 5
desired output:
SCAN 0 MATCH foo_* <some-sort-magic>
foo:b
foo:c
foo:a
I cannot store foo:a, foo:b, foo:c in a sorted set, because the values change frequently.
And it would be even better if it would work for hashs too:
stored hashs:
HSET foo:a bar 1
HSET foo:b bar 20
HSET foo:c bar 5
desired output:
SCAN 0 MATCH foo_* <some-sort-magic-for-the-'bar'-field>
foo:b
foo:c
foo:a
Any suggestions except getting all values and ordering in application?
Thx.
Is it possible to iterate over all stored redis keys sorted by their value?
Even if that's possible, you really don't want to do that. A full scan of the keyspace takes, storing the response takes RAM and sorting takes more time. Do you really want to keep your database occupied with that?
EDIT: If you do want to keep your database occupied with that, Redis doesn't have that out of the box. However, there are at least two additional ways to do that without having the application perform the filtering.
The first approach is to use a Lua script that performs a SCAN, gets the values and sorts them before returning the reply.
The second approach is to use Redis modules, and specifically RediSearch, to extend Redis' functionality.
I cannot store foo:a, foo:b, foo:c in a sorted set, because the values change frequently.
There's no such thing as changing too frequently - with Redis everything is done in RAM so writes are just as fast as... well, anything else.
Any suggestions except getting all values and ordering in application?
Use a Sorted Set.