We are storing path's to data in redis as following:
KEY: `/pathOfUniqueAsset/v11/`
VALUE: `/disk1/pathOfUniqueAsset/path/v/11/`.
As you can see, the v which stands for version, will grow over time. I was wondering if there is a way to store flavors/versions of a key/value pair?
You can use a Hash instead of a String, as the key's value type. In the Hash, you can have a field per version/flavor and have the value as the associated path.
E.g.:
HSET /pathOfUniqueAsset v11 /disk1/pathOfUniqueAsset/path/v/11/
What are you trying to achieve? Do you need to keep older versions? If not, overwrite the key. If yes, what would 'version' of the key give you? You already know your version from the key. If you arrange your keys as pathOfUniqueAsset.v11, you can later issue KEYS pathOfUniqueAsset.* (or better SCAN) to get all versions. This way you can set EXPIRE individually. If you'll use HSET like #ItamarHaber suggests you can only delete values manually, but iterating a set is much faster than KEYS lookup (EDIT: actually, it depends on some factors, mostly the amount of other keys).
If you instead want a list of assets for each version kept together, you can use dedicated set of all keys associated to this version. Like
SET /pathOfUniqueAsset/v1 ...
HSET assets.v1 pathOfUniqueAsset /pathOfUniqueAsset/v1
Related
If two distinct keys have the same value (and say the value is large) does redis store the value twice or will it use a pointer / reference. The way git does ?
Redis stores them as two independent key-value pairs.
If you want to remove the duplication, you have to build an index from multiple keys to a shared value by yourself.
I am moving some "live" data structures from MySQL to REDIS. Using StackExchange C# Redis Client, I'm writing (due to some very project-specific restrictions) my own microORM code to store and retrieve object class entities from a Redis Database.
I am pushing c# object as hash keys in Redis.
My general question is about indexing on fields other than the "primary key".
Ok, I've read all the theory of sets and sorted sets, and how to add and remove members from sets, and so on.
I've added some code to correctly create set keys which contain entities hash keys, so that I can lookup those objects by simple indexes or sorted indexes.
However I cannot find or figure out a good strategy for solving the following problems:
1. Index maintenance on expiration
I'd like to add expiration to some object (hash) keys, so that old entities get purged automatically by Redis. However I cannot find a reilable way to update/purge relevant indexes besides running periodically a background task that scans index set keys for expired members and removes them (notification is not good for me)
2. Index updating when some object fields change
In some cases I need to update only a small fraction of hash key values, not the whole entity. If the fields being updated are part of one or more index set keys, I cannot figure out the best way to properly update the set keys.
For example, let's say I need to store a "Session" entity whose primary key is its ID (simple numerical integer), and I need to add an index on the "Node" string field (Node being the reference to the server currently serving the session):
class Session {
[RedisKey]
public int ID { get; set; }
public string RemoteIP { get; set; }
[RedisSimpleIndex]
public string Node { get; set; }
}
RedisKey and RedisSimpleIndex are attributes I use to extract via reflection which fields are used as primary key and which are used for indexing.
Let's suppose I have an instance of Session like this:
{ ID = 2, RemoteIP = "1.2.3.4", Node = "Server10" }
My routines are creating the following keys in Redis:
Hash key: "obj:Session:2"
Hash values: "ID" = "1", "RemoteIP" = "1.2.3.4", "Node" = "Server10"
Set key "idx:Session:Node:Server10"
Set members: "obj:Session:2"
which is fine for looking up all sessions on Server10.
However, if the very same session needs to be moved to a different server (e.g. Server8)and I want to update only the Node field in the Hash set, how can I update indexes too?
The only way I found so far is to SCAN all index keys with pattern idx:Session:Node:* and remove from them any member obj:Session:2, then create/update the index key for the new node (idx:Session:Node:Server8).
Moreover the SCAN command is not available in IDatabase or ITransaction interfaces, and in a HA Clustered environment things get worse since I need to determine which Redis server is holding relevant keys to make this procedure work.
Is there a better way to build/represent simple indexes in Redis? Is my approach wrong?
I'd like to add expiration to some object (hash) keys, so that old entities get purged automatically by Redis. However I cannot find a reilable way to update/purge relevant indexes besides running periodically a background task that scans index set keys for expired members and removes them (notification is not good for me)
You cannot expire individual KV pairs within a hash. This is was discussed in #167. There don't appear plans to change this.
I think, you should be able to use keyspace notifications to subscribe to expire events. You would have to have some worker that subscribes for them and updates all relevant indices accordingly. However, you might get some inconsistent data. For example, your worker might crash and leave the stale indices behind. Also the indices wouldn't be updated instantaneously, so you'd end up with a bit of stale data regardless.
Probably not the best idea, but you could also hack in some custom indexing logic into expire.c. The code seems fairly straightforward. The C module API by contrast doesn't appear to provide any way to hook into the eviction logic.
Another option is to not rely on Redis when it comes to handling expiration logic. So... you would still have a background job, but it would actually issue corresponding DEL commands for expired KV-pairs. This would also allow you to keep the index 100% up to date via transactions.
In some cases I need to update only a small fraction of hash key values, not the whole entity. If the fields being updated are part of one or more index set keys, I cannot figure out the best way to properly update the set keys.
I'm not sure which Redis client you're using, but I found the following pattern to be quite useful in the past:
You have some form of "Updater" class for each hash. It has setters for all relevant fields that could be updated (setFirstName, setLastName etc.).
When you set a field, you mark that particular field as "dirty" (e.g. via a separate boolean).
When you call "save", you update indices for fields that were marked as dirty.
The only way I found so far is to SCAN all index keys with pattern idx:Session:Node:* and remove from them any member obj:Session:2, then create/update the index key for the new node (idx:Session:Node:Server8).
This is cumbersome, but seems like the way to go. Sadly I don't think there is a better solution for this. You might want to consider maintaining a separate set with keys of index KV-pairs that would have to be updated though, as that way you'd avoid going over a bunch of keys that aren't relevant.
You might also want to check out an article about how to maintain those indices. As you already alluded to, there are basically two options: real-time using MULTI transactions or using batch jobs. Once you get into the territory of using key expiration, you are more or less forced to use the batch approach.
What's the practical difference between keeping data in multiple hashes (HSET foo oof 1, HSET bar rab 2) and using plain keys in a hierarchy (SET foo:oof 1, SET bar:rab 2)?
According to the manual, you'd use hashes to represent a single object.
Also, it's not that efficient to iterate over Redis keys, so if you need to get all the data from a single object, HGETALL is your friend, not a KEYS thing:10:*/multiget fiasco.
However, you can't e.g. set expiry for only one key of a hash, so if you need that functionality, you'll want to use regular keys.
I'm not sure if this can be done in Redis, but is there a way to look up the key by value in redis?
Suppose that I have a redis DB where the key is the symptom_id and the value is the symptom_name.
For exammple:
{
"symptom_id:1": "headache",
"symptom_id:2": "stomach pain",
"symptom_id:3": "cough"
}
Let's say I want to find the symptom ID of "cough". Is it possible to do so? If so, how do I go about doing it?
If you have to do it regularly, use secondary index, as #for_stack suggested. This is a fast and proper way.
If you need an ad-hoc solution, you can iterate keys by pattern with SCAN symptom_id:* ... (see scan), and look at their values.
How I can find keys matching a pattern like this:
Eg:
I have some keys:
abc:parent1
abc:parent2
abc:parent1:child1
abc:parent2:child2
How can I find only
abc:parent1
abc:parent2
Keys is specifically noted as a command not to be run in production due to the way it works. What you need here is to create an index of your keys. Use a set for storing the key names of the pattern you want. When you add a new we key, add the name of it to the set. For example:
Set abc:parent1:child1 breakfast
Sadd abc:parent1:index abc:parent1
Then when you need the list:
Smembers abc:parent1:index
Will give you the list, without the penalties and problems associated with using the "evil" keys command. Additionally you would remove an entry with sremove on key deletion. You also get as a benefit the ability to know how many keys are in the index with a single call.
If you absolutely, positively, MUST avoid using an index use SCAN instead of keys. The only time you should even consider keys is if you are running a debug slave where the only process using it is your debugging process.
Command KEYS pattern
will help you for the same, if it is not a production environment. (Never use keys in production)
ex:
redis> MSET one 1 two 2 three 3 four 4
OK
redis> KEYS *o*
1) "two"
2) "one"
3) "four"
For your specific example, the below command will work:
redis 127.0.0.1:6379> keys *parent[0-9]
1) "abc:parent2"
2) "abc:parent1"
Here's the detailed description of the command.
Update: Though the suggestion above helps you get the desired output, the redis KEYS command is evil as the others mentioned. KEYS is blocking and can consume a lot of RAM while preparing the response.
Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using SCAN or sets.
Thanks The Real Bill and Itamar, I got a better understanding.