In a key-value persistance api I'm porting to Redis, I'm trying to implement a function that updates the time to live for a key. The original code stores ttl as a timestamp and # of minutes; the ttl is updated by writing a new timestamp (the key expires after timestamp + delta).
I've noticed that Redis provides a TTL command, but that only provides the time remaining.
What I'm wondering is if there is a way to retrieve the original TTL from Redis (set with EXPIRE, etc), or if I need to add a TTL meta field to the values I'm storing (like the original code does).
Edit:
I'm using Redis Server v2.4.10
Internally, Redis stores converts the TTL into a unix timestamp. See function expireGenericCommand in db.c. So, Redis cannot return the TTL you specified, simply because it does not store it in that format.
You will need to add a TTL meta field if it is important for your application.
Related
I need to find out whether a data has been modified in a row, so that next time when it gets the value it knows that this particular row has been modified in Redis Database? Somewhat like an SCN value in Sql but in Redis database. Is that possible?
While Redis does not expose a last modification timestamp for keys, you can easily achieve what you are looking for by storing that information on different keys, (possibly) named after the keys you are tracking: for every modification of the key named key, for example, you would also immediately set the key named key_modified_on with the current timestamp.
To make the operation transaction-like, you could use a MULTI/EXEC transaction (or even a Lua script, if needed):
var transaction = database.CreateTransaction();
transaction.StringSetAsync("mykey", "myvalue");
transaction.StringSetAsync("mykey_modified_on", DateTime.UtcNow.ToString("O"));
await transaction.ExecuteAsync();
With that being said, Redis exposes an idle time (not requested by read or write operations) for each key through the OBJECT command, provided (according to the documentation) maxmemory-policy is set to an LRU policy or noeviction and maxmemory is set. In that case, you can just use the KeyIdleTimeAsync() method:
var idleTime = await database.KeyIdleTimeAsync("mykey");
Is there any redis command which tells when key is stored in redis?
I know there is TTL command.
Depending on time, i want to take different action. like if
live since last 1 min then do x,
live since last 2 min then do y,
etc...
There's no such command. However, you can achieve your goal with the EXPIRE and TTL commands.
For each key, set its TTL to be 1000000000, i.e. EXPIRE key 1000000000. So that the key will be expired after about 32 years. It's should be long enough.
When you want to find out how long the key has been stored, just get the key's TTL, i.e. TTL key. And the key has been stored since 1000000000 - TTL seconds.
I have incoming data which i have to aggregate for some time and when the key expires process the data.
I have tried using redis keyspace notifications but it only gives the key.
Is there a better way to handle this scenario ?
Instead of setting an expiry, aggregate the data into a list or set depending on your use case. Put a timestamp in the key itself. For example, if you want to aggregate data for 1 hour, your key can be mydata:2018-26-06-1300, mydata:2018-26-06-1400, mydata:2018-26-06-1500 and so on.
Then you simply run a cron job every hour, read all the values from the key, and delete the key when you are done.
I am using jedis, a redis java client. I have a queue of string items. As per normal I am using lpush lpop rpush rpop for the necessary operations. But I will like to set expiry for each individual items in the queue. Is it possible?
This is not possible in redis by design for the sake of keeping redis simple and fast.
You can either store an expire value along with the string in the list, or store a separate list of expire times to let your application know if the key has expired.
There is also an alternative solution discussed here. You can store values in a sorted set with expire timestamps as scores and only select those members, whose scores are greater than certain timestamp. (This of course leaves it up to your app to clear the expired elements in a set)
I am designing a system that should analyze large number of user transactions and produce aggregated measures (such as trends and etc).
The system should work fast, be robust and scalable.
System is java based (on Linux).
The data arrives from a system that generate log files (CSV based) of user transactions.
The system generates a file every minute and each file contains the transactions of different users (sorted by time), each file may contain thousands of users.
A sample data structure for a CSV file:
10:30:01,user 1,...
10:30:01,user 1,...
10:30:02,user 78,...
10:30:02,user 2,...
10:30:03,user 1,...
10:30:04,user 2,...
.
.
.
The system I am planning should process the files and perform some analysis in real-time.
It has to gather the input, send it to several algorithms and other systems and store computed results in a database. The database does not hold the actual input records but only high level aggregated analysis about the transactions. For example trends and etc.
The first algorithm I am planning to use requires for best operation at least 10 user records, if it can not find 10 records after 5 minutes, it should use what ever data available.
I would like to use Storm for the implementation, but I would prefer to leave this discussion in the design level as much as possible.
A list of system components:
A task that monitors incoming files every minute.
A task that read the file, parse it and make it available for other system components and algorithms.
A component to buffer 10 records for a user (no longer than 5 minutes), when 10 records are gathered, or 5 minute have passed, it is time to send the data to the algorithm for further processing.
Since the requirement is to supply at least 10 records for the algorithm, I thought of using Storm Field Grouping (which means the same task gets called for the same user) and track the collection of 10 user's records inside the task, of course I plan to have several of these tasks, each handles a portion of the users.
There are other components that work on a single transaction, for them I plan on creating other tasks that receive each transaction as it gets parsed (in parallel to other tasks).
I need your help with #3.
What are the best practice for designing such a component?
It is obvious that it needs to maintain the data for 10 records per users.
A key value map may help, Is it better to have the map managed in the task itself or using a distributed cache?
For example Redis a key value store (I never used it before).
Thanks for your help
I had worked with redis quite a bit. So, I'll comment on your thought of using redis
#3 has 3 requirements
Buffer per user
Buffer for 10 Tasks
Should Expire every 5 min
1. Buffer Per User:
Redis is just a key value store. Although it supports wide variety of datatypes, they are always values mapped to a STRING key. So, You should decide how to identify a user uniquely incase you need have per user buffer. Because In redis you will never get an error when you override a key new value. One solution might be check the existence before write.
2. Buffer for 10 Tasks: You obviously can implement a queue in redis. But restricting its size is left to you. Ex: Using LPUSH and LTRIM or Using LLEN to check the length and decide whether to trigger your process. The key associated with this queue should be the one you decided in part 1.
3. Buffer Expires in 5 min: This is a toughest task. In redis every key irrespective of underlying datatype it value has, can have an expiry. But the expiry process is silent. You won't get notified on expiry of any key. So, you will silently lose your buffer if you use this property. One work around for this is, having an index. Means, the index will map a timestamp to the keys who are all need to be expired at that timestamp value. Then in background you can read the index every minute and manually delete the key [after reading] out of redis and call your desired process with the buffer data. To have such an index you can look at Sorted Sets. Where timestamp will be your score and set member will be the keys [unique key per user decided in part 1 which maps to a queue] you wish to delete at that timestamp. You can do zrangebyscore to read all set members with specified timestamp
Overall:
Use Redis List to implement a queue.
Use LLEN to make sure you are not exceeding your 10 limit.
Whenever you create a new list make an entry into index [Sorted Set] with Score as Current Timestamp + 5 min and Value as the list's key.
When LLEN reaches 10, remember to read then remove the key from the index [sorted set] and from the db [delete the key->list]. Then trigger your process with data.
For every one min, generate current timestamp, read the index and for every key, read data then remove the key from db and trigger your process.
This might be my way to implement it. There might be some other better way to model your data in redis
For your requirements 1 & 2: [Apache Flume or Kafka]
For your requirement #3: [Esper Bolt inside Storm. In Redis for accomplishing this you will have to rewrite the Esper Logic.]