How to get all the keys/data stored in "redis" in the last one hour. I searched to found it out, but couldn't find a way. Is there any way to get this.?
Redis does not have a direct way to do this.
Depending on your use case, in increasing order of complexity -
You can manually add newly created keys to a set. The name of the set can include the timestamp. You can then query this set to find keys that have been modified
You can use redis keyspace notification to get notified of keys when they are changed. However, be aware that pub/sub notifications are "fire and forget" - so if your connection drops - you will lose some of the keys that were updated.
You can look at the AOF file and identify keys that have been created / modified. If you are using a cloud provider for redis - they may not provide access to the AOF file. Also, the AOF file doesn't have the timestamp, but the commands are in the order they were processed by redis.
Related
If we enable the AppendFileOnly in the redis.conf file, every operation which changes the redis database is loggged in that file.
Now, Suppose Redis has used all the memory allocated to it in the "maxmemory" direcive in the redis.conf file.
To store more data., it starts removing data by any one of the behaviours(volatile-lru, allkeys-lru etc.) specified in the redis.conf file.
Suppose some data gets removed from the main memory, But its log will still be there in the AppendOnlyFile(correct me if I am wrong). Can we get that data back using this AppendOnlyFile ?
Simply, I want to ask that if there is any way we can get that removed data back in the main memory ? Like Can we store that data into disk memory and load that data in the main memory when required.
I got this answer from google groups. I'm sharing it.
----->
Eviction of keys is recorded in the AOF as explicit DEL commands, so when
the file is replayed fully consistency is maintained.
The AOF is used only to recover the dataset after a restart, and is not
used by Redis for serving data. If the key still exists in it (with a
subsequent eviction DEL), the only way to "recover" it is by manually
editing the AOF to remove the respective deletion and restarting the
server.
-----> Another answer for this
The AOF, as its name suggests, is a file that's appended to. It's not a database that Redis searches through and deletes the creation record when a deletion record is encountered. In my opinion, that would be too much work for too little gain.
As mentioned previously, a configuration that re-writes the AOF (see the BGREWRITEAOF command as one example) will erase any keys from the AOF that had been deleted, and now you can't recover those keys from the AOF file. The AOF is not the best medium for recovering deleted keys. It's intended as a way to recover the database as it existed before a crash - without any deleted keys.
If you want to be able to recover data after it was deleted, you need a different kind of backup. More likely a snapshot (RDB) file that's been archived with the date/time that it was saved. If you learn that you need to recover data, select the snapshot file from a time you know the key existed, load it into a separate Redis instance, and retrieve the key with RESTORE or GET or similar commands. As has been mentioned, it's possible to parse the RDB or AOF file contents to extract data from them without loading the file into a running Redis instance. The downside to this approach is that such tools are separate from the Redis code and may not always understand changes to the data format of the files the way the Redis server does. You decide which approach will work with the level of speed and reliability you want.
But its log will still be there in the AppendOnlyFile(correct me if I am wrong). Can we get that data back using this AppendOnlyFile ?
NO, you CANNOT get the data back. When Redis evicts a key, it also appends a delete command to AOF. After rewriting the AOF, anything about the evicted key will be removed.
if there is any way we can get that removed data back in the main memory ? Like Can we store that data into disk memory and load that data in the main memory when required.
NO, you CANNOT do that. You have to take another durable data store (e.g. Mysql, Mongodb) for saving data to disk, and use Redis as cache.
I have multiple keys in redis most of which are insignificant and can be lost in case my redis server goes down.
However I have one or two keys, which I cannot afford to lose.
Hence I would like that whenever the server restarts, redis reads only these few keys from its persistent storage, and keep on persisting these as and when they change.
Does redis have this feature? If yes what command makes a key persisted to file and how to differ b/w persisted and unpersisted keys.
If no, What approach can I use such that I need not make my own persistent file before writing to Redis
Limitations(If the answer is no)
I do not want to change client code that enters in redis.
I cannot add more servers to redis(if any such solution exists, would like to know about it though).
EDIT
Another reason I would not want to save most keys as persistence because it is huge data, hundreds of records per second- Most of which expires in 10 minutes.
Is there anyway to create a Redis database where keys HAVE TO expire after a certain time? I know I can expire an individual key using the EXPIRE command but since I am expiring every key after a certain time anyways, it would be nice to have this behavior specified in the Redis config file.
No, Redis (up to and including v3.2) does not provide the means for automatically setting the TTL of newly-created keys. You have to set it explicitly for each key you create.
I'm getting over 10000 updates in 60 seconds in my Redis server and this triggers the background save which consumes resources.
I want to track the changed keys so that I can debug my app (which method causing this much change).
Is there a way to get updated keys?
While MONITOR is perfectly valid, it does include everything that gets sent to Redis. That means filtering read requests, pings, ...
Instead I recommend that you check the keyspace notifications documentation and configure your database the AK flags. By subscribing to the __keyspace:* pattern you'll be notified about every change to keys.
As I learned, it's only possible by using MONITOR command and figure out from output.
I'm using Redis for storing simple key, value pairs; where, value is also of string type. In my Redis cluster, I've a master and two slaves. I want to propagate any of the changes to the data from one of the slaves to any other store (actually, oracle database). How can I do that reliably? The sink database only needs to be eventually consistent. Some delay is allowed.
Strategies I can think of:
a) Read the AOF file written by the slave machine and propagate the changes. (Requires parsing the AOF file and getting notified of every change to the file.)
b) Use rpoplpush. The reliable queue pattern provided. But, how to make the slave insert to that queue whenever it gets some set event from the master?
Any other possibility?
This is a very common problem faced by Redis developers. In a nutshell, it is the fact that:
Want to know all changes sinse last
Keep this change data atomic
I believe that any decision one way or another will be around these issues. So, yes AOF is one of best choises in this case, but here is not any production ready instruments for that. Yes, it is not very complex solution in case of one server but then using master/slave or cluster it can be very complex.
Using Keyspace notifications
Look's like Keyspace Notifications feature may be alternative. Keyspace notifications is a feature available since 2.8.0 and available in Redis cluster too. From original documentation:
Keyspace notifications allows clients to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way.Examples of the events that is possible to receive are the following:
All the commands affecting a given key.
All the keys receiving an LPUSH operation.
All the keys expiring in the database 0.
Events are delivered using the normal Pub/Sub layer of Redis, so clients implementing Pub/Sub are able to use this feature without modifications.
Because Redis Pub/Sub is fire and forget currently there is no way to use this feature if you application demands reliable notification of events, that is, if your Pub/Sub client disconnects, and reconnects later, all the events delivered during the time the client was disconnected are lost. This can be improved by duplicating the employees who serve this Pub/Sub channel:
The group of N workers subscribe to notification and put data to SET based "sync" list. This allow us control overhead and do not write same data to our sync list.
The other group of workers pop record with SPOP and write it other store.
Using manual update list
The other way is using special "sync" SET based list with every write operation (as i understand SET/HSET in your case). Something like:
MULTI
SET myKey value
SADD myKey
EXEC
Each time you modify your key you add key name to SET. So in other process or worker you can SPOP that key, read value and update source.
Also you can use RPOPLPUSH/LPOPRPUSH besides of SPOP in some kind of in progress list to protect your key would missed if worker failed. In this case each worker first RPOPLPUSH/LPOPRPUSH from sync set to in progress set, push data to storage and remove key from in progress set.