I have multiple keys in redis most of which are insignificant and can be lost in case my redis server goes down.
However I have one or two keys, which I cannot afford to lose.
Hence I would like that whenever the server restarts, redis reads only these few keys from its persistent storage, and keep on persisting these as and when they change.
Does redis have this feature? If yes what command makes a key persisted to file and how to differ b/w persisted and unpersisted keys.
If no, What approach can I use such that I need not make my own persistent file before writing to Redis
Limitations(If the answer is no)
I do not want to change client code that enters in redis.
I cannot add more servers to redis(if any such solution exists, would like to know about it though).
EDIT
Another reason I would not want to save most keys as persistence because it is huge data, hundreds of records per second- Most of which expires in 10 minutes.
Related
I have three microservice that needs to communicate between each other. Microservice-1 is incharge of the data and the database(he writes and read to it). I will add a redis cache store for Microservice-1 to cache data there. I want to put a redis-slave for the other 2 microservices to reduce communication with the actual microservice, if the data is already in the cache store. Since all updates to the data, has to go thru the Microservice-1 and he will always update the cache, redis replication will make sure the other two microservices will get it too. Ofcourse, if the data is not in cache, it will call the Microservice-1 for the data, which will update the cache.
Am i missing something, with this approach ?
This will definitely work in the "sunny day" case.
But sometimes there are storms, and in storms there's a chance of losing cache coherency (i.e. the DB and Redis disagree on the data).
For example, lets say that you have Microservice-1 update the DB and then update Redis. What happens if there's a crash between updating the DB and updating Redis?
On the other hand, what if you reverse the ordering (update Redis and then the DB)? Now Redis could be updated and not the DB.
Neither of these in insurmountable, but absent a means of having a transaction which ensures that 0 or 2 of Redis and the DB are updated, there will always be a time window where the change is in one but not the other. In that situation, it's probably worth embracing eventual consistency (e.g. periodically scan the DB and update redis with recently updated records).
As an elaboration on that, a Command Query Responsibility Segregation with Event Sourcing (CQRS/ES) approach may prove useful: Microservice-1 gets split into two services, one which takes commands (requests to update) and another which handles queries. Instead of updating a row in a DB, the command service now appends (in a typical DB, an INSERT) an event which describes what changed. The query service can subscribe to those events and update Redis. Other microservices can also subscribe to the stream of events and update their own views (which can be remixed in any way they want) of Microservice-1's state.
I have multiple keys in redis for session management and the lifetime of each key is 10 sec. So, losing these keys are fine for me as I can recreate a session.
However, I have one key, which I cannot afford to lose. Hence I would like that whenever the server restarts, redis reads only this keys from its persistent storage, and keep on persisting these as and when they change.
Is there an inbuilt redis way to achieve this?
Yes, reference
https://redis.io/topics/persistence
redis supports persistence.
Redis SAVE and BGSAVE commands dump the complete Redis data to a persistent file.
But is there a way to dump only one DB index?
I am using the same Redis server with multiple DB indices.
I use DB 0 as config which is edited manually and contains just a small number of keys. I wish to dump this to a file as a config snapshot (versioned) to keep track of manual changes in the prod environment.
The rest of the DBs have a large number of items, that will take too long to dump, and I don't need to back them up.
Redis' persistence scope is the entire instance, meaning all shared/numbered databases and all keys in them. Saving only a subset of these is not supported.
Instead, use two independent Redis instances and configure each to persist (or not) per your needs. The overhead of running an insurance is a few megabytes so it is practically negligible.
Ok, so what I have are 2 web servers running inside of a Windows NLB clustered environment. The servers are identical in every respect, and as you'd expect in an NLB clustered environment, everybody is hitting the cluster name and not the individual members. We also have affinity turned off on the members in the cluster.
But, what I'm trying to do is to turn on some caching for a few large files (MP3s). It's easy enough to dial up a Redis node on one particular member and hit it, everything works like you'd expect. I can pull the data from the cache and serve it up as needed.
Now, let's add the overhead of the NLB. With an NLB in play, you may not be hitting the same web server each time. You might make your first hit to member 01, and the second hit to 02. So, I'd need a way to sync between the two servers. That way it doesn't matter which cluster member you hit, you are going to get the same data.
I don't need to worry about one cache being out of date, the only thing I'm storing in there is read only data from an internal web service.
I've only got 2 servers and it looks like redis clusters need 3. So I guess that's out.
Is this the best approach? Or perhaps there is something else better?
Reasons for redis: We only want the cache to use in-memory only. No writes to the database. Thought this would be a good fit, but need to make sure the data is available in both servers.
It's not possible to have redis multi master (writing on both). And I might say it's replication is blazing fast (check the slaveof command of Redis).
But why you need it in the same server? Access it as a service. So every node will access the actual data. If the main server goes down, the slave will promptly turn itself into a master.
One observation: you might notice that Redis makes use of disk in an async way. An append only file that it does checkpoint depending on the size from time to time so.
Good day!
Suppose we have a redis-master and several slaves. The master goal is to store all data while slaves are used for quering data for users. Hovewer quering is a bit complex and some temporary data needs to be stored. And also I want to cache the query result for a couple of minutes.
How should I configure replication to save temporary data and caches?
Redis slaves have optional support to accept writes, however you have to understand a few limitations of writable slaves before to use them, since they have non trivial issues.
Keys created on the slaves will not support expires. Actually in recent versions of Redis they appear to work but are actually leaked instead of expired, until the next time you resynchronize the slave with the master from scratch or issue FLUSHALL or alike. There are deep reasons for this issue... it is currently not clear if we'll deprecate writable slaves at all, find a solution, or deny expires for writable slaves.
You may want, anyway, to use a different Redis numerical DB (SELECT command) in order to store your intermediate data (you may use MULTI/.../MOVE/EXEC transaction in order to generate your intermediate results in the currently selected DB where data belongs, and MOVE the keys off to some other DB, so it will be clear if keys are accumulating and you can FLUSHDB from time to time).
The keys you create on your slave are volatile, they may go away in any moment when the master will resynchronize with the slave. Does not look like an issue for you since if they key is no longer there, you could recompute, but care should be take,
If you elect this slave into a master you have additional keys inside.
So there are definitely things to take in mind in this setup, however it is doable in some way. However you may want to consider alternative strategies.
Lua scripts on the slave side in order to filter your data inside Lua. Not as fast as Redis C commands often.
Precomputation of data directly in the actual data set in order to make your queries possible just using read only commands.
MIGRATE in order to migrate interesting keys from a slave to an instance (another master) designed specifically to perform post-computations.
Hard to tell what's the best strategy without in-depth analysis of the actual use case / problem, but I hope this general guidelines help.