We use spring Redis data indexes to fetch some data apart from the key.
We are using Amazon Elastic Cache in a clustered mode.
It seems the Indexed entries are not getting cleared even if the original entries are getting cleared.
In our Redis Configuration we have subscribed to keyspace events on Startup. But it seems keyspace events don't work reliably as internally Spring data Redis subscribes to any random node.
Please check some links below for details
Spring Redis - Indexes not deleted after main entry expires
https://github.com/spring-projects/spring-data-redis/issues/1111
One recommendation is to subscribe to all the master nodes. I am not sure how to subscribe to all the nodes from Spring Data Redis.
Best Regards,
Saurav
I think you need the following:
#EnableRedisRepositories(enableKeyspaceEvents = EnableKeyspaceEvents.ON_STARTUP)
Check this question and its answer for further detail.
Related
I am new to redis, still reading doc, hope you could help me here.
I need a 2-stage database solution:
At local devices, there is a database cluster. It has several primaries and several replicas. To my understanding each primary or replica normally has a portion of the whole data set. This is called data sharding.
At cloud, there is another database replica. This cloud replica backs up the whole data set.
I like to use free redis for this solution, not enterprise version.
Is this achievable? From what I read so far, it seems that there is no problem if the cloud replica is just like local replica to back up a portion of data set. So I want to know whether I can use the cloud database to back up the whole cluster.
Thanks!
Nothing prevents you from having a replica hosted in the cloud, but each Redis cluster node is either a master responsible of a set of key slots (shards) or a replica of a master; in a multi-master scenario there is no way to have a single replica covering different master nodes.
With the goal of having your entire cluster data replicated in the cloud, you should configure and host there one additional Redis replica per each master node. To avoid those new replicas to ever become masters themselves, you can set their cluster-replica-no-failover configuration property accordingly in their redis.conf files:
cluster-replica-no-failover yes
In all cases, please note that replication is not a backup solution and you may want to pair your setup with a proper Redis persistence mechanism.
If I understand your questions clearly, your master dataset(in shards) are located on premise and the replicas(slave) are hosted on cloud. There is nothing preventing you from backing up your slaves(open source redis) on the cloud. Redis doesn't care where the slaves are situated provided the master can reach them. Master-slave replication is available on redis enterprise with no such restriction. You might have a little problem implementing master-master replication on redis open source but that is outside the scope of this question
I have two questions regarding the reddison client:
Does redisson support automatic synchronization of local cache with remote redis cache (when remote cache data change or invalidate)?
I understand that redisson supports data partitioning only in pro edition but isn't that feature already supported OOTB by redis cluster mode? Am I missing something here?
Answering to your questions:
RLocalCachedMap has two synchronization strategies:
INVALIDATE - Used by default. Invalidate cache entry across all RLocalCachedMap instances on map entry change.
UPDATE - Update cache entry across all LocalCachedMap instances on map entry change.
Right, all Redisson objects works also in cluster mode. Each object tied to some Redis node and its content always remain only on the same Redis node and not distributed. If your object couldn't fit in single Redis node then you need to use data partitioning feature. This feature evenly distributes content of object across multiple Redis nodes in cluster.
Re: "local cache truely local" -- I think you can just use a java Map, initially populate it with a RMap contents then from then on just serve your requests from the 'truely local' map in memory.
I'm using Redis for storing simple key, value pairs; where, value is also of string type. In my Redis cluster, I've a master and two slaves. I want to propagate any of the changes to the data from one of the slaves to any other store (actually, oracle database). How can I do that reliably? The sink database only needs to be eventually consistent. Some delay is allowed.
Strategies I can think of:
a) Read the AOF file written by the slave machine and propagate the changes. (Requires parsing the AOF file and getting notified of every change to the file.)
b) Use rpoplpush. The reliable queue pattern provided. But, how to make the slave insert to that queue whenever it gets some set event from the master?
Any other possibility?
This is a very common problem faced by Redis developers. In a nutshell, it is the fact that:
Want to know all changes sinse last
Keep this change data atomic
I believe that any decision one way or another will be around these issues. So, yes AOF is one of best choises in this case, but here is not any production ready instruments for that. Yes, it is not very complex solution in case of one server but then using master/slave or cluster it can be very complex.
Using Keyspace notifications
Look's like Keyspace Notifications feature may be alternative. Keyspace notifications is a feature available since 2.8.0 and available in Redis cluster too. From original documentation:
Keyspace notifications allows clients to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way.Examples of the events that is possible to receive are the following:
All the commands affecting a given key.
All the keys receiving an LPUSH operation.
All the keys expiring in the database 0.
Events are delivered using the normal Pub/Sub layer of Redis, so clients implementing Pub/Sub are able to use this feature without modifications.
Because Redis Pub/Sub is fire and forget currently there is no way to use this feature if you application demands reliable notification of events, that is, if your Pub/Sub client disconnects, and reconnects later, all the events delivered during the time the client was disconnected are lost. This can be improved by duplicating the employees who serve this Pub/Sub channel:
The group of N workers subscribe to notification and put data to SET based "sync" list. This allow us control overhead and do not write same data to our sync list.
The other group of workers pop record with SPOP and write it other store.
Using manual update list
The other way is using special "sync" SET based list with every write operation (as i understand SET/HSET in your case). Something like:
MULTI
SET myKey value
SADD myKey
EXEC
Each time you modify your key you add key name to SET. So in other process or worker you can SPOP that key, read value and update source.
Also you can use RPOPLPUSH/LPOPRPUSH besides of SPOP in some kind of in progress list to protect your key would missed if worker failed. In this case each worker first RPOPLPUSH/LPOPRPUSH from sync set to in progress set, push data to storage and remove key from in progress set.
I tried to find more info about it online, but cant seem to find a fitting answer.
Our new application uses HA loadbalancers on top to distribute visitors to clustered ampq and clustered mysql and everything works flawlessly.
Now we have decided that we need to store our sessions on REDIS and according to everyone out there.. REDIS seems to be a good choice.
But what I dont understand is, since Redis doesnt support cluster yet in production. How do people achieve HA with Redis? Its all great to setup a Master-Slave REDIS setup, but that means I can only write to the master. What happens if the master die? And even with Redis Sentinel promoting slaves to master.. the replication from master to slave can have a delay and reply me with stale data. How do people prevent that?
But to keep it short, I just dont "see" it. Please enlightenment me! Thank you
Have a look at Twemproxy. It was deisnged to partition data amongst multiple redis masters, so there's no single point of failure; currently, it's the recommended approach to partition redis based on this (scroll to bottom).
Bonus Alert: Here's an interesting article on how to use redis slaves and sentinel with twemproxy, so they all play nice.
try redis-mgr: https://github.com/idning/redis-mgr
redis+twemproxy+sentinel deploy/auto-failover/monitor/migrate/rolling-upgrade
Redis 3.x has clustering functionality in the core.
http://redis.io
Since the redis cluster is still a work in progress, I want to build a simplied one by myselfin the current stage. The system should support data sharding,load balance and master-slave backup. A preliminary plan is as follows:
Master-slave: use multiple master-slave pairs in different locations to enhance the data security. Matsters are responsible for the write operation, while both masters and slaves can provide the read service. Datas are sent to all the masters during one write operation. Use Keepalived between the master and the slave to detect failures and switch master-slave automatically.
Data sharding: write a consistant hash on the client side to support data sharding during write/read in case the memory is not enougth in single machine.
Load balance: use LVS to redirect the read request to the corresponding server for the load balance.
My question is how to combine the LVS and the data sharding together?
For example, because of data sharding, all keys are splited and stored in server A,B and C without overlap. Considering the slave backup and other master-slave pairs, the system will contain 1(A,B,C), 2(A,B,C) , 3(A,B,C) and so on, where each one has three servers. How to configure the LVS to support the redirection in such a situation when a read request comes? Or is there other approachs in redis to achieve the same goal?
Thanks:)
You can get really close to what you need by using:
twemproxy shard data across multiple redis nodes (it also supports node ejection and connection pooling)
redis slave master/slave replication
redis sentinel to handle master failover
depending on your needs you probably need some script listening to fail overs (see sentinel docs) and clean things up when a master goes down