In a 6 nodes Redis Cluster, with 3 master nodes and 3 slave nodes, if let's say a master will go down, the according slave will be promoted. When the old master comes back live will be a slave.
Is it possible to force it somehow from the redis config or otherwise, so that when it comes back live, the old master will be promoted as a master as it was at the beginning?
Thank you!
If the old master comes up with property:
slaveof no one
It will join the cluster as a master, but I don't think you would like to do it.
The 'old master' does not have the latest data, if you force it to become the master, there will be data loss.
Related
I'm currently exploring Redis cluster. I've started 6 instances on 3 physical servers(3 master and 3 slaves) with persistence enabled.
I've noticed that when I kill one of the master instances then it's slave is promoted to master after some time. However, it remains as master even when I start the killed instance.
Since, Redis does asynchronous replication, therefore, I was thinking of a scenario where the master, immediately after flushing the data is killed i.e. it wasn't able to replicate that data.
Will this data get replicated to the new master(initially slave), once
the instance comes back up?
NO. If the master haven't replicate data to slave, the data will be lost. When the old master recovers, it will be become a slave of some other node based on some rules. Then the old master will replicate data from its new master.
Let's say I have a variable that is being changed very frequently in the Master
set foo 1
set foo 5
set foo 4
set foo 8
set foo 4
set foo 10
set foo 7
...
If the network goes down, and more similar commands are executed, when network comes up, will the replica be flooded by all those commands, or will it receive only the last one for each different variable?
Thanks in advance
When you use Redis replication, it allows slave Redis instances to be exact copies of master instances. According to the Redis documentation, there are 3 main mechanisms:
Master sends a stream of commands to the slave to replicate the dataset in the master side.
When the link (between master and slave) breaks, for network issues or because a timeout is sensed in the master or the slave, then the slave reconnects and attempts to proceed with a partial resynchronization: it means that it will try to just obtain the part missed streams during the disconnection.
If partial resynchronization is not possible, the slave will ask for a full resynchronization that means the master needs to create a snapshot of all its data, send it to the slave, and then continue sending the stream of commands as the dataset changes.
For more, see here.
Hope this will find you well.
I have 3 replicated Redis instances running on 3 different machines: A, B and C.
I initially choose A as my master.
I also have 3 sentinels (1 on each machine) monitoring A.
In case A goes down, I want sentinels to choose a specific master to failover to (say B).
Is there a way to choose a specific master instead of leaving it to the election mechanism of the sentinels?
Since I couldn't find this question anywhere, I reckon it's not standard procedure so I'll explain the reason behind it:
My application is running on A, B and C behind a load balancer.
The master uses its local Redis db which is replicated to the other two slaves.
When A fails, the load balancer could choose B as master while Redis sentinels could elect C as Redis master.
As I just said, I need the instance to be local, so that's why I need to specify B as the Redis master.
There's a Redis configuration setting called 'slave-priority' that may help you out.
Reference:
http://download.redis.io/redis-stable/redis.conf
I know I'm asking something very obvious about cluster failover.
I read on redis.io that, if any master cluster node fails it will affect to other master nodes until slave come to take in charge. In my structure, I'm not defining any slave and just working with 3 masters.
I'm thinking to modify the redis-trib.rb file, which will remove the defected server and will start the cluster with other 2 nodes. I'm confused about a couple of things,
1) Resharding
Could not possible until failed server goes live
2) Minimum 3 node limitation for create cluster
As per bit understanding, redis-trib.rb not allowing me to create cluster for two nodes
There might be some solution in code file :)
3) Automatic Way to Re-Create new structure with live nodes
As programmer point of view, I'm searching something automatic for my system. Something that trigger one command when Redis Cluster fails some tasks happens internally. like
Shutdown all other redis cluster servers
Remove nodes-[port].conf files from all cluster nodes folder
Start redis cluster servers
Run "redis-trib.rb create ip:port ip:port"
I'm just trying to minimize administration work :). Otherwise I need to implement some other algorithm "Data Consistency" here.
If any of you guys have any solution or idea, kindly share.
Thanks,
Sanjay Mohnani
In a cluster with only master nodes, if a node fails, data is lost. Therefore no resharding is possible, since it is not possible to migrate the data (hash slots) out of the failed node.
To keep the cluster working when a master fails, you need slave nodes (one per master). This way, when a master fails, its slave fails over (becomes the new master with the same copy of the data).
The redis-trib.rb script does not handle cluster creation with less than 3 masters, however in redis-cluster a cluster can be of any size (at least one node).
Therefore adding slave nodes can be considered an automatic solution to your problem.
I know I'm asking something very obvious about cluster failover.
I read on redis.io that, if any master cluster node fails it will affect to other master nodes until slave come to take in charge. In my structure, I'm not defining any slave and just working with 3 masters.
I'm thinking to modify the redis-trib.rb file, which will remove the defected server and will start the cluster with other 2 nodes. I'm confused about a couple of things,
1) Resharding
Could not possible until failed server goes live
2) Minimum 3 node limitation for create cluster
As per bit understanding, redis-trib.rb not allowing me to create cluster for two nodes
There might be some solution in code file :)
3) Automatic Way to Re-Create new structure with live nodes
As programmer point of view, I'm searching something automatic for my system. Something that trigger one command when Redis Cluster fails some tasks happens internally. like
Shutdown all other redis cluster servers
Remove nodes-[port].conf files from all cluster nodes folder
Start redis cluster servers
Run "redis-trib.rb create ip:port ip:port"
I'm just trying to minimize administration work :). Otherwise I need to implement some other algorithm "Data Consistency" here.
If any of you guys have any solution or idea, kindly share.
Thanks,
Sanjay Mohnani
In a cluster with only master nodes, if a node fails, data is lost. Therefore no resharding is possible, since it is not possible to migrate the data (hash slots) out of the failed node.
To keep the cluster working when a master fails, you need slave nodes (one per master). This way, when a master fails, its slave fails over (becomes the new master with the same copy of the data).
The redis-trib.rb script does not handle cluster creation with less than 3 masters, however in redis-cluster a cluster can be of any size (at least one node).
Therefore adding slave nodes can be considered an automatic solution to your problem.