We have a master percona server (10.0.0.1) in the primary data center.
We also have a slave percona server (10.0.0.2) in the secondary data center that follows the master (10.0.0.1). [This slave will become the master in near future.]
In the near future, we will move our secondary data center to our primary data center. The current master (10.0.0.1) will be shut down and the current slave (10.0.0.2) will take its place to become the new master server. In effect, we just need to change the DATABASE_IP_ADDRESS in our application after migration. Before that change, I set up another slave (10.0.0.3) that follows 10.0.0.2 (current slave and future master).
Everything works but the problem is that 10.0.0.3 does not follow 10.0.0.2.
The commands are:
MySQL-slave-1> CHANGE MASTER TO MASTER_HOST='10.0.0.1', MASTER_USER='repl', MASTER_PASSWORD='somePAS$', MASTER_LOG_FILE='mysql-bin.000159', MASTER_LOG_POS=6334000543;
MySQL-slave-2> CHANGE MASTER TO MASTER_HOST='10.0.0.2', MASTER_USER='repl', MASTER_PASSWORD='somePAS$', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=143;
This is called "Master with Relay Slave" replication and finally I found the solution here!
Related
In a 6 nodes Redis Cluster, with 3 master nodes and 3 slave nodes, if let's say a master will go down, the according slave will be promoted. When the old master comes back live will be a slave.
Is it possible to force it somehow from the redis config or otherwise, so that when it comes back live, the old master will be promoted as a master as it was at the beginning?
Thank you!
If the old master comes up with property:
slaveof no one
It will join the cluster as a master, but I don't think you would like to do it.
The 'old master' does not have the latest data, if you force it to become the master, there will be data loss.
I'm currently exploring Redis cluster. I've started 6 instances on 3 physical servers(3 master and 3 slaves) with persistence enabled.
I've noticed that when I kill one of the master instances then it's slave is promoted to master after some time. However, it remains as master even when I start the killed instance.
Since, Redis does asynchronous replication, therefore, I was thinking of a scenario where the master, immediately after flushing the data is killed i.e. it wasn't able to replicate that data.
Will this data get replicated to the new master(initially slave), once
the instance comes back up?
NO. If the master haven't replicate data to slave, the data will be lost. When the old master recovers, it will be become a slave of some other node based on some rules. Then the old master will replicate data from its new master.
Imagine I have two data centers with one redis instance running in each of them. In addition, I have three sentinel instances running in these both data centers, each on a seperate machine, but two of them in the same data center.
Is this a problem?
In worst case, if the first the data center gets unavailable, two of the three sentinel instances + one redis instance shut down simultaneously. If this redis instance was the master, there would be failover to the other data center.
But what happens if data center 1 gets available again? I'd guess this would be the new configuration:
Data center 1 - Sentinel 1 -> Points to master in data center 1
Data center 1 - Sentinel 2 -> Points to master in data center 1
Data center 2 - Sentinel 1 -> Points to master in data center 2
Will redis set the master in data center 1 again as the new master? If so, what happens with database changes occured in the meantime at the master in data center 2?
If the data center with two sentinels go down and also the master Redis node goes down, then the remaining sentinel node won't be able to elect a new master Redis or promote the Redis server in the other data center as master. Majority of the sentinels have to agree upon the failure of the master Redis. They also have to elect a sentinel process as the leader to promote the live Redis server as master; for that also majority of Sentinel processes have to be available. In case of 3 Sentinels deployment, the majority is 2 and when 2 of them are down, no failover for the master Redis will happen.
I know I'm asking something very obvious about cluster failover.
I read on redis.io that, if any master cluster node fails it will affect to other master nodes until slave come to take in charge. In my structure, I'm not defining any slave and just working with 3 masters.
I'm thinking to modify the redis-trib.rb file, which will remove the defected server and will start the cluster with other 2 nodes. I'm confused about a couple of things,
1) Resharding
Could not possible until failed server goes live
2) Minimum 3 node limitation for create cluster
As per bit understanding, redis-trib.rb not allowing me to create cluster for two nodes
There might be some solution in code file :)
3) Automatic Way to Re-Create new structure with live nodes
As programmer point of view, I'm searching something automatic for my system. Something that trigger one command when Redis Cluster fails some tasks happens internally. like
Shutdown all other redis cluster servers
Remove nodes-[port].conf files from all cluster nodes folder
Start redis cluster servers
Run "redis-trib.rb create ip:port ip:port"
I'm just trying to minimize administration work :). Otherwise I need to implement some other algorithm "Data Consistency" here.
If any of you guys have any solution or idea, kindly share.
Thanks,
Sanjay Mohnani
In a cluster with only master nodes, if a node fails, data is lost. Therefore no resharding is possible, since it is not possible to migrate the data (hash slots) out of the failed node.
To keep the cluster working when a master fails, you need slave nodes (one per master). This way, when a master fails, its slave fails over (becomes the new master with the same copy of the data).
The redis-trib.rb script does not handle cluster creation with less than 3 masters, however in redis-cluster a cluster can be of any size (at least one node).
Therefore adding slave nodes can be considered an automatic solution to your problem.
I know I'm asking something very obvious about cluster failover.
I read on redis.io that, if any master cluster node fails it will affect to other master nodes until slave come to take in charge. In my structure, I'm not defining any slave and just working with 3 masters.
I'm thinking to modify the redis-trib.rb file, which will remove the defected server and will start the cluster with other 2 nodes. I'm confused about a couple of things,
1) Resharding
Could not possible until failed server goes live
2) Minimum 3 node limitation for create cluster
As per bit understanding, redis-trib.rb not allowing me to create cluster for two nodes
There might be some solution in code file :)
3) Automatic Way to Re-Create new structure with live nodes
As programmer point of view, I'm searching something automatic for my system. Something that trigger one command when Redis Cluster fails some tasks happens internally. like
Shutdown all other redis cluster servers
Remove nodes-[port].conf files from all cluster nodes folder
Start redis cluster servers
Run "redis-trib.rb create ip:port ip:port"
I'm just trying to minimize administration work :). Otherwise I need to implement some other algorithm "Data Consistency" here.
If any of you guys have any solution or idea, kindly share.
Thanks,
Sanjay Mohnani
In a cluster with only master nodes, if a node fails, data is lost. Therefore no resharding is possible, since it is not possible to migrate the data (hash slots) out of the failed node.
To keep the cluster working when a master fails, you need slave nodes (one per master). This way, when a master fails, its slave fails over (becomes the new master with the same copy of the data).
The redis-trib.rb script does not handle cluster creation with less than 3 masters, however in redis-cluster a cluster can be of any size (at least one node).
Therefore adding slave nodes can be considered an automatic solution to your problem.