I'm trying to set up Redis Sentinel.
I know that when a master goes down the sentinel pick up one of its slaves and promote it as master.
I was wondering based on which attributes the new master is selected among the slaves and which slave got selected for being a new master?
After Sentinels election, the leader sentinel will do the following steps:
Remove slaves already in down status from slave list.
Remove slaves which disconnection time is more than ten times of down-after-milliseconds + master down time
Select slave(s) by replica-priority(configured in slave)
If multiple slaves are selected, sort them by sync offset, and select the most in-sync(maximum offset) slave.
If there are still multiple selection, sort with RunId and select the smaller one.
So you can see the process order of master selection can be following order:
Disconnection time
Priority
Replication offset
Run Id
Related
we have many datacenters but datacenter1 is the main.
the master in datacenter1 is being monitored by sentinel so if the master goes down one the replicas will become master and also all data is being synced continuously.
we want to have one Redis replica in each datacenter, replicate all data from datacenter1 but without the ability to become master. (always get data from data center 1 and just replica 1 have the ability to become master but other replicas must not be able)
is there a Redis config for this or any idea?
Redis Multi Datacenter
Redis config [1] has a replica-priority parameter which should serve your purpose.
The replica priority is an integer number published by Redis in the INFO
output. It is used by Redis Sentinel in order to select a replica to promote
into a master if the master is no longer working correctly.
A replica with a low priority number is considered better for promotion, so
for instance if there are three replicas with priority 10, 100, 25 Sentinel
will pick the one with priority 10, that is the lowest.
However a special priority of 0 marks the replica as not able to perform the
role of master, so a replica with priority of 0 will never be selected by
Redis Sentinel for promotion.
By default the priority is 100.
The idea can be setting lower replica-priority value to replicas in datacenter1 and higher value to replicas in other datacenters.
[1] redis.conf file of Redis version 6.2.6: https://github.com/redis/redis/blob/6.2.6/redis.conf
Been searching for a specific info but couldn't find; forgive me for being new at this.
I will try to replicate a Firebird DB using SymmetricsDS. This is an ERP database; which in my mind will have 1 master and 2 slaves. I will have 2 slave servers which will work locally and local machines will connect them as clients.
Say for example I am a client of local slave 1. I am creating a new customer which will automatically get a customer ID 100. At the same time a client of the local slave (server) 2 creates a new customer and it takes the same customer ID. Now when these two slaves sync to the master; there will be a conflict.
I know this sounds quite noob; you know you can't hide it.
What would be the best approach to prevent this; rather solving?
I don't think there is one "the best" approach. It depends on system specific details what works best... anyway, some options are:
UUID
Use UUID as customer ID. Since version 2.5 Firebird has some built in support for generating and converting UUIDs.
Segmented generators
On each local slave init the customer ID sequence so that IDs generated by it doesn't overlap with other slaves. Ie if you use 32 bit integers as PK and need max two slaves you dedicate top bit as "slave ID". That means that on first slave you start the sequence from zero while at the second you starti it from 2147483648 (bin 1000 0000 0000 0000 0000 0000 0000 0000). See the ALTER SEQUENCE statement for how to set the sequence's starting value.
ID server
You could have a service which generates IDs. Whenever slave needs ID for a customer it recuests it from a special service. To help with the perfomance it probably makes sense to request new IDs in patches and cache them for later use.
I suppose the system is legacy and you don't have the ability to change how it works. In a similar occasion I have solved this problem letting each slave generating sequences. I've added a write filter in symmetricDs on the master node that will intercept each push from a slave and add a unique prefix per slave. If data has to be synced back to the slaves after data is routed to each slave add a write filter to symmetric slave that will strip the added prefix.
For example maximum number of slaves is 99. Let's say slave 1 creates a sequence 198976, assuming the sequence length is 10, use slave's ID, pad left the sequence with zeros and add the slave id as prefix: (0)100198976. If slave 17 generated the same sequence, master node's filter would change it to 1700198976.
If the same data has is changed on the master and has to be sent back to the slave that generated it, write filter on the slave will strip the first two digits (after left padding with 0 in case of one digit slave IDs). Slave 1's sequence from master (0)100198976 will become again 198976; and slave 17's sequence from master 1700198976 will become 198976.
If the whole length of the ID column has been used on the slaves, alter the column on the master by widening the it to accommodate for the width of slave IDs
How to handle this situation.
1) there is a setup of 1 Master (M) and 2 Slaves (S)
2) you perform an increment of a value (and then you use it as a unique identifier)
3) it increments on Master, but fails to sync to Slaves (i.e. network lag or issue)
4) Master dies same time
5) New Master has been elected
6) No nodes from the cluster knows about the increment and with next inc it will bring next value, that is a duplicate.
Maybe Redis is not best solution to have a high-speed increment key-value store. Any ideas?
In this scenario, the worker would need to call WAIT after incrementing to ensure that the change is synchronized.
I am writing a script to monitor redis replication latency in a group of redis slaves managed using sentinel. I am looking at the results of the INFO replication command, which look like this:
# Replication
role:master
connected_slaves:5
slave0:ip=x.x.x.x,port=6379,state=online,offset=22246539656,lag=0
slave1:ip=y.y.y.y,port=6379,state=online,offset=22246538633,lag=1
slave2:ip=z.z.z.z,port=6379,state=online,offset=22247193804,lag=0
slave3:ip=n.n.n.n,port=6379,state=online,offset=22246538633,lag=1
slave4:ip=m.m.m.m,port=6379,state=online,offset=22244239193,lag=1
master_repl_offset:22246539199
repl_backlog_active:1
repl_backlog_size:536870912
repl_backlog_first_byte_offset:21709668288
repl_backlog_histlen:536870912
I had thought that the offset for each slave was a measure of how much data had been replicated so far, so I could look at the difference between the master_repl_offset and the offset values for the various slaves to determine the amount of data not yet replicated. However, in the above output, the offsets for slave0 and slave2 are both higher than for the master. Have I misunderstood what these numbers mean?
What could be the reason for spontaneous master object count drop? I've added 3 more nodes to a cluster containing 17 nodes, and suddenly there'are 1 billion records less (as reported by AMC UI)?
Replica objects goes to zero as well.
Thanks!
Partitions are going to go through state transitions. Partitions that were shifted from master to replica will no longer report those objects in the master object counts. These partitions will also be in an "acting master" state until the "desync master" partitions receive the full partition. While acting master the namespace supervisor thread (NSUP) will not count them as master or replica objects. Also the node that previously was the replica for this partition will drop the partition to allow for potential incoming migrations.
There will also be partitions where the l old master and replica nodes will both be displaced by new nodes. The old master will become an acting master until the new master receives the partition. At which point the old master will drop and the new master will begin replicating to the new replica. You will periodically observe this in AMC as the number of active tx migrations will increase.
There are other possible scenarios but the main takeaway is that once migration settles, you object counts will return to normal.