The https://redis.io/topics/cluster-tutorial states that redis cluster is not strongly consistent. The reasoning it states even if WAIT is enabled is:
The node to which the update wasn't synced becomes master
After the partition and before the node timeout master in minority partition keeps receiving updates.
What if for a key k we find the master node M and the replicas r1,r2....rn using
CLUSTER SLAVES node-id
. And execute
WAIT N
and only proceed with the transaction if it return N? Wouldn't that always ensure that the data is perfectly synced before executing transaction. Wouldn't that ensure strong consistency ?
NO, it still CANNOT guarantee.
Although WAIT returns N, which means all replicas have acknowledge the writes in memory, these nodes might fail before these writes operation are written to disk.
Related
I need to load static data one time in redis in the master node and only when the synchronization is finished for all slaves I am going to be able to read. This is because we are going to have a lot reading and a few writing, and the data is not going to change for a long time.
I read from oficial documentation https://docs.redis.com/latest/rs/concepts/data-access/consistency-durability/, https://docs.redis.com/latest/rs/concepts/data-access/consistency-durability/ and https://redis.io/topics/cluster-tutorial in Redis Cluster consistency guarantees.
I read also Can the WAIT command provide strong consistency in Redis? but without to get a conclusion.
If I use synchronous replication and wait command to check if the replication was successful, do I have some guarantees about consistency ?
By default, a Redis Cluster is not able to guarantee strong consistency. It means that under certain conditions it is possible that Redis Cluster will lose writes that were acknowledged by the system to the client.
The reason why Redis Cluster can lose writes is because it uses asynchronous replication, however, you can improve consistency by forcing the database to flush data to disk before replying to the client, but this usually results in prohibitively low performance. That would be the equivalent of synchronous replication in the case of Redis Cluster. Basically, there is a trade-off to be made between performance and consistency, if you are fine with that!
Redis Cluster has support for synchronous writes when absolutely needed, implemented via the WAIT command. This makes losing writes a lot less likely. However, note that Redis Cluster does not implement strong consistency even when synchronous replication is used: it is always possible, under more complex failure scenarios, that a replica that was not able to receive the write will be elected as master.
There is another notable scenario where Redis Cluster will lose writes, that happens during a network partition where a client is isolated with a minority of instances including at least a master.
For example, imagine a 6 nodes cluster composed of A, B, C, A1, B1, C1, with 3 masters and 3 replicas. There is also a client, let's call it Z1.
After a partition occurs, it is possible that in one side of the partition we have A, C, A1, B1, C1, and in the other side we have B and Z1.
Z1 is still able to write to B, which will accept its writes. If the partition heals in a very short time, the cluster will continue normally. However, if the partition lasts enough time for B1 to be promoted to master on the majority side of the partition, the writes that Z1 has sent to B in the meantime will be lost.
Note that there is a maximum window to the amount of writes Z1 will be able to send to B: if enough time has elapsed for the majority side of the partition to elect a replica as master, every master node in the minority side will have stopped accepting writes.
This amount of time is a very important configuration directive of Redis Cluster, and is called the node timeout.
After node timeout has elapsed, a master node is considered to be failing, and can be replaced by one of its replicas. Similarly, after node timeout has elapsed without a master node to be able to sense the majority of the other master nodes, it enters an error state and stops accepting writes.
I want to restart my Scylla db cluster. But I don't want to lose any data.
Do I lose any data if I restart one after other node?
No, you will not loose data if you are doing a rolling restart.
Scylla keeps the data replicated across multiple nodes (usually 3 or more)
Depending on your Replication Factor (RF) and Consistency Level (CL) you might see read or write operations failed during the restart. See interactive calc here https://docs.scylladb.com/getting-started/consistency/#consistency-level-calculator
If "restarting a node" just involves restarting Scylla or rebooting the kernel on which it runs, then you're safe: Scylla is a distributed database, and is designed to support durability and availability even when nodes temporarily disappear from the network. When a node is temporarily down, all its data is still available for reads (from two other replicas), and also writes continue to work normally and will be eventually replicated to the down node when it finally comes up (using the "hinted handoff" and/or "repair" mechanisms).
However, if by "restarting a node" you mean something more destructive - replacing it with a brand-new node with empty storage, as in some cloud setups where nodes have transient storage. In that case you have to be more careful: If the node's data is lost, we still have two more replicas and the database continues to be available, but you should tell the cluster to "stream" the data which the node lost back to the node - before continuing to do this destructive restart to additional nodes. If you have RF=3 and destroy three nodes at the same time, you will surely lose data.
I am very new in REDIS cache implementation.
Could you please let me know what is the replication factor means?
How it works or What is the impact?
Thanks.
At the base of Redis replication (excluding the high availability features provided as an additional layer by Redis Cluster or Redis Sentinel) there is a very simple to use and configure leader follower (master-slave) replication: it allows replica Redis instances to be exact copies of master instances. The replica will automatically reconnect to the master every time the link breaks, and will attempt to be an exact copy of it regardless of what happens to the master.
This system works using three main mechanisms:
When a master and a replica instances are well-connected, the master keeps the replica updated by sending a stream of commands to the replica, in order to replicate the effects on the dataset happening in the master side due to: client writes, keys expired or evicted, any other action changing the master dataset.
When the link between the master and the replica breaks, for network issues or because a timeout is sensed in the master or the replica, the replica reconnects and attempts to proceed with a partial resynchronization: it means that it will try to just obtain the part of the stream of commands it missed during the disconnection.
When a partial resynchronization is not possible, the replica will ask for a full resynchronization. This will involve a more complex process in which the master needs to create a snapshot of all its data, send it to the replica, and then continue sending the stream of commands as the dataset changes.
Redis uses by default asynchronous replication, which being low latency and high performance, is the natural replication mode for the vast majority of Redis use cases.
Synchronous replication of certain data can be requested by the clients using the WAIT command. However WAIT is only able to ensure that there are the specified number of acknowledged copies in the other Redis instances, it does not turn a set of Redis instances into a CP system with strong consistency: acknowledged writes can still be lost during a failover, depending on the exact configuration of the Redis persistence. However with WAIT the probability of losing a write after a failure event is greatly reduced to certain hard to trigger failure modes.
wanted to ask
if the SETNX command (both the NX test and the SET) guaranteed to execute on the redis master in the context of the "Redis Sentinel System" ?
if it is guaranteed to to be atomic in the context of the "Redis Sentinel System"?
My understanding after reading the documentation is YES to both, because:
Ad. 1 only the master can accept writes and since SETNX has a set/write component it has to go to the master (because all writes go through the master).
Ad. 2 since the SET will be executed on the master it only make sense to check the NX part also on the master (no slaves queried, ever), otherwise it would be unnecessarily time consuming and could undermine atomicity.
Can someone confirm with 100% certainty, maybe point me to some documentation that clears my doubts.
Thanks in advance!
I can confirm the above with %99.97 (3 sigmas) certainty.
Ad. 1 only the master can accept writes and since SETNX has a set/write component it has to go to the master (because all writes go through the master).
Correct, excluding the scenario where you deliberately enable writing to replicas and connect to a replica.
Ad. 2 since the SET will be executed on the master it only make sense to check the NX part also on the master (no slaves queried, ever), otherwise it would be unnecessarily time consuming and could undermine atomicity.
Yep.
In redis master-slave architecture, when a master fails a slave is promoted to master. As only master can perform write operations, What happens to data in the window period when slave is promoted to master. Does my system remain unresponsive?
Define "data":)
Client connections to the master will be closed upon its failure, so your system will be notified of that. Any data that was not written to the master and the replicas before the failure will therefore still reside in your application/system.
Once your system tries using a replica it will be able to read the data in it up to the point it was synchronized before failure. Once the replica is promoted to masterhood, your system will be able to continue writing data.
Note that Redis' synchronization is asynchronous. That means that slaves may lag behind the master and therefore lose some updates in case of failure. Refer to the WAIT command for more information about ensure the consistency.