According to ML 9 doco, a database is backed up onto all nodes in the cluster, but the backup process appears to backup the forests that are only local to each node. So for a database with 6 forests across 3 nodes, I may have 2 forests backup files on each node.
If I have a 3 node cluster and lose one node ( so that one node is now 100% unrecoverable ), are all my backups now effectively useless as they will be missing the back up files for 2 forests?
Or is ML smart enough to re-create the missing data from the dead node, via parity?
Thanks.
The answer is it depends.
Typically backups are made to some sort of network storage, so loosing a node doesn't affect the backups. If for some reason the backups are stored locally to the system, then it would depend on if you have HA enabled, and whether you are backing up thee replica forests along with the primary forests.
If HA is enabled, you could lose a node and keep running, giving you time to rebuild the lost node. Alternatively, if you are backing up both the primary and replica forests in your cluster, you would have a complete data set in your backups even if you lose a node.
Related
I am new to redis, still reading doc, hope you could help me here.
I need a 2-stage database solution:
At local devices, there is a database cluster. It has several primaries and several replicas. To my understanding each primary or replica normally has a portion of the whole data set. This is called data sharding.
At cloud, there is another database replica. This cloud replica backs up the whole data set.
I like to use free redis for this solution, not enterprise version.
Is this achievable? From what I read so far, it seems that there is no problem if the cloud replica is just like local replica to back up a portion of data set. So I want to know whether I can use the cloud database to back up the whole cluster.
Thanks!
Nothing prevents you from having a replica hosted in the cloud, but each Redis cluster node is either a master responsible of a set of key slots (shards) or a replica of a master; in a multi-master scenario there is no way to have a single replica covering different master nodes.
With the goal of having your entire cluster data replicated in the cloud, you should configure and host there one additional Redis replica per each master node. To avoid those new replicas to ever become masters themselves, you can set their cluster-replica-no-failover configuration property accordingly in their redis.conf files:
cluster-replica-no-failover yes
In all cases, please note that replication is not a backup solution and you may want to pair your setup with a proper Redis persistence mechanism.
If I understand your questions clearly, your master dataset(in shards) are located on premise and the replicas(slave) are hosted on cloud. There is nothing preventing you from backing up your slaves(open source redis) on the cloud. Redis doesn't care where the slaves are situated provided the master can reach them. Master-slave replication is available on redis enterprise with no such restriction. You might have a little problem implementing master-master replication on redis open source but that is outside the scope of this question
I want to restart my Scylla db cluster. But I don't want to lose any data.
Do I lose any data if I restart one after other node?
No, you will not loose data if you are doing a rolling restart.
Scylla keeps the data replicated across multiple nodes (usually 3 or more)
Depending on your Replication Factor (RF) and Consistency Level (CL) you might see read or write operations failed during the restart. See interactive calc here https://docs.scylladb.com/getting-started/consistency/#consistency-level-calculator
If "restarting a node" just involves restarting Scylla or rebooting the kernel on which it runs, then you're safe: Scylla is a distributed database, and is designed to support durability and availability even when nodes temporarily disappear from the network. When a node is temporarily down, all its data is still available for reads (from two other replicas), and also writes continue to work normally and will be eventually replicated to the down node when it finally comes up (using the "hinted handoff" and/or "repair" mechanisms).
However, if by "restarting a node" you mean something more destructive - replacing it with a brand-new node with empty storage, as in some cloud setups where nodes have transient storage. In that case you have to be more careful: If the node's data is lost, we still have two more replicas and the database continues to be available, but you should tell the cluster to "stream" the data which the node lost back to the node - before continuing to do this destructive restart to additional nodes. If you have RF=3 and destroy three nodes at the same time, you will surely lose data.
I know I'm asking something very obvious about cluster failover.
I read on redis.io that, if any master cluster node fails it will affect to other master nodes until slave come to take in charge. In my structure, I'm not defining any slave and just working with 3 masters.
I'm thinking to modify the redis-trib.rb file, which will remove the defected server and will start the cluster with other 2 nodes. I'm confused about a couple of things,
1) Resharding
Could not possible until failed server goes live
2) Minimum 3 node limitation for create cluster
As per bit understanding, redis-trib.rb not allowing me to create cluster for two nodes
There might be some solution in code file :)
3) Automatic Way to Re-Create new structure with live nodes
As programmer point of view, I'm searching something automatic for my system. Something that trigger one command when Redis Cluster fails some tasks happens internally. like
Shutdown all other redis cluster servers
Remove nodes-[port].conf files from all cluster nodes folder
Start redis cluster servers
Run "redis-trib.rb create ip:port ip:port"
I'm just trying to minimize administration work :). Otherwise I need to implement some other algorithm "Data Consistency" here.
If any of you guys have any solution or idea, kindly share.
Thanks,
Sanjay Mohnani
In a cluster with only master nodes, if a node fails, data is lost. Therefore no resharding is possible, since it is not possible to migrate the data (hash slots) out of the failed node.
To keep the cluster working when a master fails, you need slave nodes (one per master). This way, when a master fails, its slave fails over (becomes the new master with the same copy of the data).
The redis-trib.rb script does not handle cluster creation with less than 3 masters, however in redis-cluster a cluster can be of any size (at least one node).
Therefore adding slave nodes can be considered an automatic solution to your problem.
RabbitMQ clusters need to have at least one disc node (you can't turn your last disc node to a ram node).
However (especially in a cloud context) nodes can die - what is supposed to happen to the cluster if the only disc node dies?
Does the cluster automatically appoint a new disc node, or it continues working with no disc node.
Short answer: In case all disc nodes dies and you have at least one RAM node you'll get RAM-only cluster. In case only one RAM node left and it goes down and then up, only durable entities will reside on it.
Long answer:
If you use clustering as it described in Clustering Guide queues reside only on one node:
All data/state required for the operation of a RabbitMQ broker is
replicated across all nodes, for reliability and scaling, with full
ACID properties. An exception to this are message queues, which by
default reside on the node that created them, though they are visible
and reachable from all nodes. To replicate queues across nodes in a
cluster, see the documentation on high availability (note that you
will need a working cluster first).
So when node dies (not only disc one, it applied to RAM too) you lose queues (with content) resides on that node.
If you use High Availability to mirror queue across more than one nodes (actually, it depends how you set it up, see detailed explanation on ha-mode and ha-policy policy keys - all, exactly and nodes).
With HA, if queue has some ha-policy set and the node it reside dies, that queue will be tried to be mirrored to other nodes, including RAM-only one (sure, it depends how you set up ha-mode, for example if it set to nodes and all nodes from list dies you lose the queue).
So after such intro,
If you turn off all disc nodes and you have only RAM nodes and queues fits the memory everything will works normally. If queues doesn't fit in memory, Flow Control memory limits applied which explained in clustering doc in Restarting section (at the end of e:
At least one disk node should be running at all times to prevent data
loss. RabbitMQ will prevent the creation of a RAM-only cluster in many
situations, but it still won't stop you from stopping and forcefully
resetting all the disc nodes, which will lead to a RAM-only cluster.
Doing this is not advisable and makes losing data very easy.
and a bit more from clustering doc:
A node can be a disk node or a RAM node. (Note: disk and disc are
used interchangeably. Configuration syntax or status messages normally
use disc.) RAM nodes keep their state only in memory (with the
exception of queue contents, which can reside on disc if the queue is
persistent or too big to fit in memory). Disk nodes keep state in
memory and on disk. As RAM nodes don't have to write to disk as much
as disk nodes, they can perform better. However, note that since the
queue data is always stored on disc, the performance improvements will
affect only resources management (e.g. adding/removing queues,
exchanges, or vhosts), but not publishing or consuming speed. Because
state is replicated across all nodes in the cluster, it is sufficient
(but not recommended) to have just one disk node within a cluster, to
store the state of the cluster safely.
So if you don't literally add any disc node you'll get RAM-only cluster. It may be fast in some cases, but if all nodes goes down you will lose all your queues with it content forever, except durable ones while any node dump persistent queues and messages on disc.
But don't rely on RAM node dump persistent entities on disc, while under certain situations it may not dump at all or not all entities (especially, messages).
There are old mailing list threads which may bring some extra light on situation:
Cluster with all memory nodes
Cluster Disk Node vs Ram Node explanation
I know I'm asking something very obvious about cluster failover.
I read on redis.io that, if any master cluster node fails it will affect to other master nodes until slave come to take in charge. In my structure, I'm not defining any slave and just working with 3 masters.
I'm thinking to modify the redis-trib.rb file, which will remove the defected server and will start the cluster with other 2 nodes. I'm confused about a couple of things,
1) Resharding
Could not possible until failed server goes live
2) Minimum 3 node limitation for create cluster
As per bit understanding, redis-trib.rb not allowing me to create cluster for two nodes
There might be some solution in code file :)
3) Automatic Way to Re-Create new structure with live nodes
As programmer point of view, I'm searching something automatic for my system. Something that trigger one command when Redis Cluster fails some tasks happens internally. like
Shutdown all other redis cluster servers
Remove nodes-[port].conf files from all cluster nodes folder
Start redis cluster servers
Run "redis-trib.rb create ip:port ip:port"
I'm just trying to minimize administration work :). Otherwise I need to implement some other algorithm "Data Consistency" here.
If any of you guys have any solution or idea, kindly share.
Thanks,
Sanjay Mohnani
In a cluster with only master nodes, if a node fails, data is lost. Therefore no resharding is possible, since it is not possible to migrate the data (hash slots) out of the failed node.
To keep the cluster working when a master fails, you need slave nodes (one per master). This way, when a master fails, its slave fails over (becomes the new master with the same copy of the data).
The redis-trib.rb script does not handle cluster creation with less than 3 masters, however in redis-cluster a cluster can be of any size (at least one node).
Therefore adding slave nodes can be considered an automatic solution to your problem.