I'm trying to create a Redis cluster using an RDB file taken from a single-instance Redis server. Here is what I've tried:
#! /usr/bin/env bash
for i in 6000 6001 6002 6003
do
redis-server --port $i --cluster-config-file "node-$i.cconf" --cluster-enabled yes --dir "$(pwd)" --dbfilename dump.rdb &
done
That script starts up 4 Redis processes that are cluster enabled. It also initializes each node with the dump file.
Then I run redis-trib.rb so that the 4 nodes can find each other:
redis-trib.rb create 127.0.0.1:6000 127.0.0.1:6001 127.0.0.1:6002 127.0.0.1:6003
I get the following error:
>>> Creating cluster
[ERR] Node 127.0.0.1:6060 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
I've also tried a variant where only the first node/process is initialized with the RDB file and the others are empty. I can join the 3 empty nodes into a cluster but not the one that's pre-populated with data.
What is the correct way to import a pre-existing RDB file into a Redis cluster?
In case this is an "X-Y problem" this is why I'm doing this:
I'm working on a migration from a single-instance Redis server to an Elasticache Redis Cluster (cluster mode enabled). Elasticache can easily import an RDB file on cluster startup if you upload it to S3. But it takes a long time for an Elasticache cluster to start. To reduce the feedback loop as I test my migration code, I'd like to be able to start a cluster locally also.
I've also tried using the create-cluster utility, but that doesn't appear to have any options to pre-populate the cluster with data.
Related
I have 5 redis server
2 of them run redis both Master and Slave roles ( looks like redis.conf is not setup manually but via some sort of process cause it has the following line at the bottom: Generated by CONFIG REWRITE )
From time to time I can see Master and Slave switch roles automatically - no human intervention
3 of them run redis sentinel
Question 1: I need to replicate this setup on a 5 different systems but I don’t know how is that “Generated by CONFIG REWRITE” portion setup. Where and how is this automation setup?
Question 2: Why is that /etc/redis/ has a 6329.conf file? I thought redis setup is redis.conf...
Thanks
The Config Rewrites are all caused by Redis Sentinel. The 3 sentinels you have monitor the master and in the event that enough sentinels think the master is down, they will force a failover by promoting an existing slave to the new master, then will reconfigure all other hosts to be a slave of the new master. You can read more about Redis Sentinel, including how to set it up for common scenarios, (docs page, examples section).
For the 6329.conf file, you can name the config files however you want, but however you start your redis server has to reference the non-default file name. Here's the usage example from the --help option to redis-server:
Usage: ./redis-server [/path/to/redis.conf] [options]
I want to delete all keys from redis cluster by using fast process. I know how to delete the keys by using "redis-cli FLUSHALL". But this command can be slow when the data set is large. I heard that all keys can be cleared from redis cache by re-starting the redis service. I am testing this process on my local mac laptop. I am performing following steps:-
Setting many number of keys on my local redis server by using example command redis-cli SET mykey1 "Hello"
Then Re-starting the redis service "brew services restart redis" in the hope that all keys will be deleted when the service will be back up
Then getting the keys by giving "redis-cli KEYS '*'" command
I still see the keys after step-3
The keys are gone only when I give this command--> redis-cli FLUSHALL? How I can clear the keys by re-starting the redis service locally on my mac laptop first then I will try on QA servers?
You see the keys after restart because there is either RDB or AOF persistence enabled. See https://redis.io/topics/persistence.
RDB is enabled by default. To disable persistence, you need to edit your redis.conf or start as redis-server --save "" --appendonly no
See Is there a way to flushall on a cluster so all keys from master and slaves are deleted from the db on how to use redis-cli to send the command to all cluster nodes.
As dizzyf indicates, use FLUSHALL ASYNC to have the deletion performed in the background. This will create fresh hash maps for each database, while the old ones are deleted (memory reclaimed) progressively by a background thread.
In redis 4.0 and greater, the FLUSHALL ASYNC command was introduced, as a way to delete all the keys in a non-blocking manner. Would this solve your issue?
https://redis.io/commands/flushall
Thank you for links. These were very helpful. I was able to achieve the result by making changes to my redis.conf file with--> redis-server --save "" and --appendonly no. So after these changes, when I now re-start redis service nothing is saved.
I have a Redis dump.rdb file of 177MB in size. It is a snapshot dump of one of the instances of a testing Redis cluster. When I restart the instance, it takes about 25 seconds to load the file into memory. But when I starts a standalone Redis server with the same dump, it only takes about 3.7 seconds to load.
The questions are:
Does any one know what causes the difference?
Is there a way to shorten the db loading time for the Redis cluster instance?
Suppose I have [Slave IP Address] which is the slave of [Master IP Address].
Now my master server has been shut down, and I need to set this slave to be master MANUALLY (WITHOUT using sentinel automatic failover, WITH redis command).
Is it possible doing this without restarting the redis service ? (and losing all the cached data)
use SLAVEOF NO ONE to promote a slave to master
http://redis.io/commands/slaveof
it depends, if you are in a cluster you will be better using the fail over. You will need to use the force option in the command
http://redis.io/commands/cluster-failover
Is it possible doing this without restarting the redis service? (and
losing all the cached data)
yes that's possible, you can use
SLAVEOF NO ONE (without sentinel)
But it is recommended to use sentinel to avoid data loss.
sentinel failover master-name(with sentinel)
This will force the sentinel to switch master.
The new master will have all the data that was synchronized before the old-master shutdown.
Redis will automatically choose the best slave with max. data, that will reduce the amount of data we lose when switching master.
Below 2 options in step 3 have helped me to recover the cluster once a master node is down, compute was replaced or other not recoverable state.
1 .- First you need to connect to the slave node, use redis-cli, here a link how to do that: How to connect to remote Redis server?
2 .- Once connected to the slave node run the command cluster nodes to validate master node is in fail state, also run cluster info to see the overall state of your cluster(this is always a good idea)
3 .- Inside the slave node to be promoted run command: cluster failover,
in rare cases when there is some serious issues with redis this
command could fail, and you will need to use cluster failover force
or cluster failover takeover, here more info abut the implications
of those options: https://redis.io/commands/cluster-failover
4 .- Run cluster forged $old_master_id in all your cluster nodes
5 .- Add a new node with cluster meet $new_node_IP $new_node_PORT
6 .- Subscribe your new node to your brand new master, login in to the new bode and run cluster replicate $master_node_id
Steps 1-3 are required for the slave-master promotion and 4-5 are required to left all cluster in a healthy master-slave equilibrium.
As of Redis version 5.0.0 the SLAVEOF command is regarded as deprecated.
If a Redis server is already acting as replica, the command REPLICAOF NO ONE will turn off the replication, turning the Redis server into a MASTER.
I am trying to do data replication using redis server and jedis client. What I am asking may be completely wrong. I just started exploring the things today.
I have configured 2 laptops in my network.
Config on node1 is default the config.
Config on node2:
updated redis.conf file to have this line:
slaveof <node1 ip> <port on which redis server is running>
I have a stand-alone java program on both of the nodes.
Case 1:
I did jedis.set("key1", "value1"); from node1 (master). I am able to retrieve key-value from node2.
Case 2:
I did jedis.set("key2", "value2"); from node2 (slave). But I'm not able to see that replicated on the master node.
Question is, is it expected behavior? My requirement is to keep the data in sync on all the nodes.
Thanks.
That is the expected behavior. Writes to the master will propagate to the slave(s), writes to the slave(s) will not propagate to the master. If you want to use Redis, you will have to write to the master and read from the slave(s). See Redis replication documentation.