I want to delete a master node from existing Redis Cluster. I tried following the instructions from http://redis.io/topics/cluster-tutorial but unable to follow. Help??
As per previous answer, to delete a master node it must be empty. That means we've to flushall the data or migrate hash slot to another existing master.
But as redis reconfiguration and redis cluster resharding says, additional steps may need to be added after step 5 in the previous answer.
A solution that works for me is as following according to redis official docs. The order is important here.
We'll call the node that has the current ownership of the hash slot the source node, and the node where we want to migrate the destination node.
For each slot:
Send CLUSTER SETSLOT <slot> IMPORTING <source-node-id> to destination node to set the slot to importing state.
Send CLUSTER SETSLOT <slot> MIGRATING <destination-node-id> to source node to set the slot to migrating state.
Get keys from the source node with CLUSTER GETKEYSINSLOT command and move them into the destination node using the following MIGRATE command.
MIGRATE target_host target_port key target_database_id timeout
In Redis Cluster there is no need to specify a database other than 0, but MIGRATE is a general command that can be used for other tasks not involving Redis Cluster.
When the migration process is finally finished, use CLUSTER SETSLOT <slot> NODE <destination-node-id> in both source node and destination node in order to set the slot to their normal state again. The same command is usually sent to all other nodes to avoid waiting for the natural propagation of the new configuration across the cluster.
A simple example bash script to do this is also given here:
source-ip: 172.17.0.5. source-id: 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
destination-ip: 172.17.0.4. destination-id: 7e428bae84697a3882ecad19bd0d13ac7ee97d02
another master ip: 172.17.0.7
for i in `seq 0 5460`; do
redis-cli -c -h 172.17.0.4 cluster setslot ${i} importing 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
redis-cli -c -h 172.17.0.5 cluster setslot ${i} migrating 7e428bae84697a3882ecad19bd0d13ac7ee97d02
while true; do
key=`redis-cli -c -h 172.17.0.5 cluster getkeysinslot ${i} 1`
if [ "" = "$key" ]; then
echo "there are no key in this slot ${i}"
break
fi
redis-cli -h 172.17.0.5 migrate 172.17.0.4 6379 ${key} 0 5000
done
redis-cli -c -h 172.17.0.5 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.4 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.7 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
done
Node should be empty before deleting it. As such, we need to move the data and also slots before deleting a node.
Follow below steps to migrate:
Connect to destination node:
cluster setslot (slot no) importing (source node id)
Connect to source node:
cluster setslot (slot no) migrating (destination node id)
migrate (destination ip) (destination port) key (destination db) (timeout)
Connect to destination node:
cluster setslot (slot no) node (dest node)
We can do the above in a loop for all the slots and keys and then delete the source node.
Related
When trying to create a cluster with redis-cli as follows
redis-cli --cluster create
a prompt comes up asking for configuration confirmation?
Is there a way to script this (preferably in ansible) and run it non-interactively?
I am aware of this topic however it addresses data manipulation which is not the scope of this question.
--cluster-yes is the correct option!
EDIT:
The answer of Leonardo Oliveira rightfully points out that the option --cluster-yes will avoid the prompt. For example:
redis-cli --cluster create host1:6379 host2:6379 host3:6379 --cluster-yes
As of the current Redis version (5.0.5) there doesn't seem to be a flag under --cluster that can silence or auto-answer the interactive question:
$ redis-cli --cluster help
Cluster Manager Commands:
create host1:port1 ... hostN:portN
--cluster-replicas <arg>
check host:port
--cluster-search-multiple-owners
info host:port
fix host:port
--cluster-search-multiple-owners
reshard host:port
--cluster-from <arg>
--cluster-to <arg>
--cluster-slots <arg>
--cluster-yes
--cluster-timeout <arg>
--cluster-pipeline <arg>
--cluster-replace
rebalance host:port
--cluster-weight <node1=w1...nodeN=wN>
--cluster-use-empty-masters
--cluster-timeout <arg>
--cluster-simulate
--cluster-pipeline <arg>
--cluster-threshold <arg>
--cluster-replace
add-node new_host:new_port existing_host:existing_port
--cluster-slave
--cluster-master-id <arg>
del-node host:port node_id
call host:port command arg arg .. arg
set-timeout host:port milliseconds
import host:port
--cluster-from <arg>
--cluster-copy
--cluster-replace
help
By using echo you can execute the command and auto-answer the prompt:
echo "yes" | redis-cli --cluster create host1:6379 host2:6379 host3:6379
The default Ansible Redis module only supports a few commands and not --cluster so you would have to create your own logic with command/shell tasks:
- name: Create cluster
shell: echo "yes" | redis-cli --cluster create host1:6379 host2:6379 host3:6379
run_once: true
when: not cluster_setup_done
Well, I don't know about the ansible part. But Redis official site does provide a way to create cluster with script in a little interactive-mode.
Creating a Redis Cluster using the create-cluster script (Please refer docs for more details)
If you don't want to create a Redis Cluster by configuring and executing individual instances manually as explained above, there is a much simpler system (but you'll not learn the same amount of operational details).
Just check utils/create-cluster directory in the Redis distribution. There is a script called create-cluster inside (same name as the directory it is contained into), it's a simple bash script. In order to start a 6 nodes cluster with 3 masters and 3 slaves just type the following commands:
create-cluster start
create-cluster create
Reply to yes in step 2 when the redis-cli utility wants you to accept the cluster layout.
You can now interact with the cluster, the first node will start at port 30001 by default. When you are done, stop the cluster with:
create-cluster stop
Please read the README inside this directory for more information on how to run the script.
You can try implementing this if it helps you in any way!
Cheers :)
We're using redis-cluster extensively in our production env. We currently have a 30 node cluster (15 masters, 15 slaves)
We're trying to increase the cluster, for that we've created new servers & joined them to the cluster. so far all is well.
Next - we're trying to reshard the slots to the new masters. we wrote a script that does this, using the redis-trib reshard command.
However - the migration fails midway (but not very far from the start) with this error:
[ERR] Calling MIGRATE: ERR Target instance replied with error: BUSYKEY Target key name already exists.
This happens sporadically, at times it manages to move some slots before failing, at times it fails on the first slot.
Each such failure requires a manual fixing operation which makes the reshard operation very hard to manage.
We have not found any concrete example of this, nor any idea on how to prevent this other than a downtime migration. which we are trying to avoid.
Versions:
redis server 4.0.2
redis trib 3.3.3 (downgraded from 4.0.2 following this issue : redis cluster reshard [ERR] Calling MIGRATE: ERR Syntax error)
Our next step is to upgrade to latest redis (4.0.11), even though we didn't find any indication in the release notes of this issue.
Hoping to hear we're doing something wrong and how to fix it, or is redis-cluster not built for live resharding ?
Thanks
I have faced like this problem while working with redis-clustering support for our own project. I found a problem with the redis-trib reshard command. It works fine if no key is stored in slots those are migrating from one master to another.
But redis-5 (still developing, not stable yet) has it's own
`redis-cli' that has no problem with resharding command I think. Only for lower versions of 5 it happens.
If you look at the official docs for redis say redis reconfiguration and redis cluster resharding, you'll find what they do internally to reshard.
So I solved the problem by doing those tasks by running a bash script instead of running redis-trib reshard command.
Suppose you want to reshard some slots from a master node to other master node. We'll call the node that has the current ownership of the hash slot the source node, and the node where we want to migrate the destination node.
For each slot do the following steps:
Remember that the order of these steps is important here according to redis official docs.
Send CLUSTER SETSLOT <slot> IMPORTING <source-node-id> to destination node to set the slot to importing state.
Send CLUSTER SETSLOT <slot> MIGRATING <destination-node-id> to source node to set the slot to migrating state.
Get keys from the source node with CLUSTER GETKEYSINSLOT command and move them into the destination node using the following MIGRATE command.
MIGRATE target_host target_port key target_database_id timeout
In Redis Cluster there is no need to specify a database other than 0, but MIGRATE is a general command that can be used for other tasks not involving Redis Cluster.
When the migration process is finally finished, use CLUSTER SETSLOT <slot> NODE <destination-node-id> in both source node and destination node in order to set the slot to their normal state again. The same command is usually sent to all other nodes to avoid waiting for the natural propagation of the new configuration across the cluster.
A simple example bash script to do this is also given here:
source-ip: 172.17.0.5. source-id: 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
destination-ip: 172.17.0.4. destination-id: 7e428bae84697a3882ecad19bd0d13ac7ee97d02
another master ip: 172.17.0.7
for i in `seq 0 5460`; do
redis-cli -c -h 172.17.0.4 cluster setslot ${i} importing 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
redis-cli -c -h 172.17.0.5 cluster setslot ${i} migrating 7e428bae84697a3882ecad19bd0d13ac7ee97d02
while true; do
key=`redis-cli -c -h 172.17.0.5 cluster getkeysinslot ${i} 1`
if [ "" = "$key" ]; then
echo "there are no key in this slot ${i}"
break
fi
redis-cli -h 172.17.0.5 migrate 172.17.0.4 6379 ${key} 0 5000
done
redis-cli -c -h 172.17.0.5 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.4 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.7 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
done
I have installed redis cluster 3.0.0. But Want to upgrade it to 3.0.7. Can somebody tell me the steps to do it?
I don't want to loose any data. And don't want any downtime either.
Steps I did when upgrading from 2.9.101 to 3.0 release. I hope it will do for upgrading to 3.0.7 too.
Compile 3.0.7 from the source and start several instances with cluster enabled.
Let the 3.0.7 instances replicate the 3.0.0 instances as slave
Connect to each 3.0.7 instance and do a manual failover, then the 3.0.0 masters would become slaves after several seconds.
Wait for your application to connect to the new masters; also check the configuration files, and modify the entries to the new masters on your need
Remove those slaves
UPDATE : Docker approach
As it's probably unable to replacing the binary executable while the process is still alive, you could do it by run some Redis in docker.
First you should install docker on your machine and pull the Redis image, or pull a basic OS image and manually build Redis in it, whatever
Based on this image, you are supposed to
copy your current redis.conf into it
make sure the dir exists in the image (cluster-config-file could be the same for all the containers as they are saved individually in their own fs)
make sure the directory for logfile exists and is not the same as dir (we will later map this directory to the host)
leave port logfile anything you like, as they are specified when a container is started
commit the image as redis-3.0.7
Now launch a containerized Redis. I suppose your logfile is located in /var/log/redis/, this Redis binds :8000, and your config file in the image is /etc/redis/redis.conf
docker run -d --net=host -v /var/log/redis:/var/log/redis \
-p 8000:8000 -t redis-3.0.7 \
/usr/bin/redis-server /etc/redis/redis.conf \
--port 8000 \
--logfile /var/log/redis/redis_8000.log
Now you have a Redis 3.0.7 instance, and are ready to finish the rest steps in the previous part.
I google it and found two solution:
CLUSTER FORGET (http://redis.io/commands/cluster-forget)
redis-trib.rb del-node
I think CLUSTER FORGET" is the right way to do.
But I really want to know the details about redis-trib.rb del-node.
Can someone explain the difference between them?
redis-trib.rb is a ruby utility script that antirez (lead redis developer) built as a reference implementation of building administrative tools on top of the basic redis cluster commands.
Under the hood redis-trib uses CLUSTER FORGET to implement it's own administrative del-node command. https://github.com/antirez/redis/blob/unstable/src/redis-trib.rb#L1374
Redis-trib is a lot friendlier to work with. If you're doing CLUSTER FORGET you'd need to loop over and send that command to every other node in the system, while del-node will automate that process for you.
As of Redis 6.2.3
WARNING: redis-trib.rb is not longer available!
You should use redis-cli instead.
All commands and features belonging to redis-trib.rb have been moved
to redis-cli.
In order to use them you should call redis-cli with the --cluster
option followed by the subcommand name, arguments and options.
Use the following syntax:
redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
Example:
redis-cli --cluster info 127.0.0.1:6382
~$ redis-cli
127.0.0.1:6379> CLUSTER HELP
127.0.0.1:6379> CLUSTER NODES
127.0.0.1:6379> CLUSTER FORGET <node-id>
src/redis-trib.rb del-node 192.168.0.211:6379 650e3746968e6b7c7e357f06adbde5b3b92fcceb
Note:
192.168.0.211:6379 This is any node in the cluster
650e3746968e6b7c7e357f06adbde5b3b92fcceb this is the cluster ID of the node you want to remove. You can get the value of this ID from “cluster nodes” command.
Per redis article you should use:
redis-cli --cluster del-node 127.0.0.1:7000
where the the first argument is one of your cluster nodes.
Check 'removing a node' in he following article:
https://redis.io/docs/manual/scaling/
I'm trying to create a Redis Docker image, based on the redis:3. However, I need to load a small amount of test data into the image (it is used for testing, both on developer machines and on the CI server).
My initial attempt looks like:
FROM redis:3.0.3
MAINTAINER Howard M. Lewis Ship
RUN redis-cli sadd production-stores 5555
But this fails with the error:
Step 2 : RUN redis-cli sadd production-stores 5555
---> Running in 60fb98c133c0
Could not connect to Redis at 127.0.0.1:6379: Connection refused
The command '/bin/sh -c redis-cli sadd production-stores 5555' returned a non-zero code: 1
I'm wondering if there's a trick that allows me, in a Docker file, to connect to the server started via the CMD/ENTRYPOINT of the Dockerfile. As I'm guessing right now, the RUN occurs before the command is started.
Alternately, is there a Redis command or trick that would allow me to load some data into its database.
I suspect one approach would be to mount the Redis' data directory in a volume exposed to the Docker host; this is not ideal as
I would need an extra non-Dockerfile command to load the data
The loaded data would not be shared in the image, which is to be used by others on my team
I'm on OS X and am using docker-machine, so the volumes get trickier
The error tells Redis service is not started before you run redis-cli.
Add a new line to start it first. Let me know if this can fix your issue or not.
FROM redis:3.0.3
MAINTAINER Howard M. Lewis Ship
RUN /usr/sbin/redis-server /etc/redis.conf
RUN redis-cli sadd production-stores 5555
Updates:
Review the official redis image, it has given the way to use redis-cli
# via redis-cli
$ docker run -it --link some-redis:redis --rm redis sh -c 'exec redis-cli -h "$REDIS_PORT_6379_TCP_ADDR" -p "$REDIS_PORT_6379_TCP_PORT"'
So I test the image and make it run as below:
# start redis container
$ docker run --name some-redis -d redis
# link redis container and run redis-cli command
$ docker run -it --link some-redis:redis --rm redis redis-cli -h redis
redis:6379> PING
PONG
redis:6379> exit
# Add the specified members to the set stored at key
$ docker run -it --link some-redis:redis --rm redis redis-cli -h redis sadd production-stores 5555
(integer) 1
$