I have compared the output of "redis-cli info" between redis v3.0.3 and v5.0.5 and I see that many new values are in redis v5.0.5 but the following two are missing:
client_longest_output_list
client_biggest_input_buf
Are these values removed from redis 5 or just renamed to the following two, which are also new in redis v5.0.5 compared to v3.0.3?
client_recent_max_input_buffer
client_recent_max_output_buffer
These have been indeed renamed in v5 - see here https://github.com/antirez/redis/commit/be88c0b16a53f5763d8fc1ae683f99ee39b0d68e
Related
I've looked "everywhere." I cannot find documentation for all the supported command line options for redis-server I'm using version 5.0.3
I tried redis-server --help It is no help.
The usage given doesn't even mention --port, --slaveof, --replicaof, --loglevel ... yet these options are shown in the help's examples.
Does someone know where I can find full and complete documentation for the server's command line?
Thanks.
Right at the top of the redis configuration documents it says the following:
"... it is possible to ... pass Redis configuration parameters using
the command line directly."
Therefore, every configuration file option is passable on the command line. I'm an idiot.
Edit: Note that config file parameters which have spaces in them will not work as a command line parameter. For example, --save "600 1 30 10 6 100" will not be used. Running redis-cli followed by config get save will show "". Doesn't matter if the parameter is placed at the end of the command line. Doesn't matter if it is enclosed with single quotes, double quotes, or no quotes.
redis-server command line does not parse params with spaces correctly. The issue is known and closed without being resolved:
https://github.com/redis/redis/issues/2366
The most useful information about configuring redis-server is at https://redis.io/docs/manual/config/
Passing arguments via the command line
You can also pass Redis configuration parameters using the command line directly. This is very useful for testing purposes. The following is an example that starts a new Redis instance using port 6380 as a replica of the instance running at 127.0.0.1 port 6379.
./redis-server --port 6380 --replicaof 127.0.0.1 6379
The format of the arguments passed via the command line is exactly the same as the one used in the redis.conf file, with the exception that the keyword is prefixed with --.
Note that internally this generates an in-memory temporary config file (possibly concatenating the config file passed by the user, if any) where arguments are translated into the format of redis.conf.
The .conf file with all the params has reasonably useful inline docs.
man redis-server and redis-server -h are basically useless.
man redis-server:
REDIS-SERVER(1) General Commands Manual REDIS-SERVER(1)
NAME
redis-server - Persistent key-value database
SYNOPSIS
redis-server configfile
DESCRIPTION
Redis is a key-value database. It is similar to memcached but the dataset is not volatile and other
datatypes (such as lists and sets) are natively supported.
OPTIONS
configfile
Read options from specified configuration file.
NOTES
On Debian GNU/Linux systems, redis-server is typically started via the /etc/init.d/redis-server initscript,
not manually. This defaults to using /etc/redis/redis.conf as a configuration file.
AUTHOR
redis-server was written by Salvatore Sanfilippo.
This manual page was written by Chris Lamb <lamby#debian.org> for the Debian project (but may be used by
others).
March 20, 2009 REDIS-SERVER(1)
`redis-server -h`:
Usage: ./redis-server [/path/to/redis.conf] [options] [-]
./redis-server - (read config from stdin)
./redis-server -v or --version
./redis-server -h or --help
./redis-server --test-memory <megabytes>
./redis-server --check-system
Examples:
./redis-server (run the server with default conf)
echo 'maxmemory 128mb' | ./redis-server -
./redis-server /etc/redis/6379.conf
./redis-server --port 7777
./redis-server --port 7777 --replicaof 127.0.0.1 8888
./redis-server /etc/myredis.conf --loglevel verbose -
./redis-server /etc/myredis.conf --loglevel verbose
Sentinel mode:
./redis-server /etc/sentinel.conf --sentinel
Also, if someone knows how to tuck the man and -h snippets of this answer into <details> with SO markup, please edit this response, thanks.
We're using redis-cluster extensively in our production env. We currently have a 30 node cluster (15 masters, 15 slaves)
We're trying to increase the cluster, for that we've created new servers & joined them to the cluster. so far all is well.
Next - we're trying to reshard the slots to the new masters. we wrote a script that does this, using the redis-trib reshard command.
However - the migration fails midway (but not very far from the start) with this error:
[ERR] Calling MIGRATE: ERR Target instance replied with error: BUSYKEY Target key name already exists.
This happens sporadically, at times it manages to move some slots before failing, at times it fails on the first slot.
Each such failure requires a manual fixing operation which makes the reshard operation very hard to manage.
We have not found any concrete example of this, nor any idea on how to prevent this other than a downtime migration. which we are trying to avoid.
Versions:
redis server 4.0.2
redis trib 3.3.3 (downgraded from 4.0.2 following this issue : redis cluster reshard [ERR] Calling MIGRATE: ERR Syntax error)
Our next step is to upgrade to latest redis (4.0.11), even though we didn't find any indication in the release notes of this issue.
Hoping to hear we're doing something wrong and how to fix it, or is redis-cluster not built for live resharding ?
Thanks
I have faced like this problem while working with redis-clustering support for our own project. I found a problem with the redis-trib reshard command. It works fine if no key is stored in slots those are migrating from one master to another.
But redis-5 (still developing, not stable yet) has it's own
`redis-cli' that has no problem with resharding command I think. Only for lower versions of 5 it happens.
If you look at the official docs for redis say redis reconfiguration and redis cluster resharding, you'll find what they do internally to reshard.
So I solved the problem by doing those tasks by running a bash script instead of running redis-trib reshard command.
Suppose you want to reshard some slots from a master node to other master node. We'll call the node that has the current ownership of the hash slot the source node, and the node where we want to migrate the destination node.
For each slot do the following steps:
Remember that the order of these steps is important here according to redis official docs.
Send CLUSTER SETSLOT <slot> IMPORTING <source-node-id> to destination node to set the slot to importing state.
Send CLUSTER SETSLOT <slot> MIGRATING <destination-node-id> to source node to set the slot to migrating state.
Get keys from the source node with CLUSTER GETKEYSINSLOT command and move them into the destination node using the following MIGRATE command.
MIGRATE target_host target_port key target_database_id timeout
In Redis Cluster there is no need to specify a database other than 0, but MIGRATE is a general command that can be used for other tasks not involving Redis Cluster.
When the migration process is finally finished, use CLUSTER SETSLOT <slot> NODE <destination-node-id> in both source node and destination node in order to set the slot to their normal state again. The same command is usually sent to all other nodes to avoid waiting for the natural propagation of the new configuration across the cluster.
A simple example bash script to do this is also given here:
source-ip: 172.17.0.5. source-id: 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
destination-ip: 172.17.0.4. destination-id: 7e428bae84697a3882ecad19bd0d13ac7ee97d02
another master ip: 172.17.0.7
for i in `seq 0 5460`; do
redis-cli -c -h 172.17.0.4 cluster setslot ${i} importing 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
redis-cli -c -h 172.17.0.5 cluster setslot ${i} migrating 7e428bae84697a3882ecad19bd0d13ac7ee97d02
while true; do
key=`redis-cli -c -h 172.17.0.5 cluster getkeysinslot ${i} 1`
if [ "" = "$key" ]; then
echo "there are no key in this slot ${i}"
break
fi
redis-cli -h 172.17.0.5 migrate 172.17.0.4 6379 ${key} 0 5000
done
redis-cli -c -h 172.17.0.5 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.4 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.7 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
done
I have installed redis cluster 3.0.0. But Want to upgrade it to 3.0.7. Can somebody tell me the steps to do it?
I don't want to loose any data. And don't want any downtime either.
Steps I did when upgrading from 2.9.101 to 3.0 release. I hope it will do for upgrading to 3.0.7 too.
Compile 3.0.7 from the source and start several instances with cluster enabled.
Let the 3.0.7 instances replicate the 3.0.0 instances as slave
Connect to each 3.0.7 instance and do a manual failover, then the 3.0.0 masters would become slaves after several seconds.
Wait for your application to connect to the new masters; also check the configuration files, and modify the entries to the new masters on your need
Remove those slaves
UPDATE : Docker approach
As it's probably unable to replacing the binary executable while the process is still alive, you could do it by run some Redis in docker.
First you should install docker on your machine and pull the Redis image, or pull a basic OS image and manually build Redis in it, whatever
Based on this image, you are supposed to
copy your current redis.conf into it
make sure the dir exists in the image (cluster-config-file could be the same for all the containers as they are saved individually in their own fs)
make sure the directory for logfile exists and is not the same as dir (we will later map this directory to the host)
leave port logfile anything you like, as they are specified when a container is started
commit the image as redis-3.0.7
Now launch a containerized Redis. I suppose your logfile is located in /var/log/redis/, this Redis binds :8000, and your config file in the image is /etc/redis/redis.conf
docker run -d --net=host -v /var/log/redis:/var/log/redis \
-p 8000:8000 -t redis-3.0.7 \
/usr/bin/redis-server /etc/redis/redis.conf \
--port 8000 \
--logfile /var/log/redis/redis_8000.log
Now you have a Redis 3.0.7 instance, and are ready to finish the rest steps in the previous part.
I google it and found two solution:
CLUSTER FORGET (http://redis.io/commands/cluster-forget)
redis-trib.rb del-node
I think CLUSTER FORGET" is the right way to do.
But I really want to know the details about redis-trib.rb del-node.
Can someone explain the difference between them?
redis-trib.rb is a ruby utility script that antirez (lead redis developer) built as a reference implementation of building administrative tools on top of the basic redis cluster commands.
Under the hood redis-trib uses CLUSTER FORGET to implement it's own administrative del-node command. https://github.com/antirez/redis/blob/unstable/src/redis-trib.rb#L1374
Redis-trib is a lot friendlier to work with. If you're doing CLUSTER FORGET you'd need to loop over and send that command to every other node in the system, while del-node will automate that process for you.
As of Redis 6.2.3
WARNING: redis-trib.rb is not longer available!
You should use redis-cli instead.
All commands and features belonging to redis-trib.rb have been moved
to redis-cli.
In order to use them you should call redis-cli with the --cluster
option followed by the subcommand name, arguments and options.
Use the following syntax:
redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
Example:
redis-cli --cluster info 127.0.0.1:6382
~$ redis-cli
127.0.0.1:6379> CLUSTER HELP
127.0.0.1:6379> CLUSTER NODES
127.0.0.1:6379> CLUSTER FORGET <node-id>
src/redis-trib.rb del-node 192.168.0.211:6379 650e3746968e6b7c7e357f06adbde5b3b92fcceb
Note:
192.168.0.211:6379 This is any node in the cluster
650e3746968e6b7c7e357f06adbde5b3b92fcceb this is the cluster ID of the node you want to remove. You can get the value of this ID from “cluster nodes” command.
Per redis article you should use:
redis-cli --cluster del-node 127.0.0.1:7000
where the the first argument is one of your cluster nodes.
Check 'removing a node' in he following article:
https://redis.io/docs/manual/scaling/
I was wondering if anybody could help me with this issue in deploying a spark cluster using the bdutil tool.
When the total number of cores increase (>= 1024), it failed all the time with the following reasons:
Some machine is never sshable, like "Tue Dec 8 13:45:14 PST 2015: 'hadoop-w-5' not yet sshable (255); sleeping"
Some nodes fail with an "Exited 100" error when deploying spark worker nodes, like "Tue Dec 8 15:28:31 PST 2015: Exited 100 : gcloud --project=cs-bwamem --quiet --verbosity=info compute ssh hadoop-w-6 --command=sudo su -l -c "cd ${PWD} && ./deploy-core-setup.sh" 2>>deploy-core-setup_deploy.stderr 1>>deploy-core-setup_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-f"
In the log file, it says:
hadoop-w-40: ==> deploy-core-setup_deploy.stderr <==
hadoop-w-40: dpkg-query: package 'openjdk-7-jdk' is not installed and no information is available
hadoop-w-40: Use dpkg --info (= dpkg-deb --info) to examine archive files,
hadoop-w-40: and dpkg --contents (= dpkg-deb --contents) to list their contents.
hadoop-w-40: Failed to fetch http://httpredir.debian.org/debian/pool/main/x/xml-core/xml-core_0.13+nmu2_all.deb Error reading from server. Remote end closed connection [IP: 128.31.0.66 80]
hadoop-w-40: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
I tried 16-core 128-nodes, 32-core 64-nodes, 32-core 32-nodes and other over 1024-core configurations, but either the above Reason 1 or 2 will show up.
I also tried to modify the ssh-flag to change the ConnectTimeout to 1200s, and change bdutil_env.sh to set the polling interval to 30s, 60s, ..., none of them works. There will be always some nodes which fail.
Here is one of the configurations that I used:
time ./bdutil \
--bucket $BUCKET \
--force \
--machine_type n1-highmem-32 \
--master_machine_type n1-highmem-32 \
--num_workers 64 \
--project $PROJECT \
--upload_files ${JAR_FILE} \
--env_var_files hadoop2_env.sh,extensions/spark/spark_env.sh \
deploy
To summarize some of the information that came out from a separate email discussion, as IP mappings change and different debian mirrors get assigned, there can be occasional problems where the concurrent calls to apt-get install during a bdutil deployment can either overload some unbalanced servers or trigger DDOS protections leading to deployment failures. These do tend to be transient, and at the moment it appears I can deploy large clusters in zones like us-east1-c and us-east1-d successfully again.
There are a few options you can take to reduce the load on the debian mirrors:
Set MAX_CONCURRENT_ASYNC_PROCESSES to a much smaller value than the default 150 inside bdutil_env.sh, such as 10 to only deploy 10 at a time; this will make the deployment take longer, but would lighten the load as if you just did several back-to-back 10-node deployments.
If the VMs were successfully created but the deployment steps fail, instead of needing to retry the whole delete/deploy cycle, you can try ./bdutil <all your flags> run_command -t all -- 'rm -rf /home/hadoop' followed by ./bdutil <all your flags> run_command_steps to just run through the whole deployment attempt.
Incrementally build your cluster using resize_env.sh; initially set --num_workers 10 and deploy your cluster, and then edit resize_env.sh to set NEW_NUM_WORKERS=20, and run ./bdutil <all your flags> -e extensions/google/experimental/resize_env.sh deploy and it will only deploy the new workers 10-20 without touching those first 10. Then you just repeat, adding another 10 workers to NEW_NUM_WORKERS each time. If a resize attempt fails, you simply ./bdutil <all your flags> -e extensions/google/experimental/resize_env.sh delete to only delete those extra workers without affecting the ones you already deployed successfully.
Finally, if you're looking for more reproducible and optimized deployments, you should consider using Google Cloud Dataproc, which lets you use the standard gcloud CLI to deploy cluster, submit jobs, and further manage/delete clusters without needing to remember your bdutil flags or keep track of what clusters you have on your client machine. You can SSH into Dataproc clusters and use them basically the same as bdutil clusters, with some minor differences, like Dataproc DEFAULT_FS being HDFS so that any GCS paths you use should fully-specify the complete gs://bucket/object name.