YARN version
Apache Hadoop 3.1.2
Scheduler
Capacity Scheduler
Labels
highpriority, elastic
Node Label assignment
node1 [default]
node2 [default]
node3 [default]
node4 highpriority
node5 highpriority
node6 highpriority
node7 elastic
node8 elastic
What I do
I use this command to replace node1, node2 and node3 to label highpriority:
yarn rmadmin -replaceLabelsOnNode "node1=highpriority node2=highpriority node3=highpriority" -failOnUnknownNodes
This takes effect immediately. Then I submit some apps and they goes to this label as the queue assignment defines. Good.
However ~
After some time, when I decide to get node1, node2 and node3 back to normal [default], I tried:
yarn rmadmin -replaceLabelsOnNode "node1= node2= node3=" -failOnUnknownNodes
Yet, nothing happens.
After some diggings and attempts, what I find was:
Run this:
yarn rmadmin -replaceLabelsOnNode "node1=, node2=, node3=," -failOnUnknownNodes
The labels assigned to those nodes are removed.
That is, to simply add the label delimiter (,) does the job.
: ]
Related
We're using redis-cluster extensively in our production env. We currently have a 30 node cluster (15 masters, 15 slaves)
We're trying to increase the cluster, for that we've created new servers & joined them to the cluster. so far all is well.
Next - we're trying to reshard the slots to the new masters. we wrote a script that does this, using the redis-trib reshard command.
However - the migration fails midway (but not very far from the start) with this error:
[ERR] Calling MIGRATE: ERR Target instance replied with error: BUSYKEY Target key name already exists.
This happens sporadically, at times it manages to move some slots before failing, at times it fails on the first slot.
Each such failure requires a manual fixing operation which makes the reshard operation very hard to manage.
We have not found any concrete example of this, nor any idea on how to prevent this other than a downtime migration. which we are trying to avoid.
Versions:
redis server 4.0.2
redis trib 3.3.3 (downgraded from 4.0.2 following this issue : redis cluster reshard [ERR] Calling MIGRATE: ERR Syntax error)
Our next step is to upgrade to latest redis (4.0.11), even though we didn't find any indication in the release notes of this issue.
Hoping to hear we're doing something wrong and how to fix it, or is redis-cluster not built for live resharding ?
Thanks
I have faced like this problem while working with redis-clustering support for our own project. I found a problem with the redis-trib reshard command. It works fine if no key is stored in slots those are migrating from one master to another.
But redis-5 (still developing, not stable yet) has it's own
`redis-cli' that has no problem with resharding command I think. Only for lower versions of 5 it happens.
If you look at the official docs for redis say redis reconfiguration and redis cluster resharding, you'll find what they do internally to reshard.
So I solved the problem by doing those tasks by running a bash script instead of running redis-trib reshard command.
Suppose you want to reshard some slots from a master node to other master node. We'll call the node that has the current ownership of the hash slot the source node, and the node where we want to migrate the destination node.
For each slot do the following steps:
Remember that the order of these steps is important here according to redis official docs.
Send CLUSTER SETSLOT <slot> IMPORTING <source-node-id> to destination node to set the slot to importing state.
Send CLUSTER SETSLOT <slot> MIGRATING <destination-node-id> to source node to set the slot to migrating state.
Get keys from the source node with CLUSTER GETKEYSINSLOT command and move them into the destination node using the following MIGRATE command.
MIGRATE target_host target_port key target_database_id timeout
In Redis Cluster there is no need to specify a database other than 0, but MIGRATE is a general command that can be used for other tasks not involving Redis Cluster.
When the migration process is finally finished, use CLUSTER SETSLOT <slot> NODE <destination-node-id> in both source node and destination node in order to set the slot to their normal state again. The same command is usually sent to all other nodes to avoid waiting for the natural propagation of the new configuration across the cluster.
A simple example bash script to do this is also given here:
source-ip: 172.17.0.5. source-id: 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
destination-ip: 172.17.0.4. destination-id: 7e428bae84697a3882ecad19bd0d13ac7ee97d02
another master ip: 172.17.0.7
for i in `seq 0 5460`; do
redis-cli -c -h 172.17.0.4 cluster setslot ${i} importing 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
redis-cli -c -h 172.17.0.5 cluster setslot ${i} migrating 7e428bae84697a3882ecad19bd0d13ac7ee97d02
while true; do
key=`redis-cli -c -h 172.17.0.5 cluster getkeysinslot ${i} 1`
if [ "" = "$key" ]; then
echo "there are no key in this slot ${i}"
break
fi
redis-cli -h 172.17.0.5 migrate 172.17.0.4 6379 ${key} 0 5000
done
redis-cli -c -h 172.17.0.5 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.4 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.7 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
done
I google it and found two solution:
CLUSTER FORGET (http://redis.io/commands/cluster-forget)
redis-trib.rb del-node
I think CLUSTER FORGET" is the right way to do.
But I really want to know the details about redis-trib.rb del-node.
Can someone explain the difference between them?
redis-trib.rb is a ruby utility script that antirez (lead redis developer) built as a reference implementation of building administrative tools on top of the basic redis cluster commands.
Under the hood redis-trib uses CLUSTER FORGET to implement it's own administrative del-node command. https://github.com/antirez/redis/blob/unstable/src/redis-trib.rb#L1374
Redis-trib is a lot friendlier to work with. If you're doing CLUSTER FORGET you'd need to loop over and send that command to every other node in the system, while del-node will automate that process for you.
As of Redis 6.2.3
WARNING: redis-trib.rb is not longer available!
You should use redis-cli instead.
All commands and features belonging to redis-trib.rb have been moved
to redis-cli.
In order to use them you should call redis-cli with the --cluster
option followed by the subcommand name, arguments and options.
Use the following syntax:
redis-cli --cluster SUBCOMMAND [ARGUMENTS] [OPTIONS]
Example:
redis-cli --cluster info 127.0.0.1:6382
~$ redis-cli
127.0.0.1:6379> CLUSTER HELP
127.0.0.1:6379> CLUSTER NODES
127.0.0.1:6379> CLUSTER FORGET <node-id>
src/redis-trib.rb del-node 192.168.0.211:6379 650e3746968e6b7c7e357f06adbde5b3b92fcceb
Note:
192.168.0.211:6379 This is any node in the cluster
650e3746968e6b7c7e357f06adbde5b3b92fcceb this is the cluster ID of the node you want to remove. You can get the value of this ID from “cluster nodes” command.
Per redis article you should use:
redis-cli --cluster del-node 127.0.0.1:7000
where the the first argument is one of your cluster nodes.
Check 'removing a node' in he following article:
https://redis.io/docs/manual/scaling/
I am trying to do clustering on RABBITMQ. I have added 2 nodes but unable to add 3rd one.i have clustered rabbit#node1 and rabbit#node2. Now I am trying to cluster rabbit#node3 with rabbit#node1.
Here is what I am trying to do
rabbitmqctl join_cluster rabbit#node1
Clustering node rabbit#node3 with rabbit#node1 ...
Error: mnesia_not_running
Is there any solution that how to add a third node in cluster? Or any solution for the Error: mnesia_not_running
When joining cluster, target node application should be started, while source (current) node application should be stopped. Application stopped and started with rabbitmqctl stop_app/rabbitmqctl start_app.
Maybe you have stopped application on rabbit#node1, while joining it to cluster, in that case you should to run rabbitmqctl start_app on rabbit#node1, or rabbitmqctl -n rabbit#node1 start_app to be able to join it's cluster. Or you can join rabbit#node2 cluster and start app later.
To have working cluster you should start application on all nodes after joining.
It happens when the target node's app is stopped. When joining a node to a rabbitmq cluster, only the source node(the node which you are trying to link) should be stopped.
master node:
rabbitmqctl start_app
on the current node:
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit#node1
I want to delete a master node from existing Redis Cluster. I tried following the instructions from http://redis.io/topics/cluster-tutorial but unable to follow. Help??
As per previous answer, to delete a master node it must be empty. That means we've to flushall the data or migrate hash slot to another existing master.
But as redis reconfiguration and redis cluster resharding says, additional steps may need to be added after step 5 in the previous answer.
A solution that works for me is as following according to redis official docs. The order is important here.
We'll call the node that has the current ownership of the hash slot the source node, and the node where we want to migrate the destination node.
For each slot:
Send CLUSTER SETSLOT <slot> IMPORTING <source-node-id> to destination node to set the slot to importing state.
Send CLUSTER SETSLOT <slot> MIGRATING <destination-node-id> to source node to set the slot to migrating state.
Get keys from the source node with CLUSTER GETKEYSINSLOT command and move them into the destination node using the following MIGRATE command.
MIGRATE target_host target_port key target_database_id timeout
In Redis Cluster there is no need to specify a database other than 0, but MIGRATE is a general command that can be used for other tasks not involving Redis Cluster.
When the migration process is finally finished, use CLUSTER SETSLOT <slot> NODE <destination-node-id> in both source node and destination node in order to set the slot to their normal state again. The same command is usually sent to all other nodes to avoid waiting for the natural propagation of the new configuration across the cluster.
A simple example bash script to do this is also given here:
source-ip: 172.17.0.5. source-id: 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
destination-ip: 172.17.0.4. destination-id: 7e428bae84697a3882ecad19bd0d13ac7ee97d02
another master ip: 172.17.0.7
for i in `seq 0 5460`; do
redis-cli -c -h 172.17.0.4 cluster setslot ${i} importing 1f70a5107e0042a7d33a9efaf88dbdfecd78076a
redis-cli -c -h 172.17.0.5 cluster setslot ${i} migrating 7e428bae84697a3882ecad19bd0d13ac7ee97d02
while true; do
key=`redis-cli -c -h 172.17.0.5 cluster getkeysinslot ${i} 1`
if [ "" = "$key" ]; then
echo "there are no key in this slot ${i}"
break
fi
redis-cli -h 172.17.0.5 migrate 172.17.0.4 6379 ${key} 0 5000
done
redis-cli -c -h 172.17.0.5 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.4 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
redis-cli -c -h 172.17.0.7 cluster setslot ${i} node 7e428bae84697a3882ecad19bd0d13ac7ee97d02
done
Node should be empty before deleting it. As such, we need to move the data and also slots before deleting a node.
Follow below steps to migrate:
Connect to destination node:
cluster setslot (slot no) importing (source node id)
Connect to source node:
cluster setslot (slot no) migrating (destination node id)
migrate (destination ip) (destination port) key (destination db) (timeout)
Connect to destination node:
cluster setslot (slot no) node (dest node)
We can do the above in a loop for all the slots and keys and then delete the source node.
I created a Django application which runs inside a Docker container. I needed to create a thread inside the Django application so I used Celery and Redis as the Celery Database.
If I install redis in the docker image (Ubuntu 14.04):
RUN apt-get update && apt-get -y install redis-server
RUN pip install redis
The Redis server is not launched: the Django application throws an exception because the connection is refused on the port 6379. If I manually start Redis, it works.
If I start the Redis server with the following command, it hangs :
RUN redis-server
If I try to tweak the previous line, it does not work either :
RUN nohup redis-server &
So my question is: is there a way to start Redis in background and to make it restart when the Docker container is restarted ?
The Docker "last command" is already used with:
CMD uwsgi --http 0.0.0.0:8000 --module mymodule.wsgi
RUN commands are adding new image layers only. They are not executed during runtime. Only during build time of the image.
Use CMD instead. You can combine multiple commands by externalizing them into a shell script which is invoked by CMD:
CMD start.sh
In the start.sh script you write the following:
#!/bin/bash
nohup redis-server &
uwsgi --http 0.0.0.0:8000 --module mymodule.wsgi
When you run a Docker container, there is always a single top level process. When you fire up your laptop, that top level process is an "init" script, systemd or the like. A docker image has an ENTRYPOINT directive. This is the top level process that runs in your docker container, with anything else you want to run being a child of that. In order to run Django, a Celery Worker, and Redis all inside a single Docker container, you would have to run a process that starts all three of them as child processes. As explained by Milan, you could set up a Supervisor configuration to do it, and launch supervisor as your parent process.
Another option is to actually boot the init system. This will get you very close to what you want since it will basically run things as though you had a full scale virtual machine. However, you lose many of the benefits of containerization by doing that :)
The simplest way altogether is to run several containers using Docker-compose. A container for Django, one for your Celery worker, and another for Redis (and one for your data store as well?) is pretty easy to set up that way. For example...
# docker-compose.yml
web:
image: myapp
command: uwsgi --http 0.0.0.0:8000 --module mymodule.wsgi
links:
- redis
- mysql
celeryd:
image: myapp
command: celery worker -A myapp.celery
links:
- redis
- mysql
redis:
image: redis
mysql:
image: mysql
This would give you four containers for your four top level processes. redis and mysql would be exposed with the dns name "redis" and "mysql" inside your app containers, so instead of pointing at "localhost" you'd point at "redis".
There is a lot of good info on the Docker-compose docs
use supervisord which would control both processes. The conf file might look like this:
...
[program:redis]
command= /usr/bin/redis-server /srv/redis/redis.conf
stdout_logfile=/var/log/supervisor/redis-server.log
stderr_logfile=/var/log/supervisor/redis-server_err.log
autorestart=true
[program:nginx]
command=/usr/sbin/nginx
stdout_events_enabled=true
stderr_events_enabled=true