I have a cron job run on local server. I would like to use it to
start a new EMR cluster
trigger a Spark model training job on the cluster
like this ssh -o "StrictHostKeyChecking no" -i xxx.pem hadoop#10.10.x.xxxx "bash ~/train_model.sh". Since the EMR cluster is new every time, use -o "StrictHostKeyChecking no" flag to avoid new host check.
finally shut down the EMR cluster.
The problem is the the model training takes 10+hrs and the ssh connection in step2 timeout every time.
By searching around, I find it might be resolved by editing ssh config on EMR cluster master node, but since the EMR cluster is new every time so I have to also do the edit every time. I am interesting to find out if there is a more neat way?
Related
I have installed Redis GUI redis-commander by using https://github.com/joeferner/redis-commander
Redis running on localhost:6379 as a container at docker.
This says if I run redis on localhost:6379, all I need to get started is;
docker run --rm --name redis-commander -d -p 8081:8081 ghcr.io/joeferner/redis-commander:latest
But I encountered with this problem... Is there anyone who got this error and found a solution for this ??
There are some things you have to take into account.
Redis commander is running inside a container so localhost no longer points to your laptop/desktop/developing machine/server. It points to the container itself where no redis is running. So it will never connect. You need to point to the other container.
For this, you should be using some-redis (the name of the container) instead of localhost. In Redis Commander click more and add server to add a new connection
But this will not work unless both containers are running inside the same network.
You need to create first a new docker network
docker network create redis
And then run your containers using this parameter --network=redis
More about docker network here
More about docker run with networks here
Recently, I started to have some trouble with one of me Redis cluster. used_memroy and used_memory_rss increasing constantly.
According to some Googling, I found following discussion:
https://github.com/antirez/redis/issues/4570
Now I am wandering if it is safe to run SCRIPT FLUSH command on my production Redis cluster?
Yes - you can run the SCRIPT FLUSH command safely in a production cluster. The only potential side effect is blocking the server while it executes. Note, however, that you'll want to call it in each of your nodes.
I'm trying to create a Redis cluster using an RDB file taken from a single-instance Redis server. Here is what I've tried:
#! /usr/bin/env bash
for i in 6000 6001 6002 6003
do
redis-server --port $i --cluster-config-file "node-$i.cconf" --cluster-enabled yes --dir "$(pwd)" --dbfilename dump.rdb &
done
That script starts up 4 Redis processes that are cluster enabled. It also initializes each node with the dump file.
Then I run redis-trib.rb so that the 4 nodes can find each other:
redis-trib.rb create 127.0.0.1:6000 127.0.0.1:6001 127.0.0.1:6002 127.0.0.1:6003
I get the following error:
>>> Creating cluster
[ERR] Node 127.0.0.1:6060 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
I've also tried a variant where only the first node/process is initialized with the RDB file and the others are empty. I can join the 3 empty nodes into a cluster but not the one that's pre-populated with data.
What is the correct way to import a pre-existing RDB file into a Redis cluster?
In case this is an "X-Y problem" this is why I'm doing this:
I'm working on a migration from a single-instance Redis server to an Elasticache Redis Cluster (cluster mode enabled). Elasticache can easily import an RDB file on cluster startup if you upload it to S3. But it takes a long time for an Elasticache cluster to start. To reduce the feedback loop as I test my migration code, I'd like to be able to start a cluster locally also.
I've also tried using the create-cluster utility, but that doesn't appear to have any options to pre-populate the cluster with data.
I am very new to Google Cloud Platform and was trying to restart my VM instance. I entered $ sudo poweroff into my SSH console, as suggested by https://cloud.google.com/compute/docs/instances/stopping-or-deleting-an-instance#stop_an_instance
and the console did not return anything. Afterwords I started the VM instance again and the SSH console started returning the message "We are unable to connect to the VM on port 22.".
I have a snapshot of my root disk, but I would really like my instance to be running properly again.
When you run either sudo poweroff or sudo shutdown -h now, your VM will shutdown right away. This involves flushing any in-memory buffers for disks back to the disks so that you do not lose any unflushed data.
Since you're initiating this command over a ssh session, you will not be able to look at any shutdown messages over ssh while the instance is shutting down (since the network service on the VM will also be brought down).
You can use gcloud compute instances list or gcloud compute instances describe VM_NAME commands to find out the status of the VM.
If it says RUNNING, it means the instance is running and you will be able to ssh to the VM. If it says TERMINATED, it means the instance was shutdown/terminated and you will not be billed for this instance.
Suppose I have [Slave IP Address] which is the slave of [Master IP Address].
Now my master server has been shut down, and I need to set this slave to be master MANUALLY (WITHOUT using sentinel automatic failover, WITH redis command).
Is it possible doing this without restarting the redis service ? (and losing all the cached data)
use SLAVEOF NO ONE to promote a slave to master
http://redis.io/commands/slaveof
it depends, if you are in a cluster you will be better using the fail over. You will need to use the force option in the command
http://redis.io/commands/cluster-failover
Is it possible doing this without restarting the redis service? (and
losing all the cached data)
yes that's possible, you can use
SLAVEOF NO ONE (without sentinel)
But it is recommended to use sentinel to avoid data loss.
sentinel failover master-name(with sentinel)
This will force the sentinel to switch master.
The new master will have all the data that was synchronized before the old-master shutdown.
Redis will automatically choose the best slave with max. data, that will reduce the amount of data we lose when switching master.
Below 2 options in step 3 have helped me to recover the cluster once a master node is down, compute was replaced or other not recoverable state.
1 .- First you need to connect to the slave node, use redis-cli, here a link how to do that: How to connect to remote Redis server?
2 .- Once connected to the slave node run the command cluster nodes to validate master node is in fail state, also run cluster info to see the overall state of your cluster(this is always a good idea)
3 .- Inside the slave node to be promoted run command: cluster failover,
in rare cases when there is some serious issues with redis this
command could fail, and you will need to use cluster failover force
or cluster failover takeover, here more info abut the implications
of those options: https://redis.io/commands/cluster-failover
4 .- Run cluster forged $old_master_id in all your cluster nodes
5 .- Add a new node with cluster meet $new_node_IP $new_node_PORT
6 .- Subscribe your new node to your brand new master, login in to the new bode and run cluster replicate $master_node_id
Steps 1-3 are required for the slave-master promotion and 4-5 are required to left all cluster in a healthy master-slave equilibrium.
As of Redis version 5.0.0 the SLAVEOF command is regarded as deprecated.
If a Redis server is already acting as replica, the command REPLICAOF NO ONE will turn off the replication, turning the Redis server into a MASTER.