I was reading Hortonworks documenrtation to remove regionserver from any host of cluster (http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_system-admin-guide/content/admin_decommission-slave-nodes-3.html).
It uses graceful_stop.sh script . The same script is described at Apache Hbase book (https://hbase.apache.org/book/node.management.html)
I was trying to find this script but not able to locate it .
hbase#node ~]$ ls /usr/lib/hbase/bin/
draining_servers.rb hbase.cmd hbase-daemon.sh region_status.rb test
get-active-master.rb hbase-common.sh hbase-jruby replication
hbase hbase-config.cmd hirb.rb start-hbase.cmd
hbase-cleanup.sh hbase-config.sh region_mover.rb stop-hbase.cmd
[hbase#node ~]$
Is this script is removed from hbase ?
Is there any other way to stop a region server from anyother host of cluster. For eg - I want to stop region server 1 . Can I do this by logging into region server2?
Yes, the script is removed from hbase if you use package install. But you can still find it in src files.
If you want to stop a region server A from another host B, then host B must have privilege to access A. e.g. You have added public key of host B to authorized_keys in A. For a typical cluster, a RS cannot login to other RS directly for security.
For how to write graceful_stop.sh by yourself, you can look at: https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/fA3019_vpZY
Related
I have been able to successfully set-up password-less ssh from host A to host B.
Now I want a script that I run on host A that goes to B and then reads a file or just does something on host B and returns back to A.
On host A:
#!/bin/bash
ssh hostB
echo "$hostname"
ssh hostA
But I just see that this script just lands me on hostB and doesn't perform the echo as I intended it to.
Can someone please suggest what's to be done in such a case? I have gone through some suggestions on using sshpass. But since these are Production hosts, there is a fair bit of paper trail I need to go through to install anything on the hosts and the deadlines are looming to get this fixed.
Answered this question: -
Just needed the following
ssh
Also to note that no $pwd or ./ references hold good on the absolute path . So if there is a script that you are running on the remote host with such references , make sure to change the script so the absolute path is mentioned instead of those references
I have a trouble with NFS client file caching. The client read the file which was removed from the server many minutes before.
My two servers are both CentOS 6.5 (kernel: 2.6.32-431.el6.x86_64)
I'm using server A as the NFS server, /etc/exports is written as:
/path/folder 192.168.1.20(rw,no_root_squash,no_all_squash,sync)
And server B is used as the client, the mount options are:
nfsstat -m
/mnt/folder from 192.168.1.7:/path/folder
Flags: rw,sync,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,nosharecache,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.20,minorversion=0,lookupcache=none,local_lock=none,addr=192.168.1.7
As you can see, "lookupcache=none,noac" options are already used to disable the caching, but seems doesn't work...
I did the following steps:
Create a simple text file from server A
Print the file from the server B by cmd "cat", and it's there
Remove the file from the server A
Wait couple minutes and print the file from the server B, and it's still there!
But if I do "ls" from the server B at that time, the file is not in the output. The inconsistent state may last a few minutes.
I think I've checked all the NFS mount options...but can't find the solution.
Is there any other options I missed? Or maybe the issue is not about NFS?
Any ideas would be appreciated :)
I have tested the same steps you have given with below parameters. Its working perfectly. I have added one more parameter "fg" in the client side mounting.
sudo mount -t nfs -o fg,noac,lookupcache=none XXX.XX.XX.XX:/var/shared/ /mnt/nfs/fuse-shared/
I am running an Hadoop Cluster on Google Cloud Platform, using Google Cloud Storage as backend for persistent data. I am able to ssh to the master node from a remote machine and run hadoop fs commands. Anyway when I try to execute the following code I get a timeout error.
Code
FileSystem hdfs =FileSystem.get(new URI("hdfs://mymasternodeip:8020"),new Configuration());
Path homeDir=hdfs.getHomeDirectory();
//Print the home directory
System.out.println("Home folder: " +homeDir);
// Create a directory
Path workingDir=hdfs.getWorkingDirectory();
Path newFolderPath= new Path("/DemoFolder");
newFolderPath=Path.mergePaths(workingDir, newFolderPath);
if(hdfs.exists(newFolderPath))
{
hdfs.delete(newFolderPath, true); //Delete existing Directory
}
//Create new Directory
hdfs.mkdirs(newFolderPath);
When executing the hdfs.exists() command I get a timeout error.
Error
org.apache.hadoop.net.ConnectTimeoutException: Call From gl051-win7/192.xxx.1.xxx to 111.222.333.444.bc.googleusercontent.com:8020 failed on socket timeout exception: org.apache.hadoop.net.ConnectTimeoutException: 20000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=111.222.333.444.bc.googleusercontent.com/111.222.333.444:8020]
Are you aware of any limitation using the Java Hadoop APIs against Hadoop on Google Cloud Platform ?
Thanks!
It looks like you're running that code on your local machine and trying to connect to the Google Compute Engine VM; by default, GCE has strict firewall settings to avoid exposing your external IP addresses to arbitrary inbound connections. If you're using defaults then your Hadoop cluster should be on the "default" GCE network. You'll need to follow the adding a firewall instructions to allow incoming TCP connections on port 8020 and possible on other Hadoop ports as well from your local IP address for this to work. It'll look something like this:
gcloud compute firewall-rules create allow-http \
--description "Inbound HDFS." \
--allow tcp:8020 \
--format json \
--source-ranges your.ip.address.here/32
Note that you really want to avoid opening a 0.0.0.0/0 source-range since Hadoop isn't doing authentication or authorization on those incoming requests. You'll want to restrict it as much as possible to only the inbound IP addresses from which you plan to dial in. You may need to open up a couple other ports as well depending on what functionality you use connecting to Hadoop.
The more general recommendation is that wherever possible, you should try to run your code on the Hadoop cluster itself; in that case, you'll use the master hostname itself as the HDFS authority rather than external IP:
hdfs://<master hostname>/foo/bar
That way, you can limit the port exposure to just the SSH port 22, where incoming traffic is properly gated by the SSH daemon, and then your code doesn't have to worry about what ports are open or even about dealing with IP addresses at all.
I have two amazon ec2 instances
i can connect to those ec2 instance from my windows using putty (by the public key generated from the private key provided by amazon)
now i want to install tungsten replicator into my ec2 instances
and tungsten replicator needs ssh access from one ec2 instance to another ec2 instance
i tried to check that ssh is working or not from one ec2 instance to another
i tried:
ssh ec2-user#public ip of destination instance
//also tried
ssh ec2-user#private ip destination instance
but its not working
i got following error:
Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
i have search on google and tried some trick but none of them worked
sometime i got following error:
Address public_ip maps to xxxx.eu-west-1.compute.amazonaws.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
can anyone please tell me how to connect ssh from one ec2 instance to another
I'd suggest you to create a special keypair for the tungsten user.
cd tungsten-user-home/.ssh
ssh-keygen -t rsa
mv id-rsa.pub authorized-keys
And then copy both files to the other host in the same place and permissions.
This will allow tungsten to work without requiring your own key.
Just like when you have to ssh from you local machine to an EC2 instance, you need to provide the ssh command the proper pem file:
ssh -i my_pem_file.pem ec2-user#private-or-public-ip-or-dns
Just in case anyone ponder on this question, here is my 2 cents.
Connecting one EC2 instance from another EC2 instance will work as suggested by "Uri Agassi". Considering best practices and security, it will be good idea to create and assign a role to source EC2 instance.
One way to allow one EC2 instance to connect to another is to set an ingress rule on the target EC2 instance that lets it accept traffic from the source EC2 instance's security group. Here's a Python function that uses Boto3 to do this:
import boto3
ec2 = boto3.resource('ec2')
def allow_security_group_ingress(target_security_group_id, source_security_group_name):
try:
ec2.SecurityGroup(target_security_group_id).authorize_ingress(
SourceSecurityGroupName=source_security_group_name)
logger.info("Added rule to group %s to allow traffic from instances in "
"group %s.", target_security_group_id, source_security_group_name)
except ClientError:
logger.exception("Couldn't add rule to group %s to allow traffic from "
"instances in %s.",
target_security_group_id, source_security_group_name)
raise
After you've set this, put the private key of the key pair on the source instance and use it when you SSH from the source instance:
ssh -i {key_file_name} ec2-user#{private_ip_address_of_target_instance}
There's a full Python example that shows how to do this on GitHub /awsdocs/aws-doc-sdk-examples.
See, if you have deployed both machines with the same key pair, or different, it's not a problem just go to your host ec2 machine and in .ssh folder make a key file with the same name of the key that is used to create the second machine, now use chmod 400 keypair name and then try ssh -i keyname user-name#IP
I've been doing some tests with mongodb and sharding and at some point I tried to add new config servers to my mongos router (at that time, I was playing with just one config server). But I couldn't find any information on how to do this.
Have anybody tried to do such a thing?
Unfortunately you will need to shutdown the entire system.
Shutdown all processes (mongod, mongos, config server).
Copy the data subdirectories (dbpath tree) from the config server to the new config servers.
Start the config servers.
Restart mongos processes with the new --configdb parameter.
Restart mongod processes.
From: http://www.mongodb.org/display/DOCS/Changing+Config+Servers
Use DNS CNAMES
make sure to use DNS entries or at least /etc/hosts entries for all mongod and mongo config servers when you set-up multiple config servers in /etc/mongos.conf , when you set-up replica sets and/or sharding.
e.g. a common pitfall on AWS is to use just the private DNS name of EC2 instances, but these can change over time... and when that happens you'll need to shut down your entire mongodb system, which can be extremely painful to do if it's in production.
The Configure Sharding and Sample Configuration Session pages appear to have what you're looking for.
You must have either 1 or 3 config servers; anything else will not work as expected.
You need to dump and restore content from your original config server to 2
new config servers before adding them to mongos's --configdb.
The relevant section is:
Now you need a configuration server and mongos:
`$ mkdir /data/db/config
$ ./mongod --configsvr --dbpath /data/db/config --port 20000 > /tmp/configdb.log &
$ cat /tmp/configdb.log
$ ./mongos --configdb localhost:20000 > /tmp/mongos.log &
$ cat /tmp/mongos.log `
mongos does not require a data directory, it gets its information from the config server.