This are the steps I've done so far:
Download spark-1.4.1-bin-hadoop2.6.tgz
unzip
.spark-1.4.1-bin-hadoop2.6/sbin/start-all.sh
Master is working but slave doesn't start
This is the output:
[ec2-user#ip-172-31-24-107 ~]$ sudo ./spark-1.4.1-bin-hadoop2.6/sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/ec2-user/spark-1.4.1-bin-hadoop2.6/sbin/../logs/spark-root-org.apache.spark.deploy.master.Master-1-ip-172-31-24-107.out
localhost: Permission denied (publickey).
[ec2-user#ip-172-31-24-107 ~]$
This is the secure log
Aug 9 00:09:30 ip-172-31-24-107 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=./spark-1.4.1-bin-hadoop2.6/sbin/start-all.sh
Aug 9 00:09:32 ip-172-31-24-107 sshd[4828]: Connection closed by 127.0.0.1 [preauth]
I believe the problem is with SSH but I haven't been able to find the solution on google...
Any idea how to fix my SSH issue?
You need to set up passwordless ssh. Try:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Then restart the cluster. If that does not work please post new error message(s).
Its infact a two step process;
Generate public/private rsa keypair.
ubuntu#master:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa):
Your identification has been saved in /home/ubuntu/.ssh/id_rsa.
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub.
The key fingerprint is:
80:4d:40:f6:3a:09:32:07:74:25:cc:cd:f2:b3:75:10 ubuntu#master.flexilogix
The key's randomart image is:
+--[ RSA 2048]----+
|o.o+Bo.E. |
| ..=.B . |
|o o + + . |
| + . = o . |
| + + S |
| o |
| |
| |
| |
+-----------------+
set passwordless ssh;
ubuntu#master:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
Once both steps are done, you should be able to verify it;
ubuntu#master:~$ ssh localhost
Related
[root#spectrumscale ~]# chmod 700 .ssh
[root#spectrumscale ~]# cd .ssh
[root#spectrumscale .ssh]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
/root/.ssh/id_rsa already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
05:63:ff:2a:82:fc:c9:31:87:fc:a1:61:dc:4e:5a:52 root#spectrumscale
The key's randomart image is:
+--[ RSA 2048]----+
| + |
| . + |
| o |
| . . |
| E . |
| . + + . |
| o # B . |
| + / o |
| * o |
+-----------------+
[root#spectrumscale .ssh]# ssh-copy-id -i /root/.ssh/id_rsa.pub root#192.168.1.215
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root#192.168.1.215's password:
Permission denied, please try again.
root#192.168.1.215's password:
Number of key(s) added: 1
Now try logging into the machine, with: ssh 'root#192.168.1.215'"and check to make sure that only the key(s) you wanted were added.
[root#spectrumscale .ssh]# ssh 192.168.1.215
root#192.168.1.215's password:
Last failed login: Tue Nov 12 17:47:37 IST 2019 from 192.168.1.203 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Tue Nov 12 14:44:01 2019 from localhost
You have to diagnose the root cause for this issue. You can find this by reading logs related sshd using journalctl command on the system you want to login.
Reading logs :
journalctl -t sshd
If the log shows some thing similar to Authentication refused:
bad ownership or modes for directory, this is due to bad ownership or modes for directory /home/<your_user>/.ssh.
fixing permissions by
chmod go-w /home/<your_user>
chmod 700 /home/<your_user>/.ssh
chmod 600 /home/<your_user>/.ssh/authorized_keys
Also make sure that inside sshd configuration file /etc/ssh/sshd_config, make sure that PubkeyAuthentication is not commented and set yes.
Inside /etc/ssh/sshd_config make sure these is a line,
PubkeyAuthentication yes
It might needed to restart sshd service after edit in sshd configuration file.
sudo service sshd restart
This worked for me and hope this helps!.
If you have verified all your permissions are correct, but are still being prompted for a password, make sure to add the below line to the file /etc/ssh/sshd_config on the system you want to login to without a password. This will allow the SSH daemon to accept ssh-rsa key types
pubkeyacceptedkeytypes ssh-rsa
After doing this, simply run the command service sshd restart and passwordless login should work now
I often see unofficial documents that says you should save vagrant user’s ssh public key when creating your own box like below:
curl https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub >> /home/vagrant/.ssh/authorized_keys
And when vagrant up with the box, the following message is shown:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
It seems to me that mitchellh’s vagrant.pub key above is not appropriate.
What is the best way to save vagrant user’s ssh public key?
The key you point on raw.githubusercontent.com is a sample to no be used.
The returned message seem tell that another keypair is automatically generated:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
so if everything is going correctly, your container now use a new public key in /home/vagrant/.ssh/authorized_keys and your local host use a new private key in $HOME/.ssh/id_rsa.
Compare this files with files you originally downloaded, look at modification time of both files (on local host and in vagrant container)
Build your own key
Simply run:
ssh-keygen -f ~/.ssh/vagrant-dedicated
see man ssh-keygen for key length, cipher, etc...
Sample output:
dialog:
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Simple output:
Generating public/private rsa key pair.
Your identification has been saved in vagrant-dedicated.
Your public key has been saved in vagrant-dedicated.pub.
The key fingerprint is:
SHA256:U2YfVbMlCUed7tXrvf3xBQoLB3glpSpto4hwdjTKwV0 user #host
The key's randomart image is:
+---[RSA 2048]----+
| E ..o .o==+|
| . . . . + o.o=|
| o + . + + . ...|
| . + o o = . . .o|
|. = o = S o . o o|
|.+ o + . + o . + |
|. . . . . ..o|
| .*|
| .*|
+----[SHA256]-----+
This will create two files:
ls -l ~/.ssh/vagrant-dedicated*
-rw------- 1 user user 1679 Oct 20 12:18 vagrant-dedicated
-rw-r--r-- 1 user user 394 Oct 20 12:18 vagrant-dedicated.pub
head -n1 ~/.ssh/vagrant-dedicated*
==> vagrant-dedicated <==
-----BEGIN RSA PRIVATE KEY-----
==> vagrant-dedicated.pub <==
ssh-rsa AAAAB3...0y/5 user#host
Replace content of containers /home/vagrant/.ssh/authorized_keys by content of ~/.ssh/vagrant-dedicated.pub, then use vagrant-dedicated as private key for ssh connection.
ssh -i ~/.ssh/vagrant-dedicated vagrant#container
Note about fingerprint
Before 1st connection to a new target host, ssh will prompt you about host's fingerprint.
You could compare output of
ssh-keygen -vlf /etc/ssh/ssh_host_rsa_key.pub
on target vagrant container with output of your 1st connection output:
ssh -o VisualHostKey=true -i ~/.ssh/vagrant-dedicated vagrant#container
First run will begin output like:
The authenticity of host 'container (10.12.34.56)' can't be established.
Then, the fingerprint, something like
ECDSA key fingerprint is SHA256:9M+2wGn0nZO3GPYkWuuxzXqI3nIbk5IJJ5xwhsxwbXk
And the Ascii art representation:
+---[ECDSA 256]---+
| . .. . |
| = .+ E |
| =oo. |
| .=.. |
| S=o. |
| o+=o..o |
| =+*X*..|
| . =*+#+.|
| .o=O+= |
+----[SHA256]-----+
Both commands must give identical fingerprint and ascii art.
I'm using circleci 2.0 and I'm trying to have circleci enter a gcp instance via ssh and fire off a deployment script with this command
sudo /opt/google-cloud-sdk/bin/gcloud compute ssh instance-1
--command=/home/deploy_staging.sh --zone=us-east1-b
Doing this from my local machine works just fine, but when I try to do it from circleci I get this error:
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Too long with no output (exceeded 10m0s)
Obviously it's ssh issues. I read something about putting ssh keys in the instance on gcp, and I have put my local machine's ssh keys there, but that still doesn't work.
gcloud \
--quiet \
--project="${PROJECT}" \
compute ssh "${INSTANCE_NAME}" \
--zone "${ZONE}" \
--strict-host-key-checking=no \
--command "echo works"
--quiet is all you need and it will generate a key if one is not available. I am using this with a service account.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/google_compute_engine.
Your public key has been saved in /root/.ssh/google_compute_engine.pub.
The key fingerprint is:
SHA256:un2aZmExTGVD0KvebEVqAujrlXoAb0u7jO3Z5boCWaA root#581dc589b7fa
The key's randomart image is:
+---[RSA 2048]----+
| .== |
| . .... |
| . . . o . |
|E . o . + . . |
| * .S+ o |
| o * o= o . |
| + =+o.* . |
| +==.=+.= |
| .oO+===+ |
+----[SHA256]-----+
WARNING: Using OS Login user [sa_102839341411404994442] instead of default user [root]
According to the official CircleCI documentation, you first need to add the SSH key for the server in the CircleCI application - this can be done on your project’s settings and under the SSH Permissions section.
I am not able to create an SSH key pair on OpenShift.com. I get the following error message
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/openshift/$USER/.ssh/id_rsa): mytestkeys
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
open mytestkeys failed: Permission denied.
Saving the key failed: mytestkeys.
By default you don't have permissions to change anything in the ~/.ssh directory. However you can create a key-pair by going over to ~/app-root/data and generating your key there. For example:
ssh-keygen -t rsa -C "test#redhat.com"
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/openshift/540e021350044613d80000be/.ssh/id_rsa): ./id_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ./id_rsa.
Your public key has been saved in ./id_rsa.pub.
The key fingerprint is:
8c:d5:eb:3f:5c:86:07:d8:40:7e:aa:cc:6e:70:6a:42 test#redhat.com
The key's randomart image is:
+--[ RSA 2048]----+
| .. |
| o. |
| . o+. |
| + .+o |
| . S o o |
| E .o.o . + |
| . ++ .. + |
| . o.. .o |
| o .. .. |
+-----------------+
I am trying to setup ssh keys via this method.
Here is the starting point on server
server:~/.ssh$ ls
#shows empty
Here is the starting point on client
client: ~/.ssh$ls #shows empty dir
On client, I run
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/me/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/me/.ssh/id_rsa.
Your public key has been saved in /Users/me/.ssh/id_rsa.pub.
The key fingerprint is:
26:25:e0:2f:90:d8:d9:fb:79:03:5d:99:a1:61:a9:dc me#machine
The key's randomart image is:
+--[ RSA 2048]----+
| . o.. |
| o = . ..o + |
|. = o..oo + |
| . oo+E. |
| o + S |
| o = |
| o o |
| . . |
| |
+-----------------+
$ ls
id_rsa id_rsa.pub
So far so good. So then I run
$ ssh-copy-id -p 3457 me#server #running on non standard port
The authenticity of host '[server]:3457 ([104.131.226.216]:3457)' can't be established.
RSA key fingerprint is 1c:52:db:19:22:b8:47:18:24:ad:07:2f:e5:d3:c4:8e.
Are you sure you want to continue connecting (yes/no)? yes
/usr/local/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/local/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
me#server's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -p '3457' 'me#server'"
and check to make sure that only the key(s) you wanted were added.
I run ssh -p '3457' 'me#server' and see that ~/.ssh/authorized_keys shows that the keys match. I logout, trying sshing in again. But it STILL asks me for a password -- even after running sudo service ssh restart on server. I'm not sure what else I can do. What are the next steps?
Make sure you have that key already loaded in your session
$ ssh-add
It should print ~/.ssh/id_rsa
In case you get an error saying "couldn't get a connection to the authentication agent" then execute
$ eval $(ssh-agent)
and repeat ssh-add
The former should be enough, but I also reccomend having an ssh config file. For this, edit (or create) an ~/.ssh/config file with contents
Host server
HostName 104.131.226.216
User me
Port 3457
IdentityFile ~/.ssh/id_rsa
Are the file access permissions for the authorized_keys file 644 (-rw-r--r--)?
(I would leave this as a comment, but I don't have enough points)
eval ssh-agent
After that, add identity to key file
ssh-add $HOME/.ssh/xxx.pem
After execution of above command it should not prompt you for a password
You need to do this every time if you use a new bash prompt. For that one do one thing.
Edit the above lines in $HOME/.bashrc file