This are the steps I've done so far:
Download spark-1.4.1-bin-hadoop2.6.tgz
unzip
.spark-1.4.1-bin-hadoop2.6/sbin/start-all.sh
Master is working but slave doesn't start
This is the output:
[ec2-user#ip-172-31-24-107 ~]$ sudo ./spark-1.4.1-bin-hadoop2.6/sbin/start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/ec2-user/spark-1.4.1-bin-hadoop2.6/sbin/../logs/spark-root-org.apache.spark.deploy.master.Master-1-ip-172-31-24-107.out
localhost: Permission denied (publickey).
[ec2-user#ip-172-31-24-107 ~]$
This is the secure log
Aug 9 00:09:30 ip-172-31-24-107 sudo: ec2-user : TTY=pts/0 ; PWD=/home/ec2-user ; USER=root ; COMMAND=./spark-1.4.1-bin-hadoop2.6/sbin/start-all.sh
Aug 9 00:09:32 ip-172-31-24-107 sshd[4828]: Connection closed by 127.0.0.1 [preauth]
I believe the problem is with SSH but I haven't been able to find the solution on google...
Any idea how to fix my SSH issue?
You need to set up passwordless ssh. Try:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Then restart the cluster. If that does not work please post new error message(s).
Its infact a two step process;
Generate public/private rsa keypair.
ubuntu#master:~$ ssh-keygen -t rsa -P ""
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa):
Your identification has been saved in /home/ubuntu/.ssh/id_rsa.
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub.
The key fingerprint is:
80:4d:40:f6:3a:09:32:07:74:25:cc:cd:f2:b3:75:10 ubuntu#master.flexilogix
The key's randomart image is:
+--[ RSA 2048]----+
|o.o+Bo.E. |
| ..=.B . |
|o o + + . |
| + . = o . |
| + + S |
| o |
| |
| |
| |
+-----------------+
set passwordless ssh;
ubuntu#master:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
Once both steps are done, you should be able to verify it;
ubuntu#master:~$ ssh localhost
I have an encryption gateway connecting over sFTP to ~500 vendors, all working fine thru ssh-key based authentication. However for a particular host, it simply does not. It turned out to have a very unique behaviour. I have created a specific key for this particular vendor, but...
authorized_keys uploaded to destination box:
sftp> put vendorXX.id_rsa.pub ./.ssh/authorized_keys
Uploading vendorXX.id_rsa.pub to /.ssh/./authorized_keys
sftp> ls -l
drwxrwxrwx 2 user FTP_root 1024 Mar 31 01:04 .
drwxrwxrwx 7 user FTP_root 1024 Aug 25 2014 ..
-rw------- 1 user FTP_root 1680 Apr 1 02:54 authorized_keys
the pair sets matches to each other:
$ ssh-keygen -lf vendorXX.id_rsa
2048 e7:64:53:21:2d:12:62:bb:13:99:1b:c4:dc:24:78:09
$ ssh-keygen -lf vendorXX.id_rsa.pub
2048 e7:64:53:21:2d:12:62:bb:13:99:1b:c4:dc:24:78:09
Key is specified on ssh config for given host:
$ cat ~/.ssh/config | ggrep vendorxx –A2
Host ftp.vendorxx.com
PubkeyAuthentication yes
IdentityFile ~/.ssh/vendorXX
And bada bim bada boom… When I try to connect, it uses the key but still fire the password challenge:
$ sftp -v user#ftp.vendorxx.com 2>&1 | grep -i ident
debug1: identity file /home/gtwadmin/.ssh/vendorXX.id_rsa type 1
Password:
However, if I explicit specify the identity on sftp command, it goes on:
$ sftp -oIdentityFile=~/.ssh/vendorXX.id_rsa user#ftp.vendorxx.com
Connecting to ftp.vendorxx.com...
sftp>
Any insight on this? My source box is Solaris 10 and Sun_SSH_1.1.6, SSH protocols 1.5/2.0, OpenSSL 0x0090704f.
I am trying to setup ssh keys via this method.
Here is the starting point on server
server:~/.ssh$ ls
#shows empty
Here is the starting point on client
client: ~/.ssh$ls #shows empty dir
On client, I run
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/me/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/me/.ssh/id_rsa.
Your public key has been saved in /Users/me/.ssh/id_rsa.pub.
The key fingerprint is:
26:25:e0:2f:90:d8:d9:fb:79:03:5d:99:a1:61:a9:dc me#machine
The key's randomart image is:
+--[ RSA 2048]----+
| . o.. |
| o = . ..o + |
|. = o..oo + |
| . oo+E. |
| o + S |
| o = |
| o o |
| . . |
| |
+-----------------+
$ ls
id_rsa id_rsa.pub
So far so good. So then I run
$ ssh-copy-id -p 3457 me#server #running on non standard port
The authenticity of host '[server]:3457 ([104.131.226.216]:3457)' can't be established.
RSA key fingerprint is 1c:52:db:19:22:b8:47:18:24:ad:07:2f:e5:d3:c4:8e.
Are you sure you want to continue connecting (yes/no)? yes
/usr/local/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/local/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
me#server's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -p '3457' 'me#server'"
and check to make sure that only the key(s) you wanted were added.
I run ssh -p '3457' 'me#server' and see that ~/.ssh/authorized_keys shows that the keys match. I logout, trying sshing in again. But it STILL asks me for a password -- even after running sudo service ssh restart on server. I'm not sure what else I can do. What are the next steps?
Make sure you have that key already loaded in your session
$ ssh-add
It should print ~/.ssh/id_rsa
In case you get an error saying "couldn't get a connection to the authentication agent" then execute
$ eval $(ssh-agent)
and repeat ssh-add
The former should be enough, but I also reccomend having an ssh config file. For this, edit (or create) an ~/.ssh/config file with contents
Host server
HostName 104.131.226.216
User me
Port 3457
IdentityFile ~/.ssh/id_rsa
Are the file access permissions for the authorized_keys file 644 (-rw-r--r--)?
(I would leave this as a comment, but I don't have enough points)
eval ssh-agent
After that, add identity to key file
ssh-add $HOME/.ssh/xxx.pem
After execution of above command it should not prompt you for a password
You need to do this every time if you use a new bash prompt. For that one do one thing.
Edit the above lines in $HOME/.bashrc file
I am trying to push my project onto my bitbucket, been messing with this for about 4 days pouring through countless problem solving/pages/troubleshooting/tutorials. Im at a loss and very frustrated. I have done this before but on different computers...anyway here is the code/response that I'm getting
~/dev/sample_app git push -u origin --all
The authenticity of host 'bitbucket.org (131.103.20.168)' can't be established.
RSA key fingerprint is 81:7b:2c:f5:6f:18:2b:7c:4b:ec:aa:46:46:74:7c:40.
Are you sure you want to continue connecting (yes/no)?
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
~/dev/sample_app
I am on a mac running 10.8.4.
So a little progress has been made, initially there was no .ssh folder so I created that way back in the beginning, there was no known_hosts file so I ran
ssh -T git#bitbucket.org
I chose yes and this created a known_hosts file and when I tried to push again I got:
~/dev/sample_app git push -u origin --all
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
My .ssh folder is 700 and the keys inside are all 600.
You can set IdentityFile flag file in ~/.ssh/config file as follows:
Host bitbucket.org
IdentityFile ~/.ssh/id_rsa
When you run
ssh git#bitbucket.org
the ssh client allows you to selects a file from which the identity (private key) for RSA or DSA authentication is read.
SSH Client To Use Given Private Key ( identity file )
You might be using ssh as the git origin url. Try removing the ssh origin like so
git remote rm origin
Then add new origin with HTTPS url and try pushing again.
git remote add origin https://git#bitbucket.org/SOMETHING/SOMETHING.git
git push -u origin master
Make sure you paste your url from bitbucket as origin.
In my case on fresh Ubuntu 16 machine I was missing files in ~/.ssh folder so what worked:
Go to folder ~/.ssh
Run ssh-keygen and name your file i.e. id_rsa
Run cat ~/.ssh/id_rsa.pub | xclip -sel clip
If you miss xclip just apt-get install xclip :)
Go to (in url change USERNAME to your bitbucket username:) ) https://bitbucket.org/account/user/USERNAME/ssh-keys/
Click Add key and paste the key from the clipboard
Magic - it works now :)
Edit: As Dan Swain points out in the comments, from 1 March 2022 this answer will have been superseded by authentication policy changes: https://bitbucket.org/blog/deprecating-atlassian-account-password-for-bitbucket-api-and-git-activity
The same applies to Github repositories as well, FWIW.
Thanks for the heads-up, Dan.
It might make sysadmins recoil in horror, but after suffering this problem (Windows) I gave up on SSH and went back to HTTPS.
When first adding the remote repository to Git, replace the SSH reference 'git#bitbucket.org...' with the HTTPS URL 'https://<username>#bitbucket.org'.
You have to type your password in every time but, particularly under Windows where SSH is not as commonly available as with the *nix family, I see this as a minor inconvenience compared with the headaches of SSH.
After setting up git with git config --global user.name "My Name" and
git config --global user.email myemail#x.com, I was still having trouble with the Permission Denied, (publickey) error. To solve this, I first generated a new ssh token with
ssh-keygen
and copied it with
pbcopy < ~/.ssh/YOUR_KEY
After that, I went to bitbucket.com to add it as a new SSH key in my settings. Then, I returned to my terminal to add the new key with
ssh-add ~/.ssh/YOUR_KEY.
The big problem that I was having was that I missed the critical ssh-add [key] command.
I had similar problem with BitBucket. in my case, it only fixed after I found out I should remove sudo from git clone command!
According to Attlassian:
You shouldn't use sudo when cloning, pushing, or pulling because the
ssh-agent runs on the user level, not the root level.
A more sustainable solution is to edit .bashrc (e.g. vi ~/.bashrc) and then add the following line to it (replace the key name):
KEY="$HOME/.ssh/YOUR_KEY"
if [ -e "${KEY}" ]; then
ssh-add -q "${KEY}"
fi
This will load the key automatically when you start the shell
If you're using Fedora 33+ and using the RSA algorithm. Use more secure alogrithm like ECDSA or ED25519 instead:
ssh-keygen -t ed25519 -C "your_email#example.com"
Check out the bitbucket support for more details
Cause
The RSA algorithm is being quickly deprecated across operating systems and SSH clients because of various security vulnerabilities, with many of these technologies now outright denying the use of this algorithm.
(info) For example - here is the announcement from OpenSSH regarding
their upcoming deprecation of the ssh-rsa algorithm. In the event that
you are using an operating system or SSH client whose version has this
algorithm disabled, it's possible that any SSH keys previously
generated using this algorithm will no longer be accepted by these
technologies.
Resolution
To fully resolve this issue, our team recommends that these deprecated
keys be re-generated using a supported and more secure algorithm such
as ECDSA and ED25519
I faced same issues in Linux (Ubuntu).
I solved it using setup in git:
git config --global user.name "Your Name"
git config --global user.email your.email#example.com
Printing the public key using cat and SSH key to bitbucket.org:
$ cat ~/.ssh/id_rsa.pub
Adding Bitbucket and pushing up the repository:
git remote add origin git#bitbucket.org:<username>/your repository name.git
git push -u origin --all
That's all!
In my case, this issue happened because I had a number of ssh keys in the ~/.ssh. I had to create a bitbucket.org specific entry in ~/.ssh/config as follows:
Host bitbucket.org
Hostname bitbucket.org
IdentityFile <location-of-.ssh-directory>/bb-rsa
IdentitiesOnly=yes
My guess is that since we don't specify a key while cloning, ssh tries all the keys in ~/.ssh which bitbucket thinks as a hacking attempt and rejects our repo clone attempt.
In my case it solved the problem to add the ssh key from the directory
~/.ssh/id_rsa.pub
on bitbucket.org. I named it also id_rsa.pub on the website.
At the beginning I added another key I created just for bitbucket and named it like that. The first remote actions worked but after some days the request have been denied.
Check for exisiting SSH Key
ls -al ~/.ssh
Copy the SSH Key
cat ~/.ssh/id_rsa.pub | pbcopy
Add the copied SSH Key to 'Bitbucket Settings', 'Security', 'SSH Keys'.
If you have multiple keys in your computer make sure you add bitbucket to the list such as below in
.ssh/config
# Company account
Host company
HostName github.com
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa_accelya
# Personal account
Host personal
HostName github.com
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa_personal
# Personal account bitbucket
Host bitbucket
HostName bitbucket.org
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa_personal_bitbucket
This may be obvious, but I spent quite a bit of time on it.
Check the destination when running git remote -v
In my case I had the ssh keys perfectly set up but the output from this command was:
origin get#github.com:USERNAME/REPOSITORY.git
(notice the get not git)
and not
origin git#github.com:USERNAME/REPOSITORY.git
Again, this was a very particular case, but be sure to check the strings carefully of this system if you're having trouble.
You can fix this with the following commands:
git remote set-url origin git#github.com:USERNAME/REPOSITORY.git
Make sure your have switched to the correct user on terminal.
In my case root user was not the one which has ssh keys added at the bitbucket settings panel. Running git with sudo makes it run from root user and my own user was the one who has keys added.
In my case my issue was that I tried using the .ppk file the putty generated and no matter what I tried nothing worked.
In the end I figured that the it was the wrong file and I had to export it, save it as the id_rsa file and load it, then everything worked.
If any.ssh fix didn't work or you cloned as https there can be a validation issue. in my case, I fixed this error by providing my username and password when cloning the repo. This issue can occur when you are using multiple accounts in a same machine.
use "git clone https://username:password#github.com/username/repository.git" command with your user name and password and repo URL.
I like the Answers here, but they all kind of miss a possible root cause.
with the command:
ssh -T git#bitbucket.org
replace bitbucket.org with your own bitbucket host.
If you get an answer like:
This deploy key has read access to the following repositories:
team-name/repository-name
that is why pushing to the repository is not working.
This you can also double check in the Bitbucket Web UI, notice the read-only access in the description:
Hope this gives a different perspective to the same problem.
I update config file with the top line to get it working
PubkeyAcceptedKeyTypes +ssh-rsa
Host <yourhost>
IdentityFile ~/.ssh/id_rsa
My Solution:
git remote rm origin
Add you user name before #bitbucket.org to the repo URL
git remote add origin https://{USER_NAME}#bitbucket.org/{NAME}/{REPO_NAME}.git
git push -u origin master
In Windows, #efesaid answer worked for solving issues with the ssh connection test. By the way, you can add a -v to see what keys (by name) are being attempted and why the connection fails.
However, when pushing to bitbucket, using git#bitbucket.org:user/repo.git, it seems that the host is not precisely bitbucket.org so I still was getting permission denied problems. I solved them by (re)naming my key to id_rsa (this is the key name that was being attempted in the ssh test).
This works if you have a single rsa key. For multiple keys, perhaps the host in the config file must be
bitbucket.org:username
but I am no sure this is unde
I think that the bitbucket instructions are best. Check if ssh is installed and if not install it
krasen#krasen-Lenovo-Y50-70:~$ ssh -v
usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
[-D [bind_address:]port] [-E log_file] [-e escape_char]
[-F configfile] [-I xxxxx] [-i identity_file]
[-L [bind_address:]port:host:hostport] [-l login_name] [-m mac_spec]
[-O ctl_cmd] [-o option] [-p port]
[-Q cipher | cipher-auth | mac | kex | key]
[-R [bind_address:]port:host:hostport] [-S ctl_path] [-W host:port]
[-w local_tun[:remote_tun]] [user#]hostname [command]
krasen#krasen-Lenovo-Y50-70:~$ ls -a ~/.ssh
. .. google_compute_engine google_compute_engine.pub identity identity.pub known_hosts
krasen#krasen-Lenovo-Y50-70:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/krasen/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/krasen/.ssh/id_rsa.
Your public key has been saved in /home/krasen/.ssh/id_rsa.pub.
The key fingerprint is:
xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx krasen#krasen-Lenovo-Y50-70
The key's randomart image is:
+--[ RSA 2048]----+
| . |
| xx x |
| xxxxx |
| xxxxxxxxx |
| .xxxxxxxx |
| xxxxx |
| xxxxxxxxxxxx|
| xxxxxxxxxxxxx|
| xxxxxxxxxxx |
+-----------------+
krasen#krasen-Lenovo-Y50-70:~$ ls -la ~/.ssh
total 40
drwx------ 2 krasen krasen 4096 Jun 29 14:30 .
drwxr-xr-x 110 krasen krasen 4096 Jun 29 13:00 ..
-rw------- 1 krasen krasen 1675 Mar 18 2015 google_compute_engine
-rw-r--r-- 1 krasen krasen 409 Mar 18 2015 google_compute_engine.pub
-rw------- 1 krasen krasen 1679 Jun 29 13:15 identity
-rw-r--r-- 1 krasen krasen 409 Jun 29 13:15 identity.pub
-rw------- 1 krasen krasen 1679 Jun 29 14:30 id_rsa
-rw-r--r-- 1 krasen krasen 409 Jun 29 14:30 id_rsa.pub
-rw-r--r-- 1 krasen krasen 4698 Jun 29 13:16 known_hosts
krasen#krasen-Lenovo-Y50-70:~$ ssh-agent /bin/bash
to check if the agent is started
krasen#krasen-Lenovo-Y50-70:~$ ps -e | grep [s]sh-agent
26503 ? 00:00:00 ssh-agent
krasen#krasen-Lenovo-Y50-70:~$ ssh-add ~/.ssh/id_rsa
Identity added: /home/krasen/.ssh/id_rsa (/home/krasen/.ssh/id_rsa)
krasen#krasen-Lenovo-Y50-70:~$ ssh-add -l
2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx /home/krasen/.ssh/id_rsa (RSA)
krasen#krasen-Lenovo-Y50-70:~$ cat ~/.ssh/id_rsa.pub
ssh-rsa xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
get this key and add it as key in the bitbucket settings
I got round a similar issue where I had previously used HTTPS to access the repository and had to switch to SSH by setting the url like so;
git remote set-url origin ssh://git#bitbucket.org/...
My problem was to do with permissions.
My project directory was owned by root, but I was logged in as ubuntu. I would get PERMISSION DENIED if I typed in a git command, e.g. git pull origin master, so I used sudo git pull origin master.
I had registered ubuntu's SSH key from /home/ubuntu/.ssh/id_rsa.pub with BitBucket.
However, I was using sudo. So the SSH key used was actually /home/root/.ssh/id_rsa.pub which was different to what BitBucket had.
Solution for my case
chown -R username_here:username_here project/folder/here
Now it should work if you don't prepend sudo
OR give BitBucket root's key
In source tree select your project right click then you find an option "Convert to SSH"-> Repair -> login this solved for me
If you are using SourceTree with Bitbucket, the solution is the next:
Go to your personal Bitbucket settings
Got to App passwords and create an app password
Give the next permissions to the app password:
Repositories (R-W-A-D)
Projects (R-W)
Pull request (R-W)
After that, keep the password generated
Try to clone again your repo
A password popup will be displayed, input the generated password.
That's all.