I am trying to copy data from SFTP to blob but got stuck when creating SFTP source.
I have the connection details and can easily connect on Filezilla or WinSCP. However, I am unable to get it to work in Azure data factory.
I am not using code but the user interface.
The connection details on the page creating the SFTP source:
Connect via integration runtime: AutoResolveIntegrationRuntime (default)
Host: xyz
Port: 22 (can't remove it as it doesn't like it)
SSH Host Key Validation: Enable SSH Host Key Validation
SSH Host Key Finger-print: taken from WinSCP - Session - Server/protocol information
Authentication type: SSH Public Key Authentication -can't use basic as the private key holds the security info
User name:XXX
Private Key Type: Use Key Content
Private key content: loaded the .ppk file, tried also tried loading the .pem file and got different errors
Pass Phrase: none
When setting up this sftp in WinSCP or FileZilla it automatically converted the provided .pem file into .ppk.
When I loaded the .ppk file into ADF I got an error: Invalid Sftp credential provided for 'SshPublicKey' authentication type.
When I loaded the .pem file I got: Meet network issue when connect to Sftp server 'spiderftp.firstgroup.com', SocketErrorCode: 'TimedOut'.
I have also tried 'Disable SSH Host Key Validation' in SSH Host Key Validation and made no difference.
I have also opened the .ppk file in PuttyGen and used that host key finger print and still no luck.
Only getting these 2 errors depending on which file I load.
Can't find anything about this online so would be grateful for some advice.
Have you read this note in this doc?
https://learn.microsoft.com/en-us/azure/data-factory/connector-sftp#using-ssh-public-key-authentication
SFTP connector supports RSA/DSA OpenSSH key. Make sure your key file content starts with "-----BEGIN [RSA/DSA] PRIVATE KEY-----". If the private key file is a ppk-format file, please use Putty tool to convert from .ppk to OpenSSH format.
Got this working today. Like you, could connect using WinSCP and failed when using ADF.
The link Fang Liu shared contains our answers, but my issue was not the private key. I suspect Fang's suggestion resolved your problem and I'm sharing my answer here to help others who may encounter similar.
My issue:
When using Private Key Authentication in ADF the password becomes a Pass Phrase and you no longer have the ability to supply a password. To overcome the problem we disabled password authentication for the user and the SFTP connection started working.
As stated in the documentation. The Pass Phrase is used to decrypt the private key if it is encrypted.
Also worth noting:
If you store the contents of the private key in Key Vault you need
to base64 encode the entire contents of the exported key and use
that string. This includes "-----BEGIN RSA PRIVATE KEY-----" and the
end. The same applies if you want to paste the value into the
textbox of the SFTP linked service edit screen.
I did not try to manually edit the JSON of the Linked Service to explicitly provide a password and this could be workaround for someone to test if they are unable to disable the password.
I used PuTTYGen to export the PPK to a private key and had the same fingerprint issue too so I just disabled cert validation. Funnily you can use the fingerprint provided by the error and it passes validation so not sure where the bug lies. :-)
Related
I am trying to automate a backup process with WinSCP. I am trying to set up a WinSCP script, but I have troubles with the SSH authentication.
My very simple script is this:
# Connect to SFTP server using a password
open sftp://myusername:mypw#mywebspace.com/ -hostkey="ssh-rsa 2048 xxxxxxxxxxx...="
# Upload file
put C:\pic.jpg /backup/
# Exit WinSCP
exit
The part I have troubles with is the -hostkey switch. I generated a key with PuTTYgen, and saved it on my webspace. Yet I don't know what to put into my script. Everything I try seems to get a "authentication failed" when running the script.
This is what I get in PuTTYgen:
Which of this should I put in the script? What exactly does that look like?
I'm grateful for all helpful advice, as I a'm new to this.
The host key is a public key of your SSH server. It has nothing to do with the key pair you generate for authentication to the server. What goes to the -hostkey switch is a fingerprint of the server's public key – Just follow the switch documentation. You use it to verify the server identity to avoid MITM attacks.
See WinSCP FAQ Where do I get SSH host key fingerprint to authorize the server?.
As you seem to get to the authentication phase ("authentication failed"), your immediate problem is not the host key. The authentication happens only after verifying the host key. As you seem to put a wrong value to the -hostkey switch, yet the host key is verified – it means that you have the host key cached in registry anyway. If you have problem with authentication, please first make sure you can login to the SSH server anyhow (e.g. from WinSCP GUI). And only then try to automate the process.
See also Understanding SSH key pairs.
I have been able to set up SSH access to my Google Cloud Platform VM via SFTP using WinSCP, but I now wish to do the same using another VM.
I have tried the ssh-keygen -t rsa , ssh-copy-id demo#198.51.100.0 method but always come up against the "Permission denied (public key)" error which from researching seems to be a pretty widespread issue with few reliable fixes (all the ones I tried didn't work).
I used PuttyGen to create the public and private key, and inserted the public key onto the server just through GCP settings, adding it under the SSH settings for my instance.
I am just confused on what to do with the private key when simply trying to sftp through the terminal on a separate VM, as before I would load the private key into WinSCP settings. Is there a folder I need to place it in or?
Regarding your first issue of "Permission denied (public key)" error, please follow the troubleshooting in this link and this.
About your other question of "what to do with the private key when simply trying to sftp through the terminal", that depends on the settings of the specific the 3rd party SFTP tool you are using. To locate the locations of SSH key after generating them, please review this document.
Once you have added the public key in the VM, you would need to boot the VM for public key to take effect. Try rebooting it and try
I am trying to access my droplet though ssh, however i keep getting the same message Disconnected: No supported authentication methods available (server sent: publickey). I searched for online solutions - changing the mod of the .ssh folder and authorization_keys, changing sshd_config file but nothing has worked.
Initially, i used the public key in the digitalcloud interface and after that manually added a new ssh key though console panel.
Could you help me with solving this problem?
I had the same thing happen to me but my situation was I had a ppk file loaded in pageant, plus referenced in putty. But what bit me was I also had an id_rsa file under the $USER_HOME/.ssh folder. Most everything will defer to id_rsa as your default private key. I renamed id_rsa to id_rsa-backup and retried. Problem went away.
Check your public key, might have a copy/past error
You need to download the puttyGEN and generate a public and a private key save the private and public keys to a file. Then run pageant.exe and right-click on the pageant icon in the notification area and add the private key you just saved. Now add the public key to the server in your account.
If you follow the GitHub HowTo "Generating SSH Keys", you get three files in your ~/.ssh directory: known_hosts, id_rsa, and id_rsa.pub.
The file known_hosts is used for the server authentication, id_rsa is used for the client authentification (here is an article, that explains the difference).
Why should I create / why GitHub does need both -- a host and a user authentification files? How does the GitHub authentification work?
Thx
This is just plain old SSH authentication; nothing about it is specific to GitHub.
id_rsa and id_rsa.pub are the two halves of your key: the private key and the public key. Effectively, the public key is the lock for the private key. You put the lock (public key) on whatever servers you want easy access to, without too much worry that someone else will see it, because it's just a lock. You keep the (private) key on your machine, and use it to log into those servers; they see you have a key fitting the lock, and let you in.
(Not to say that you should put your public key on completely untrustworthy machines; there are malicious tricks that can take advantage of shortcuts like ssh -A.)
known_hosts doesn't actually have much to do with this; it's just where ssh stores the fingerprints of all the servers you've connected to, so it can throw up a big scary warning if the fingerprint changes. (That would mean it's not the same machine: either something has changed radically on the server side, or your connection has been hijacked.)
So, anyway, one of the protocols Git itself understands is SSH. When you use git#github.com:... as a repository URL, Git is just connecting over SSH. Of course, GitHub doesn't want you mucking around on their machines, so they only let you do Git things, not get a full shell.
As usual, the Arch wiki has a whole lot more words on this.
known_hosts stores the server's identity the first time you connect, so that you know the next time that you're connecting to the same server. This prevents someone from pretending to be the server the next time you connect (but sadly not the first time)
id_rsa is your secret key that proves that you are really you. Never give this away.
id_rsa.pub is the public key, its purpose for authentication is basically just to prove that you have the secret key without giving it out. This key you can give to anyone what needs it since there's nothing secret about it.
When you connect to the server, SSH first checks that the server has the correct key (ie it should match the one in known hosts. If the client is comfortable that the server is genuine, it uses its private key to sign the following data and sends it to the server;
string session identifier
byte SSH_MSG_USERAUTH_REQUEST
string user name
string service name
string "publickey"
boolean TRUE
string public key algorithm name
string public key to be used for authentication
The server verifies the signature using the public key (which you earlier uploaded to Github), and if it is correct, the client is authenticated.
The known_hosts file is used by ssh whenever you actually connect to a host via SSH. It stores a signed key of sorts for the server. Then, if it changes, you will know.
ssh-keygen -t rsa -C yourgithub#accountemail.com is used to generate the SSH key in which you will give the id_rsa.pub to github. Then, when you connect to github you have the private key id_rsa in your ~/.ssh folder which is then used to validate your information with github.
This is a very low-level explanation, but the private key (non .pub) file is your end, the .pub is for github and the known_hosts is for your box to know what is what.
You can also generate a config file in ~/.ssh for use to specify which key goes to which host..
authorized_keys and known_hosts are entirely different..
Your SSH server (sshd, ie) uses authorized_keys, or whatever file is defined within your /etc/ssh/sshd_config/ for knowing the public side of another key. So when a user connects to your server, they pass their private key, your SSH server verifies against the public key it has within authorized_keys and if it doesn't match, it doesn't work.
Github maintains an authorized_keys so-to-speak on their users. Your public key goes into your authorized_keys on your account and then when you connect via ssh to clone,push,etc, it checks your private key you send over with your public key they already know.
I am a newbie to Amazon web services, was trying to launch an Amazon instance and SSH to it using putty from windows. These are the steps I followed:
Created a key pair.
Added a security group rule for SSH and HTTP.
Launched and instance of EC2 using the above key pair and security group.
Using PuTTYgen converted the *.pem file to *.ppk
Using putty tried connecting to the public DNS of the instance and provided the *.ppk file.
I logged in using 'root' and 'ec2-user', and created the PPK file using SSH1 and SSH2, for all these attempts I get the following error in putty,
"Server refused our key"
Can you guys please help, any suggestions would be greatly appreciated.
I assume that the OP figured this out or otherwise moved on, but the answer is to use ubuntu as the user (if the server is ubuntu).
1) Make sure you have port 22 (SSH) opened in Security Group of EC2 Instance.
2) Try connecting with Elastic IP instead of public DNS name.
I hope you have followed these steps Connecting EC2 from a Windows Machine Using PuTTY
Another situation where I got the "Server refused our key" error when using putty, from windows, to ssh to an EC2 instance running ubuntu:
The private key was wrongly converted from .pem to .ppk.
puttygen has two options for "converting keys".
Load your .pem file into puttygen using the File->Load Private Key option and then save as .ppk file using the Save Private Key Button.
DO NOT use the menu option Conversions->Import Key to load the .pem file generated by EC2.
See the puttygen screenshots below, with the two menu options marked.
Check the username, it should be "ubuntu" for your machine.
Check if traffic is enabled on port 22 in Security group.
Check if you are using the correct url i.e ubuntu#public/elasticip
Maybe worth of checking one more thing. Go to AWS console, right mouse click on the instance and choose "Connect...". It will show you the DNS name that you want to use. If you restarted that instance at some point, that DNS name could have changed.
I had a similar problem when I tried to connect an instance created automatically by the Elastic Beanstalk service (EBS). But, once I linked my existing key name to the EBS (under Environment Details -> Edit Configuration -> Server Tab -> Existing Key Pair), I was able to login with 'ec2-user' and my existing key file (converted to .ppk) with putty.
This, however, terminates the running instance and rebuilds a new instance with access through the key pair named above.
Just in case it helps anyone else, I encountered this error after changing the permissions on the home folder within my instance. I was testing something and had executed chmod -R 777 on my home folder. As soon as this had occurred, once I had logged out I was effectively locked out.
You won't face this error if you SSH AWS directly using ".pem" file instead of converted ".ppk" file.
1) Use Git Bash instead of putty. Since you can run all the Linux commands in Git Bash. By installing Git you get to access Git Bash Terminal
2) Right click from the folder where you have ".pem" and select "Git Bash Here".
3) Your key must not be publicly viewable for SSH to work. So run "chmod 400 pemfile.pem".
4) Connect to your instance using its Public DNS - "ssh -i "pemfile.pem" ec2-user#ec2-x-x-x-x.us-west-1.compute.amazonaws.com"
5) Make sure to whitelist your Network IP for SSH in your_instance->security_group->inbound_rules
I assume you're following this guide, and connecting using the instructions on the subsequent page. Verify a couple of things:
You converted the key correctly, e.g. selected the right .pem file, saved as private key, 1024-bit SSH-2 RSA
The Auth settings (step 4 in the connection tutorial) are correct
I was having the same trouble (and took the same steps) until I changed the user name to 'admin' for the debian AMI I was using.
You should lookup the user name ofthe AMI you are using. The debian AMI is documented here
http://wiki.debian.org/Cloud/AmazonEC2Image/Squeeze
I have had this same problem. The AMI you are using is the one that is also used by the "Cloud Formation" templating solution.
In the end I gave up with that, and created a Red Hat instance. I was then able to connect by SSH fine using the user root.
The instructions here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html work fine using a Red Hat instance but not using an Amazon Linux instance. I assume they have some username that I didn't think to try (root, ec2-user, and many other obvious ones, all were refused)
Hope that helps someone!
I use Debain AMI and I try ec2-user, root but correct login is 'admin'.
I was getting the same error when I tried to create a new key pair and tried to use that new pem/ppk file. I noticed that the Key Pair Name field on the instance was still the old one and in poking around. Apparently, you can't change a key pair. So I went back to the original key pair. Fortunately, I didn't delete anything so this was easy enough.
Try an alternative SSH client, like Poderosa. It accepts pem files, so you will not need to convert the key file.
If you already have a key pair, follow these steps:
Convert *.pem to *.ppk using PuTTYgen (Load pem file key then Save ppk)
Add ppk auth key file to Putty SSH>Auth options
Enter "Host Name (or IP address)" field: ubuntu#your-ip-address-of-ubuntu-ec2-host))