I have to fetch the files from hdfs using python code and Im using paramiko client connection to run the command on hadoop file system.
A code snippet as follows
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(AutoAddPolicy())
ssh.connect(server,
username=username,
password=password)
ssh.exec_command('hdfs dfs -cat '+ file_path)
This works fine and i can get the file content from hdfs as required. However, if i run this for multiple files in sequence or run other commands to find or list files, after a few commands it starts failing with error message
Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
Even if I try to do kinitad before running each command, still i get this error randomly.
How can i get rid of this error?
Related
I am implementing a python script, which uses paramiko to connect to a hadoop cluster. My problem is that I can SSH to a root user only, and from inside I have to switch user to hdfs to execute my command.
now I need something to automate this switching to HDFS user and then cding into /tmp/ and then executing command from there. I have tried invoke_shell() , it hangs, and also the && inside the exec_command, it also doesnt work.
I am getting a permission denied exception:
java.io.FileNotFoundException: file.txt (Permission denied)
There are two workflows that I have thought of:
1st one:
1. sudo -u hdfs -s
2. cd /tmp/
3. <execute the command> <outputDir>
2nd one:
sudo -u hdfs <execution command> /tmp/<outputDir>
The first one doesnt give the above error. But the second one throws this. I was trying second one just to avoid the dependent command issue.
Any help or suggestions will be appreciated.
I am able to ssh on my windows 10 computer using ubuntu subsystem when I run the command:
/mnt/c/Users/marti/Downloads$ sudo ssh -i credfile.pem ec2-user#3.333.33.333
Where I have a file located at /mnt/c/Users/marti/Downloads/credfile.pem
I am trying to get VSCode's ssh extension to work but it keeps failing with an error "The process tried to write to a nonexistent pipe"
In my vscode settings it points to my ssh config file:
In order to connect I click the green bottom left corner of my vscode window and select the first option 'remote-ssh connect current window to host'
I paste in my command, hit enter, it prompts me to choose either mac/win/linux and I pick linux.
But it then just leads to an error saying it tried to write to a nonexistent pipe?
[20:50:27.876] "install" terminal command done
[20:50:27.877] Install terminal quit with output: The process tried to write to a nonexistent pipe.
[20:50:27.877] Received install output: The process tried to write to a nonexistent pipe.
[20:50:27.883] Resolver error: Error: The process tried to write to a nonexistent pipe
Is there something else I need to setup or change for my vscode ssh connection to work?
Instead of pasting your command, you should code the same command in your .ssh/config file.
Add in it:
Host ec2
Hostname 3.333.33.333
User ec2-user
IdentityFile C:\path\to\credfile.pem
Then you will be able to select the entry 'ec2' after selecting 'remote-ssh connect current window to host'
I met this problem when I was trying to config Jenkins to fetch the repository on the server of a fellow(university course).
Started by user anonymous
Building in workspace /var/lib/jenkins/jobs/TEST1/workspace
$ hg clone --rev default --noupdate ssh://XXX#partch.anu.edu.au/XXX/XXX /var/lib/jenkins/jobs/TEST1/workspace
remote: Host key verification failed.
abort: no suitable response from remote hg!
ERROR: Failed to clone ssh://XXX#partch.anu.edu.au/XXX/XXX
ERROR: Failed to clone ssh://XXX#partch.anu.edu.au/XXX/XXX
Finished: FAILURE
Then I tried to do the same command in terminal, with the different error
abort: Permission denied: /var/lib/jenkins/jobs/TEST1/workspace
I tried to change the location for the command:
hg clone --rev default --noupdate ssh://XXX#partch.anu.edu.au/XXX/XXX /home/administrator/TEST
And it works
According to all these, and the fact that this is no workspace(Error: no workspace)--I know it's because of no successful build ever.
Is it a problem with the workspace configuration of Jenkins? but don't know what to do.
Thank you very much if anyone gives me some advice!
Your error doesn't mention anything about a workspace. There error clearly states that it comes from remote hg and it is an authentication failure.
Also, are you using the Mercurial Plugin to perform the SCM checkout or trying to replicate it with a Build Shell build step?
I'm running PuTTy ssh on windows in the hope to copy a file from a remote server.
The command I used is below:
rsync -avz user#server:/home/user/imitate/tool /home/tool
But I received the error:
rsync: mkdir "/home/tool" failed: Permission denied (13)
rsync error: error in file IO (code 11) at main.c(605) [Receiver=3.0.9]
I'm totally new to this and I have no idea what is happening. So I logged into an account on a remote server using ssh, and want to copy files from that remote server to the laptop I'm using. What should I do?
you cannot write to /home/tool locally on your windows box. either run the command with more privileges or download to another directories where you have write permissions.
I have a linux shell script file which collects various data from linux server. (Services, Process, FreeSpace etc.).
From windows to collect the data we are using Plink to connect to linux Boxes and run the shell script
plink root#servername -pw Password -noagent -m Batch-File.
and using pscp to copy the file to windows location.
Now when I try to do the same for Esxi the plink command fails with the error below.
FATAL ERROR: Server unexpectedly closed network connection
though If i give a direct command as below.
plink root#servername -pw Password -noagent ls /etc
works out.
Let me know how to use the plink for esxi .. if possible.
After seeing the messages log it looks like that the issue is with esxi's limitation to read long character string. The message log fails in the session with String Too Long and then post a message of closing the connection.
Thus the approach was to copy the shell script as a pscp connection, run the file with executable permission and collect the data gathered and delete the file from system.