I have about a thousand files on a remote server (all in different directories). I would like to scp them to my local machine. I would not want to run scp command a thousand times in a row, so I have created a text file with a list of file locations on the remote server. It is a simple text file with a path on each line like below:
...
/iscsi/archive/aat/2005/20050801/A/RUN0010.FTS
/iscsi/archive/aat/2006/20060201/A/RUN0062.FTS
/iscsi/archive/aat/2013/20130923/B/RUN0010.FTS
/iscsi/archive/aat/2009/20090709/A/RUN1500.FTS
...
I have searched and found someone trying to do a similar but not the same thing here. The command I would like to edit is below:
cat /location/file.txt | xargs -i scp {} user#server:/location
In my case I need something like:
cat fileList.txt | xargs -i scp user#server:{} .
To download files from a remote server using the list in fileList.txt located in the same directory I run this command from.
When I run this I get an error: xargs: illegal option -- i
How can I get this command to work?
Thanks,
Aina.
You get this error xargs: illegal option -- i because -i was deprecated. Use -I {} instead (you could also use a different replace string but {} is fine).
If the list is remote, the files are remote, you can do this to retrieve it locally and use it with xargs -I {}:
ssh user#server cat fileList.txt | xargs -I {} scp user#server:{} .
But this creates N+1 connections, and more importantly this copies all remote files (scattered in different directories you said) to the same local directory. Probably not what you want.
So, in order to recreate a similar hierarchy locally, let's say everything under /iscsi/archive/aat, you can:
use cut -d/ to extract the part you want to be identical on both sides
use a subshell to create the command that creates the target directory and copies the file there
Thus:
ssh user#server cat fileList.txt \
| cut -d/ -f4- \
| xargs -I {} sh -c 'mkdir -p $(dirname {}); scp user#server:/iscsi/archive/{} ./{}'
Should work, but that's starting to look messy, and you still have N+1 connections, so now rsync looks like a better option. If you have passwordless ssh connection, this should work:
rsync -a --files-from=<(ssh user#server cat fileList.txt) user#server:/ .
The leading / is stripped by rsync and in the end you'll get everything under ./iscsi/archive/....
You can also copy the files locally first, and then:
rsync -a --files-from=localCopyOfFileList.txt user#server:/ .
You can also manipulate that file to remove for example 2 levels:
rsync -a --files-from=localCopyOfFileList2.txt user#server:/iscsi/archive .
etc.
Related
tar zcf - source-file | ssh root#remote-server-IP "cat > remote-server-location.tar.gz"
The command is perfect , but I want to understand the command.
I got the part where it pipes the output to ssh, but why is cat command used for?
Can anyone help me understand?
Thanks.
The cat is used to grab the information which is sent via ssh and redirect it to file on remote system. You just can't do it on this way:
tar zcf - source-file | ssh root#remote-server-IP "> remote-server-location.tar.gz"
because this will only create empty remote-server-location.tar.gz file.
If you try without quotes:
tar zcf - source-file | ssh root#remote-server-IP > remote-server-location.tar.gz
this will create the file on local filesystem
Whenever I try to SCP files (in bash), they end up in a seemingly random(?) order.
I've found a simple but not-very-elegant way of keeping a desired order, described below. Is there a clever way of doing it?
Edit: deleted my early solution from here, cleaned, adapted using other suggestions, and added as an answer below.
To send files from a local machine (e.g. your laptop) to a remote (e.g. your calculation server), you can use Merlin2011's clever solution:
Go into the folder in your local machine where you want to copy files from.
Execute the scp command, assuming you have an access key for the remote server:
scp -r $(ls -rt) user#foo.bar:/where/you/want/them/.
Note: if you don't have a public access key it may be better to do something similar using tar, then send the tar file, i.e. tar -zcvf files.tar.gz $(ls -rt), and then send that tar file on its own using scp.
But to do it the other way around you might not be able to run the scp command directly from the remote server to send files to, say, your laptop. Instead, you may need to, let's say bring files into your laptop. My brute-force solution is:
In the remote server, cd into the folder you want to copy files from.
Create a list of the files in the order you want. For example, for reverse order of creation (most recent copied last):
ls -rt > ../filenames.txt
Now you need to add the path to each file name. Before you go up to the directory where the list is, print the path using pwd. Now do go up: cd ..
You now need to add this path to each file name in the list. There are many ways to do this, here's one using awk:
cat filenames.txt | awk '{print "path/to/files/" $0}' > delete_me.txt
You need the filenames to be in the same line, separated by a space, so change newlines to spaces:
tr '\n' ' ' < delete_me.txt > filenames.txt
Get filenames.txt to the local server, and put it in the folder where you want to copy the files into.
The scp run would be:
scp -r user#foo.bar:"$(cat filenames.txt)" .
Similarly, this assumes you have a private access key, otherwise it's much simpler to tar the file in the remote, and bring that.
One can achieve file transfer with alphabetical order using rsync:
rsync -P -e ssh -r user#remote_host:/some_path local_path
P allows partial downloading, e sets the SSH protocol and r downloads recursively.
You can do it in one line without an intermediate using xargs:
ls -r <directory> | xargs -I {} scp <Directory>/{} user#foo.bar:folder/
Of course, this would require you to type your password multiple times if you do not have public key authentication.
You can also use cd and still skip the intermediate file.
cd <directory>
scp $(ls -r) user#foo.bar:folder/
What does the ssh-copy-id command do, exactly? I've used it numerous times and it works great. However, when I try to manually cut and paste my .pub keyfile to my remote authorized_keys, it doesn't work.
I've compared the contents of my authorized_keys file where I've cut and pasted the .pub into it vs subsequently using ssh-copy-id and I'm not seeing any differences between the two, including whitespace.
Is there anything that ssh-copy-id does beyond copying the public key into authorized_keys?
This little one liner script works on sh, bash, and zsh. I use it every time there is no ssh-copy-id, for example when I'm on older version of OSX.
cat ~/.ssh/id_rsa.pub | ssh <user>#<hostname> 'cat >> ~/.ssh/authorized_keys'
How it works
I am sending the public keay to the Unix standard output (STDOUT) using the cat command. I then connect the STDOUT of cat to the standard input (STDIN) of the ssh.
The ssh executes the cat command on the server. Remember that the we have our key in the STDIN now? This key gets passed from ssh to the cat command executed on a server. The >> operator redirects the STDOUT of the cat to the end of the ~/.ssh/authorized_keys file. This way the key from public keys is appended to the authorized_keys on the server.
IMO It's better than manual copying and pasting: in this case you know exactly what content will end up in the file
I usually copy-paste keys into authorized_keys as you describe (I forget about ssh-copy-id), so it can work. Note thatchmod 600 ~/.ssh/authorized_keys is required if you're creating the file.
ssh-copy-id is a shell script so you can open it in a text editor to see what it does, this looks like the relevant bit:
printf '%s\n' "$NEW_IDS" | ssh "$#" "
umask 077 ;
mkdir -p .ssh && cat >> .ssh/authorized_keys || exit 1 ;
if type restorecon >/dev/null 2>&1 ; then restorecon -F .ssh .ssh/authorized_keys ; fi"
restorecon in the last line restores default SELinux security contexts. I haven't had to run that, but it might be necessary in your case.
I'm trying to run rsync over ssh in parallel to transfer files between two machines for evaluation purposes. I wanna see how faster can I get compared to a single rsync process.
I tried these two solutions:
https://wiki.ncsa.illinois.edu/display/~wglick/Parallel+Rsync but with no great success.
https://gist.github.com/rcoup/5358786 (I couldn't make it work)
Based on the first link I run a command like this:
ssh HOST "mkdir -p ~/destdir/basefolder"
cd ./basefolder; ls | xargs -n1 -P 4 -I% rsync -arvuz -e ssh % HOST:~/destdir/basefolder/.
and I get the files transfered, but it doesn't seem to work well... In this case, It will run a process for every file and folder in the basefolder, but when it finds a folder, it will transfer everything inside that folder using only 1 process.
I tried to use find -type f, but I got problems because I loose the file hierarchy.
Does anyone how some methods to do what I want? (Use rsync in parallel over ssh while keeping files and folders hierarchy).
Since you tagged your question 'gnu-parallel' the obvious is to refer you to http://www.gnu.org/software/parallel/man.html#EXAMPLE:-Parallelizing-rsync
cd src-dir; find . -type f -size +100000 | parallel -v ssh fooserver mkdir -p /dest-dir/{//}\;rsync -Havessh {} fooserver:/dest-dir/{}
I'm trying to automate a script that copies a file from my local server to a remote server on the command line. I've done the research on scp and know how to copy the file to the remote server, but then I want to append that file to another.
This is my code:
scp ~/file.txt user#host:
ssh user#host cat file.txt >> other_file.txt
When I enter everything into the command line manually as such, everything works fine:
scp ~/file.txt user#host:
ssh user#host
cat file.txt >> other_file.txt
But when I run the script, only the file is copied, not appended to the end of other_file.txt. Help?
The second line of your code should be
ssh user#host "cat file.txt >> other_file.txt"
Three important points:
You don't want your local shell to interpret >> in any way (which it does if it's unquoted)
There is a remote shell which will interpret >> in the command correctly.
Final arguments to ssh are "joined" to form a command, not carried into an argv array as they are. It may be convenient but it also may lead to confusion or bugs: ssh cat "$MYFILE" and ssh "cat '$MYFILE'" both work in a common use case, but they both break for different values of $MYFILE.
You need to enclose the command to be run on the remote host in quotes. Otherwise, the redirection is being done locally rather than remotely. Try this instead:
scp ~/file.txt user#host:
ssh user#host 'cat file.txt >> other_file.txt'
Try this:
$ cat file.txt| ssh hostname 'cat >> other_file.txt'