I need to select and download many folders stored on a computer that I only have access to with remote ssh connection. I create a list ("list.txt") to download only the folders of my interest, I
tried using a "for" loop with
for i in "list.txt"; do
scp -r /pwd/of/folder/of/origen/ /pwd/of/folder/destiny;
done
But don´t read my list and dowload all folders,
also I tried with
for i in "list.txt"; do
rsync -Pavizuh /pwd/of/folder/of/origen/$i /pwd/of/folder/destiny;
done
but send mensagge:
sent 29 bytes received 20 bytes 98.00 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files could not be transferred (code 23) at /System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Sources/rsync/rsync-55/rsync/main.c(996) [sender=2.6.9]
building file list ...
rsync: link_stat "/Users/rtorres/daniela/Proyecto/Anotacion/Strain9998" failed: No such file or directory (2)
0 files to consider
What can I do ?
Thanks !
You want the contents of "list.txt", not the literal name "list.txt".
for i in $(cat list.txt); do
scp -r server:/pwd/of/folder/of/origen/$i /pwd/of/folder/destiny/$i
done
Related
I try to copy files with rsync and i have checked multiple times that the address is correct but still i get feedback: no such directory. The code is:
rsync -azvrP -e ssh master_123#165.x.x.115:/applications/123/public_html/wp-content/uploads/2023/ /applications/321/public_html/wp-content/uploads/2023/
What could be the error?
I am conducting both experiments below with empty /home/pantelis folder (which is the destination directory)
This command succeeds:
rsync -zalP --progress --exclude=.git --exclude=.vscode /Users/pantelis/Workspace/my-work/terragrunt/modules/ my-server:/home/pantelis/my-work/
i.e. on my-server, my-work directory is created and has the contents of /Users/pantelis/Workspace/my-work/terragrunt/modules/
On the remote machine, I now delete /home/pantelis/my-work so /home/pantelis is once again empty.
I try to run the rsync command as follows which now fails
▶ rsync -zalP --progress --exclude=.git --exclude=.vscode /Users/pantelis/Workspace/my-work/terragrunt/modules/ my-server:/home/pantelis/my-work/terragrunt/modules/
building file list ...
1114 files to consider
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Sources/rsync/rsync-55/rsync/io.c(453) [sender=2.6.9]
I am trying this since, apparently I want the remote file structure to match the local one.
Why is it failing in the second attempt?
It it because for (some inherent reason) rsync cannot create any other dir than the leaf? (my-work)? In that case I have tried the --relative option as suggested here but with no success whatsoever.
Add this to your command to create missing directory hierarchy on my-server:
--rsync-path="mkdir -p /home/pantelis/my-work/terragrunt/modules/ && rsync"
Assuming there are folders and files like below
/master1/dir1/dir1_1/file1
/master1/dir1/dir1_1/file2
/master1/dir1/dir1_1/file2
/master1/dir1/dir1_1/file2
/master1/dir1/dir1_1/file2
/master1/dir1/dir1_2/file1
/master1/dir1/dir1_2/file2
/master1/dir1/dir1_2/file2
/master1/dir1/dir1_2/file2
/master1/dir1/dir1_2/file2
/master1/dir2/dir2_1/file1
/master1/dir2/dir2_1/file2
/master1/dir2/dir2_1/file2
/master1/dir2/dir2_1/file2
/master1/dir2/dir2_1/file2
/master1/dir2/dir2_2/file1
/master1/dir2/dir2_2/file2
/master1/dir2/dir2_2/file2
/master1/dir2/dir2_2/file2
/master1/dir2/dir2_2/file2
I have a terminal server from where password-less authentication is enabled for all the servers in my landscape and I would like to transfer files from one server to another.
When I run SCP with -3 option on the terminal server I was able to copy the files from one server to another and now I have to exclude certain files with file name file1 or under directory dir1_1
I am using the below command inside a Perl script to copy all the files
scp -o StrictHostKeyChecking=no -3 -v -r $srcHostname:$srcPath/* $tgtHostname:$tgtPath
In rsync I was unable to get a similar option like (scp -3) so i need to use only scp
how do i exclude a file pattern in SCP
I have an S3 bucket mounted locally at /mnt/s3 using s3fs.
I can manually cp -r /my-dir/. /mnt/s3, and the file testfile.txt in /mnt/s3 will be overwritten as expected, without error.
However, when using rsync to do this, I get errors about unlinking and copying if the file already exists in the bucket. (If a file of the same name does not exist in the bucket, it's copied properly, without any errors.)
$ rsync -vr --temp-dir=/tmp/rsync /my-dir/. /mnt/s3
sending incremental file list
testfile.txt
rsync: unlink "/mnt/s3/testfile.txt": Operation not permitted (1)
rsync: copy "/tmp/rsync/testfile.txt.Kkyy5n" -> "testfile.txt": Operation not permitted (1)
sent 274 bytes received 428 bytes 1,404.00 bytes/sec
total size is 95 speedup is 0.14
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]
I'm using --temp-dir because otherwise rsync was copying temporary files into /mnt/s3 and trying to rename them to their permanent names. However, rsync failed to rename them, and also failed to delete the temporary files, resulting in improperly copied files and lots of clutter in the S3 bucket.
You may want to try the rsync --inplace flag (instead of the temp_dir workaround), as per the writeup here:
https://baldnerd.com/preventing-rsync-from-doubling-or-even-tripling-your-s3-fees/
I have a specific list of files that I need to copy from a remote server. Is this possible with SCP?
I know I can copy individual files using scp {user_name}#{host}:{filepath} . , but is there any way to take a .csv or .txt and run a foreach loop?
while read file; do scp "user#host:$file" .; done < files
I found that it was easier to use tar with a list then send the files individually via scp:
tar -czvf archive.tar.gz -T file-list.txt && scp archive.tar.gz user#host:/path/
I use this on systems that don't have rsync available, this way you also avoid iterating password prompts or TCP/SSH connection limits.