For instance, I have tried this (notice sources is remote):
scp root#$node:/sourcepath/sourcefile.log /destinationpath/destinationfile.log
The other option is to rename the file afterwards, but would be more convenient to do it on the fly while the data is downloaded via scp, therein my question. Thanks.
Maybe without scp:
ssh yourserver "cat >tmpfile && mv tmpfile datafile" <datafile
This command copies the "datafile" file to a remote server under the name "tmpfile".
Only after successful copy renames the temporary file "tmpfile" to the right name "datafile" on remote host.
If copying was not successful, the remote host will be only a temporary file.
Thus, you are protected from getting no full "datafile" file.
Sorry for my English.
Related
The Bitvise SSH Client version history states that v8.15 supports directory mirroring:
The graphical SSH Client and sftpc now support recursive directory mirroring. A directory and all of its subdirectories and files can be synchronized either in the upload or download direction.
I can find it in the GUI, but I can't find how to do using sftpc.exe. There is no mention of mirroring in sftpc.exe -help.
How can I do directory mirroring from the command line?
You point out a tangential design issue in sftpc: getting help for SFTP commands requires you to use sftpc interactively and connect to the server. You can then get help from the interactive prompt.
This is inconvenient, so I opened a feature request for us to make the interactive help available from the command line, as well.
The help text you are looking for is as follows - for the put command:
sftp> help put
USAGE: put local-path [remote-path] [-bg | -fg] [-s] [-o] [-r]
[-f] [-noTime] [-m=mode] [-dm=mode] [-mirror [-erase]]
[-b | -lf | -std | -tlf | -t]
DESCRIPTION: Upload file.
PARAMETERS:
-bg Start (queue) upload in background.
-fg Start upload in foreground.
-s Include subdirectories (recursive).
-r Synchronize file content. If synchronization is not available,
resume existing incomplete files using a heuristic resume.
Heuristic resume MAY result in an inconsistent destination file
if the destination file content has been modified in the middle.
-o Synchronize file content. If synchronization is not available,
force existing file to be overwritten. If -r is also specified,
heuristic resume is tried first.
-del Remove local file after successful upload.
-f Assume remote-path is a file (not a directory)
-noTime Do not synchronize file modification times.
-m=mode Set the access mode for remote files to 'mode'.
-dm=mode Set the access mode for new remote directories to 'mode'.
If directory already exists, access mode will not be changed.
-mirror Mirror local-path to remote-path. Local files that do not exist
remotely will be uploaded. Remote files that are different than
their local versions will be overwritten.
-erase With -mirror, erase remote files that are not present locally.
FILE TRANSFER MODE - if present, overrides mode selected with "type":
-b Upload files as binary; no conversions.
-lf Auto-detect text files. In text files, replace CRLF with LF.
Binary files are unaffected.
-std Auto-detect text files. Upload text files using the SFTP v4+ text
file transfer mechanism. Binary files are unaffected. Not
available when SFTP version 3 or lower is in use.
-tlf Upload all files as textual. Replace all CRLF bytes with LF.
-t Upload all files using the SFTP v4+ text file transfer mechanism.
Not available when SFTP version 3 or lower is in use.
And for the get command:
sftp> help get
USAGE: get remote-path [local-path] [-bg | -fg] [-s] [-o] [-r]
[-f] [-noTime] [-lit] [-mirror [-erase]]
[-b | -lf | -std | -tlf | -t]
DESCRIPTION: Download file.
PARAMETERS:
-bg Start (queue) download in background.
-fg Start download in foreground.
-s Include subdirectories (recursive).
-r Synchronize file content. If synchronization is not available,
resume existing incomplete files using a heuristic resume.
Heuristic resume MAY result in an inconsistent destination file
if the destination file content has been modified in the middle.
-o Synchronize file content. If synchronization is not available,
force existing file to be overwritten. If -r is also specified,
heuristic resume is tried first.
-del Remove remote file after successful download.
-f Assume remote-path is a file (not a directory).
-noTime Do not synchronize file modification times.
-lit Treat remote-path literally (not a wildcard pattern).
-mirror Mirror remote-path to local-path. Remote files that do not exist
locally will be downloaded. Local files that are different than
their remote versions will be overwritten.
-erase With -mirror, erase local files that are not present remotely.
FILE TRANSFER MODE - if present, overrides mode selected with "type":
-b Download files as binary; no conversions.
-lf Auto-detect text files. In text files, replace LF with CRLF.
Binary files are unaffected.
-std Behaves same as -lf when downloading. Not available when SFTP
version 3 or lower is in use.
-tlf Download all files as textual. Replace all LF bytes with CRLF.
-t Download all files using the SFTP v4 text file transfer mechanism.
Not available when SFTP version 3 or lower is in use.
I hope this helps!
I don't normally monitor Stack Overflow, so please feel free to call my attention by opening a support case with Bitvise if you need me to look at something else.
I recommend also using the latest Bitvise SSH Client version. Currently, this is 8.35. It's free of charge for use in any environment, and we try to ensure that each version is a strict upgrade that does not introduce new difficulties. We want there to be no reason to stay behind. :-)
A similar question has been asked before but this one is different.
I am doing a rsync of a single file on remote server and destination directory does not exist. I would like the destination directory to be created if it does not exist. I am using :: syntax which uses modules and I could not find a similar case in the forums.
Here is the syntax. remote_dir2 does not exist and i want it to be created.
rsync -avz --password-file=<file> <source-file> remote-user#remote-server::remote_dir1/remote_dir2
Note: there is a module named remote-user in /etc/rsyncd.conf in the remote server and connection and everything else works, except that source file ends up in the remote_dir1 with the name of remote_dir2
Is there any solution that is different from what mentioned below ?
I do not want to open an ssh to remote server to 'mkdir'
I do not want to use -R, --relative because directory structure names in source and destination are very different.
I also know that there is a trick mentioned
here but it does not work when you specify a module. There is no error or anything in the logs, apparently it gets ignored.
I have a text file with some data and wants to import data to Greenplum database.After online research , I found that its better to use COPY command if your data size is small. So i decided to use this.
Here is the scenario:
I have placed my Text file at location /bin/bash /data , I can access this file using terminal, but once I run the following COPY sql script at Greenplum database it's says :
could not open file "/bin/bash /data/data.txt" for reading: No such file or directory
Below is the my sql script:
COPY userdata(customerid,time,trans,quantity) from '/bin/bash /data/data.txt' WITH DELIMITER ',';
From Greenplum database documentation I found the following line :
The COPY source file must be accessible to the master host. Specify the COPY source file name relative to the master host location.
But I do not know how to make it accessible to master host and relative to master host location.
The path to your file doesn't make any sense.
/bin/bash /data/data.txt is certainly not a valid name for a path.
If you data.txt file is located in the /data folder with content
in the following format :
12345,5:32AM,air,2
67890,6:42PM,rail,4
You could use the below command :
COPY userdata(customerid,time,trans,quantity) FROM '/data/data.txt' WITH DELIMITER AS ',';
Also sql user you should have the permission to access the the data.txt from the location /data folder.
Perhaps do a ls -l and check if the sql user can read files from data.txt
I want to take backup of my website which is hosted on godaddy.
I used pscp command from my windows dos and try to download whole public_html folder.
my command is :
pscp -r user#host:public_html/ d:\sites\;
Files are downloading properly and folders also. But the issue is public_html and other subfolders has two folder like "./" and "../". Due to these two folders my copy is getting failed and I am getting
"security violation: remote host attempted to write to " a '.' or '..' path!"error.
Hope any one can help for this.
Note : I have only ssh access and have to download it from ssh commands itself.
Appending a star to the source should fix it, e.g.
pscp -r user#host:public_html/* d:\sites\;
Also you can do same thing by not adding '/' at the end of your source path.
For eg.
pscp -r user#host:public_html d:\sites
Above command will create public_html directory if not exists at your destination (i.e. d:\sites).
Simply we can say using above command we can make a as it is clone of public_html at d:\sites.
One important thing: You need to define the port number over here "-P 22".
pscp -r -P 22 user#host:public_html/* D:\sites
In my case, it works when I use port number 22 with the above script.
I would like to copy a directory of files from a remote server. As it is a large number of files, the option of ignoring existing files on the destination server is desirable.
Unfortunately, rsync is not available for some reason (the remote server is from a CDN service, and beyond my control).
So I think I am stuck using scp -r on the folder in question.
Is there anyway of doing this with ignoring existing files?
thanks
You could also create a *.tar.gz or *.tar.bz2 archive, scp it, and then unpack it. I don't know if scp -r uses any compression. If not, compressing everything first might, potentially, make it faster.
It's easy to write an script in Perl to do that using the module Net::SFTP::Foreign:
#!/usr/bin/perl
use Net::SFTP::Foreign;
my $sftp = Net::SFTP::Foreign->new('user#host');
$sftp->die_on_error;
$sftp->rget('/remote/path', '/local/path',
resume => 'auto',
on_error => sub { my ($sftp, $e) = #_;
warn "error processing $e->{filename}: "
. $sftp->error;
}
);
SCP needs a writable file so that it can replace that file.
Using this, for the files which you do not want to replace, you can remove the permission to write for them. And continue with your scp for all files.
https://unix.stackexchange.com/a/51932/284063