So I have discovered I can do the following with scp via stdin:
Directory creation
scp -tr .
stdin -> D0755 0 <directory_name>
stdin -> \x00
File writing
scp -tr .
stdin -> C<filemode, eg. 0744> <file_size_in_bytes> <filename>
stdin -> actual file bytes
stdin -> \x00
In the man pages I can't find any mention of this, nor have I had luck with googling. Where do I find the spec for these various commands: file creation, directory creation? What else can I do? I'm curious where this is defined. I'm struggling to find where I even found this code initially. Why is there no mention of the -t flag in the scp man page?
scp transfers files by opening an SSH connection to a remote server and invoking another copy of scp on the remote system. The two scp instances communicate through a simple protocol; one instance sends commands and file data; the other instance acts on the commands to store the files on its local system.
The -t option tells scp that it was invoked by another scp instance and that it'll be receiving files. There is another option -f which tells scp that it was invoked by another instance and should send files.
You'd have to ask the OpenSSH developers why the options aren't documented. One might presume that it's because they're not intended for use by humans and so not really part of the user interface.
The best online descriptions of the SCP protocol that I know of are:
How the SCP protocol works
Ruby net-scp source code
OpenSSH scp source code
Related
I am trying to run mpi on an external server. As a part of my goal, I'm attempting to run something in parallel across multiple nodes.
However, this external server has a bad default configuration file that is readonly, such that when I try to ssh to another external server without using ssh <server> -F ~/.ssh/config then it will simply return four different "Bad configuration option"s. However, -F is not an option that I can use for mpirun, and I don't know if there is any way to manually change the ssh configuration file for mpirun.
What should I do?
I'm trying to copy a file from a remote server to my desktop and i'm getting the above error. I've SSH'd to the server.
Here is what i'm doing:
deploy#ip-10-91-135-76 /data/project/current/lib/data $ scp customer_record.ods /Users/James/Desktop
I have very limited experience and don't understand what is going on?
Thanks a lot
man scp tells you how to use scp. In particular, most usages look like:
scp [user1#]host1:]file1 [[user2#]host2:]file2
You can omit putting the user in if its the same as your current user, and likewise for the host. Since you've SSH'd onto the server already, the start of your command is okay to be scp customer_records.ods, but the next argument has to include the user name and host of the target machine that you want to copy the file to, namely your home computer. Chances are you actually want to go the other way, since your home computer may not have a publicly accessible IP.
End the SSH session, go back to your home machine.
Do:
scp <user-you-sshd-as>#<server-you-sshd-to>:/data/project/current/lib/data/customer_records.ods /Users/James/Desktop
If you need to specify a private key, you can use the -i option: scp -i <path-to-key> ...
I have a problem while trying to use RSYNC with daemon and SSH connection.
What I wan't to do is simply login to rsync without pass and be able to use the rsync daemon.
Here is my conf file (/etc/rsyncd.conf):
uid = rsync
gid = rsync
[yxz]
path = /home/pierre/xyz
read only = false
auth users = rsync
hosts allow = <myIP>
/home/pierre/xyz has gid wich rsync user can reach.
This is working (but is not using the daemon):
rsync -rzP --stats --ignore-existing --remove-sent-files rsync#mydomain.fr:/home/pierre/xyz/ /media/xyz --include="*.cfg" --exclude="*"
This is not working (using the daemon), but rsync asks me for pass and then says "#ERROR: auth failed on module xyz" because I don't have configure authentification this way :
rsync -rzP --stats --ignore-existing --remove-sent-files rsync://rsync#mydomain.fr/xyz/ /media/xyz --include="*.cfg" --exclude="*"
This is not working (using the daemon):
rsync -rzP -e "ssh -l rsync" --stats --ignore-existing --remove-sent-files rsync://rsync#mydomain.fr/xyz/ /media/xyz --include="*.cfg" --exclude="*"
Here is the error message:
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
With -v option to the ssh command, it says connection is allowed, so I suppose rsync is the problem, not ssh.
Any idee ?
Thanks for your help :)
Make sure that you stop and disable the rsync system service. E.g. if you are using systemd: systemctl disable --now rsync.
Remove -l rsync from the rsync command
rsync -rzP -e "ssh" --stats --ignore-existing --remove-sent-files rsync://mydomain.fr/xyz/ /media/xyz --include="*.cfg" --exclude="*"
Remove auth users = rsync from rsyncd.conf
I found that if I was not using root, I had to also add use chroot = no in rsyncd.conf.
Great it works, but what sort of authentification is made ?
The connection is authenticated as usual for the ssh command (specifically, the same as ssh mydomain.fr).
This does not involve the system service rsync. Instead it uses SSH to start and communicate with an instance of rsync --server --daemon .. You can see this command being started if you replace -e "ssh" with -e "ssh -v".
The problem with using the system service rsync is that it does not encrypt the network connection, so the network is able to intercept and modify the data in transit. This somewhat defeats the point of using any authentication.
Often this approach is used with a dedicated SSH key, using the command="" option in authorized_keys to restrict it to rsync only. A side-benefit of doing so is that it overrides the command rsync tries to use, so you can force it to use --config=~/rsyncd.conf instead of creating a global /etc/rsyncd.conf. IMO this is useful to avoid confusion IMO. It is good practice because if you create the global config file, there is some risk that you will accidentally run the insecure system service. For example Debian 9 enables the rsync system service by default, and will start it automatically at boot if you have created /etc/rsyncd.conf.
https://gist.github.com/trendels/6582e95012f6c7fc6542
https://indico.cern.ch/event/577279/contributions/2354037/attachments/1366772/2071442/Hepsysman-keeping-in-sync.pdf
https://serverfault.com/questions/6367/cant-get-rsync-to-work-in-daemon-over-ssh-mode
Unusual variant using a dedicated user with a custom shell, instead of command="" / ForceCommand, for some reason: http://mennucc1.debian.net/howto-ssh-rsyncd.html
To use rsync daemon without a password, you should remove auth users line from your config file.
uid = rsync
gid = rsync
[yxz]
path = /home/pierre/xyz
read only = false
hosts allow = <myIP>
After starting the daemon, you can refer the module either using :: syntax or using rsync:// prefix as follows
rsync -rzv rsync#mydomain.fr::xyz/ /media/xyz
rsync -rzv rsync://rsync#mydomain.fr/xyz/ /media/xyz
More info: man rsyncd.conf
I have a script which executes remote command and redirect output to local file.
Remote command just reads list of pcap files continuously and writes to stdout.
The final command is like this -
ssh root#host /sbin/path-to-utility | cat > local-file
The script which executes this remote command needs to have signal handler to save the state of overall transfer.
Also I want to send signal to remote command or process to stop reading pcap files, so that exit after finishing writing current file.
I tried -t option and signal handling works perfectly fine, but it adds some extra characters to the actual output written by remote command and disturbs my pcap data.
Either I need to handle signal without -t option over ssh or I need to find out why ssh -t is adding additional bytes to actual data.
Please help!
Thanks,
Sachin.
The -t option tells ssh to allocate a terminal. The extra characters are intended to be interpreted by your terminal.
Perhaps then you should tell ssh you are using a different terminal, one that will not generate any extra characters, but will still pass on signals. Does this work?
TERM=dumb ssh root#host /sbin/path-to-utility | cat > local-file
(I don't know what would be the best value to use for TERM.)
This question already has answers here:
scp or sftp copy multiple files with single command
(19 answers)
Closed last year.
I would like to know an easy way to use scp to copy files and folders that are present in different paths on my file system. The SSH destination server requests a password and I cannot put this in configuration files. I know that scp doesn't have a password parameter that I could supply from a script, so for now I must copy each file or directory one by one, writing my password every time.
in addition to the already mentioned glob:
you can use {,} to define alternative paths/pathparts in one single statement
e.g.: scp user#host:/{PATH1,PATH2} DESTINATION
From this site:
Open the master
SSHSOCKET=~/.ssh/myUsername#targetServerName
ssh -M -f -N -o ControlPath=$SSHSOCKET myUsername#targetServerName
Open and close other connections without re-authenticating as you like
scp -o ControlPath=$SSHSOCKET myUsername#targetServerName:remoteFile.txt ./
Close the master connection
ssh -S $SSHSOCKET -O exit myUsername#targetServerName
It's intuitive, safer than creating a key pair, faster than creating a compressed file and worked for me!
If you can express all the names of the files you want to copy from the remote system using a single glob pattern, then you can do this in a single scp command. This usage will only support a single destination folder on the local system for all files though. For example:
scp 'RemoteHost:/tmp/[abc]*/*.tar.gz' .
copies all of the files from the remote system which are names (something).tar.gz and which are located in subdirectories of /tmp whose names begin with a, b, or c. The single quotes are to protect the glob pattern from being interpreted from the shell on the local system.
If you cannot express all the files you want to copy as a single glob pattern and you still want the copy to be done using a single command (and a single SSH connection which will ask for your passsword only once) then you can either:
Use a different command than scp, like sftp or rsync, or
Open an SSH master connection to the remote host and run several scp commands as slaves of that master. The slaves will piggyback on the master connection which stays open throughout and won't ask you for a password. Read up on master & slave connections in the ssh manpage.
create a key pair, copy the public key to the server side.
ssh-keygen -t rsa
Append content inside the file ~/.ssh/identity.pub to file ~/.ssh/authorized_keys2 of server side user. You need not to type password anymore.
However, be careful! anybody who can access your "local account" can "ssh" to the server without password as well.
Alternatively, if you cannot use public key authentication, you may add the following configuration to SSH (either to ~/.ssh/config or as the appropriate command-line arguments):
ControlMaster auto
ControlPath /tmp/ssh_mux_%h_%p_%r
ControlPersist 2m
With this config, the SSH connection will be kept open for 2 minutes so you'll only need to type the password the first time.
This post has more details on this feature.