I'm dealing with a crappy ISP that resets my WAN connection at random points while my script is running. I want the transfer to survive this reset and go on. I manually launch this script vs using cron / launchd currently.
I have a fairly basic script as shown below:
rsync -rltv --progress --partial -e "ssh -i <key> -o ConnectTimeout=300" <remotedir> <localdir>
Am I better off putting the timeout in the rsync section instead?
For example:
rsync -rltv --progress--partial --timeout=300 -e "ssh -i <key>" <remotedir> <localdir>
Thanks!
ConnectTimeout only applies when SSH is trying to establish the connection with the server, it doesn't have anything to do with timeouts during the data transfer. So you need to use the --timeout option to do what you want.
Try re-running the rsync. Also try without the ssh option. The job failed probably due to losing your network connection. I have an rsync job copying files between datacenters running every 2 hours via cron and it will fail about once per day.
Related
To test ssh I read you need to use something like ssh -T user#domain.com:
According to man ssh:
-T Disable pseudo-terminal allocation.
-t Force pseudo-terminal allocation. This can be used to execute arbitrary screen-based programs on a remote
machine, which can be very useful, e.g. when implementing menu services. Multiple -t options force tty alloca-
tion, even if ssh has no local tty.
but I'm still not clear, even after reading this, what the purpose of using -T is when testing SSH.
I read up on pseudo-terminals (https://linux.die.net/man/7/pty) but that didn't seem to help.
To have interactive prompt in your ssh shell, you need to have allocated PTY also on the server side. It is done automatically when you call ssh host.
When you allocate this PTY on server, then your local terminal and the remote one are exchanging some additional messages (Terminal control characters), which give the remote shell information about the size of your local terminal, the remote can update title of your window and so on. This is something you really don't want when you want to transfer files or just pass the output "as it is". It would modify that and you would get generally something else. Again, this is done automatically in case you use scp or just noninteractive script as ssh host my_script.
So far good. You don't need the switches. But things might not be always so simple.
You might want to invoke some interactive shell as a command, for example ssh host /bin/zsh. This would work, but it will not be interactive. In this case, you need to use the -t switch to make it working properly: ssh -t host /bin/zsh.
The same thing can go other way round. You might have server set up, that it will give you some output regardless the command you ask for. In that case, you really don't want to mess it up with any terminal control characters and then you might want to use ssh -T host to avoid this clutch.
You can consider the -T also as the way to safe resources on server and as some "second line of defense". You can disable the TTY allocation in the server configuration, but what if ...
The most common use case for the -T switch is
ssh -T git#github.com
to verify that you have set up your ssh keys properly to authenticate to github.
I have a script which loops through a list of hosts, connecting to each of them with SSH using an RSA key, and then saving the output to a file on my local machine - this all works correctly. However, the commands to run on each server take a while (~30 minutes) and there are 10 servers. I would like to run the commands in parallel to save time, but can't seem to get it working. Here is the code as it is now (working):
for host in $HOSTS; do
echo "Connecting to $host"..
ssh -n -t -t $USER#$host "/data/reports/formatted_report.sh"
done
How can I speed this up?
You should add & to the end of the ssh call, it will run on the background.
for host in $HOSTS; do
echo "Connecting to $host"..
ssh -n -t -t $USER#$host "/data/reports/formatted_report.sh" &
done
I tried using & to send the SSH commands to the background, but I abandoned this because after the SSH commands are completed, the script performs some more commands on the output files, which need to have been created.
Using & made the script skip directly to those commands, which failed because the output files were not there yet. But then I learned about the wait command which waits for background commands to complete before continuing. Now this is my code which works:
for host in $HOSTS; do
echo "Connecting to $host"..
ssh -n -t -t $USER#$host "/data/reports/formatted_report.sh" &
done
wait
Try massh http://m.a.tt/er/massh/. This is a nice tool to run ssh across multiple hosts.
The Hypertable project has recently added a multi-host ssh tool. This tool is built with libssh and establishes connections and issues commands asynchronously and in parallel for maximum parallelism. See Multi-Host SSH Tool for complete documentation. To run a command on a set of hosts, you would run it as follows:
$ ht ssh host00,host01,host02 /data/reports/formatted_report.sh
You can also specify a host name or IP pattern, for example:
$ ht ssh 192.168.17.[1-99] /data/reports/formatted_report.sh
$ ht ssh host[00-99] /data/reports/formatted_report.sh
It also supports a --random-start-delay <millis> option that will delay the start of the command on each host by a random time interval between 0 and <millis> milliseconds. This option can be used to avoid thundering herd problems when the command being run accesses a central resource.
I have a problem while trying to use RSYNC with daemon and SSH connection.
What I wan't to do is simply login to rsync without pass and be able to use the rsync daemon.
Here is my conf file (/etc/rsyncd.conf):
uid = rsync
gid = rsync
[yxz]
path = /home/pierre/xyz
read only = false
auth users = rsync
hosts allow = <myIP>
/home/pierre/xyz has gid wich rsync user can reach.
This is working (but is not using the daemon):
rsync -rzP --stats --ignore-existing --remove-sent-files rsync#mydomain.fr:/home/pierre/xyz/ /media/xyz --include="*.cfg" --exclude="*"
This is not working (using the daemon), but rsync asks me for pass and then says "#ERROR: auth failed on module xyz" because I don't have configure authentification this way :
rsync -rzP --stats --ignore-existing --remove-sent-files rsync://rsync#mydomain.fr/xyz/ /media/xyz --include="*.cfg" --exclude="*"
This is not working (using the daemon):
rsync -rzP -e "ssh -l rsync" --stats --ignore-existing --remove-sent-files rsync://rsync#mydomain.fr/xyz/ /media/xyz --include="*.cfg" --exclude="*"
Here is the error message:
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [Receiver=3.0.9]
With -v option to the ssh command, it says connection is allowed, so I suppose rsync is the problem, not ssh.
Any idee ?
Thanks for your help :)
Make sure that you stop and disable the rsync system service. E.g. if you are using systemd: systemctl disable --now rsync.
Remove -l rsync from the rsync command
rsync -rzP -e "ssh" --stats --ignore-existing --remove-sent-files rsync://mydomain.fr/xyz/ /media/xyz --include="*.cfg" --exclude="*"
Remove auth users = rsync from rsyncd.conf
I found that if I was not using root, I had to also add use chroot = no in rsyncd.conf.
Great it works, but what sort of authentification is made ?
The connection is authenticated as usual for the ssh command (specifically, the same as ssh mydomain.fr).
This does not involve the system service rsync. Instead it uses SSH to start and communicate with an instance of rsync --server --daemon .. You can see this command being started if you replace -e "ssh" with -e "ssh -v".
The problem with using the system service rsync is that it does not encrypt the network connection, so the network is able to intercept and modify the data in transit. This somewhat defeats the point of using any authentication.
Often this approach is used with a dedicated SSH key, using the command="" option in authorized_keys to restrict it to rsync only. A side-benefit of doing so is that it overrides the command rsync tries to use, so you can force it to use --config=~/rsyncd.conf instead of creating a global /etc/rsyncd.conf. IMO this is useful to avoid confusion IMO. It is good practice because if you create the global config file, there is some risk that you will accidentally run the insecure system service. For example Debian 9 enables the rsync system service by default, and will start it automatically at boot if you have created /etc/rsyncd.conf.
https://gist.github.com/trendels/6582e95012f6c7fc6542
https://indico.cern.ch/event/577279/contributions/2354037/attachments/1366772/2071442/Hepsysman-keeping-in-sync.pdf
https://serverfault.com/questions/6367/cant-get-rsync-to-work-in-daemon-over-ssh-mode
Unusual variant using a dedicated user with a custom shell, instead of command="" / ForceCommand, for some reason: http://mennucc1.debian.net/howto-ssh-rsyncd.html
To use rsync daemon without a password, you should remove auth users line from your config file.
uid = rsync
gid = rsync
[yxz]
path = /home/pierre/xyz
read only = false
hosts allow = <myIP>
After starting the daemon, you can refer the module either using :: syntax or using rsync:// prefix as follows
rsync -rzv rsync#mydomain.fr::xyz/ /media/xyz
rsync -rzv rsync://rsync#mydomain.fr/xyz/ /media/xyz
More info: man rsyncd.conf
I'm trying to set up an automatic rsync backup (using cron) over an ssh tunnel but am getting an error "Connection to localhost closed by remote host.". I'm running Ubuntu 12.04. I've searched for help and tried many solutions such as adding ALL:ALL to /etc/hosts.allow, check for #MaxStartups 10:30:60 in sshd_config, setting UsePrivilegeSeparation no in sshd_config, creating /var/empty/sshd but none have fixed the problem.
I have autossh running to make sure the tunnel is always there:
autossh -M 25 -t -L 2222:destination.address.edu:22 pbeyersdorf#intermediate.address.edu -N -f
This seems to be running fine, and I've been able to use the tunnel for various rsync tasks, and in fact the first time I ran the following rsync task via cron it succeeded:
rsync -av --delete-after /tank/Documents/ peteman#10.0.1.5://Volumes/TowerBackup/tank/Documents/
with the status of each file and the output
sent 7331634 bytes received 88210 bytes 40215.96 bytes/sec
total size is 131944157313 speedup is 17782.61
Ever since that first success, every attempt gives me the following output
building file list ... Connection to localhost closed by remote host.
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]
An rsync operation of a smaller subdirectory works as expected. I'd appreciate any ideas on what could be the problem.
It seems the issues is related to autossh. If I create my tunnel via ssh instead of autossh it works fine. I suspect I could tweak the environment variables that affect the autossh configuration, but for my purposes I've solved the problem by wrapping the rsycn command in a script that first opens a tunnel via ssh, executes the backup then kills the ssh tunnel, thereby eliminating the need for the always open tunnel created by autossh:
#!/bin/sh
#Start SSH tunnel
ssh -t -L 2222:destination.address.edu:22 pbeyersdorf#intermediate.address.edu -N -f
#execute backup commands
rsync -a /tank/Documents/ peteman#localhost://Volumes/TowerBackup/tank/Documents/ -e "ssh -p 2222"
#Kill SSH tunnel
pkill -f "ssh.*destination.address"
I'm using cygrunsrv and autossh on windows XP to create a service building a tunnel to a remote server but i also want to create another tunnel from the remote server to another server.
I can achieve it with this command line :
autossh -M 5432 serverA -t 'autossh -M 4321 serverB -N'
but when I want to set it up in cygwin through cygrunsrv to make it works as a service :
cygrunsrv -I TUNNEL -p /usr/bin/autossh -a "-M 5432 serverA -t 'autossh -M 4321 serverB -N'" -e AUTOSSH_NTSERVICE=yes -e AUTOSSH_POLL=20 -e AUTOSSH_GATETIME=30
It's not fully working. The service is creating the tunnel correctly to ServerA but it's not sending the autossh command "autossh -M 4321 serverB -N" to ServerA.
I tried to escape the quote but all my efforts didn't make any difference and I'm not seeing any command sent in the autossh logs.
I think the problem is related to pseudo terminal that is not created through the cygrunsrv.
I'd like to know if there's a way to fix my cygrunsrv command line to make it work or should I consider a different approach ?
Lionel, try removing the AUTOSSH_NTSERVICE=yes from the cygrunsrv invocation. As /usr/share/doc/autossh/README.Cygwin explains:
Setting AUTOSSH_NTSERVICE=yes in the calling environment ...
change[s] autossh's behavior in three useful
ways:
(1) Add an -N flag to each invocation of ssh, thus disabling shell
access. The idea is that if you're running autossh as a system
service, you're using it to forward ports; it wouldn't make sense to
run a shell session as a system service. (If you think this reasoning
is wrong, please send a bug report to the author or Cygwin maintainer,
and tell us what you're trying to do.)
Despite what the above says, it seems that you may have a good reason for not wanting -N (which suppresses command execution) in your service's ssh invocation. Removing AUTOSSH_NTSERVICE=yes should take care of it. It will have a couple of other minor disadvantages, but you can probably live with it. Read the rest of README.Cygwin for the details.