I've been hitting a wall with this one and i'm not entirely sure what the issue may be. I will be posting two examples of what i'm trying to do one which works and one which doesn't, unfortunately I need the latter to work.
The Error:
Unexpected local arg: /data/
If arg is a remote file/dir, prefix it with a colon (:).
rsync error: syntax or usage error (code 1) at main.c(1215) [receiver=3.0.6]
This Command executes without issues, however it seems that it does not fully rsync my /data directory.
rsync -e "ssh -e 'none'" --force -azPxvIh --delete-after --exclude-from="/tmp/exclude.txt" root#myserver:/ / > "/tmp/wetrun.txt"
This is the command I've been trying utilize to rsync my /data directory and seems to fail.
rsync -e "ssh -e 'none'" --force -azPxvIh --delete-after --exclude-from="/tmp/exclude.txt" root#myserver:/data/ /data/ > "/tmp/wetrun.txt"
Related
I am trying to rsync a local folder to a remote location. This a command that I have run successfully a week ago, but now if i run:
rsync -vrtzu\
--chown=user:webadm
--delete
--exclude-from=.rsyncignore
FOLDER/
USER#REMOTE:/DESTINATION
Then I get the following error message:
zsh:1: no matches found: --usermap=*:USER
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.3]
make: *** [makefile:39: push] Error 12
The command is run from a makefile, hence the last line.
I am using a regular WSL2 Ubuntu shell, not zsh.
I am able to ssh into the remote location with USER#REMOTE.
I have also checked that both locations have rsync installed (same version).
Finally, there is plenty of disk space available on the remote location.
Any pointers? What should I be checking to improve my diagnostic?
Thanks in advance!
This can happen when the remote shell messes with the command. Not sure exactly why and what it does but it modifies escaping so that the file path becomes invalid.
In your case the shell outputs --usermap=*:USER at log in.
The solution is to change the remote (zsh) shell to bash using the chsh command
I'm pretty sure this is an rsync bug:
zsh:1: no matches found: --usermap=*:USER
It only happens the remote machine's default shell is zsh.
It was fixed somewhere between rsync 3.2.3 (where it's broken) and 3.2.5 (where the bug is gone).
You can verify this by passing -vv to rsync. This prints as one of the first output lines which command invocation rsync is doing on the remote server via SSH.
On a broken version, it prints e.g.:
... ssh ... rsync --server -vvnlogDtpRe.LsfxCIvu "--usermap=*:user" "--groupmap=*:webadm"
On a fixed version, it prints e.g.:
... ssh ... rsync --server -vvnlogDtpRe.LsfxCIvu "--usermap=\*:user" "--groupmap=\*:webadm"
As you can see, they inserted a \ to fix the string being interpreted by zsh.
I need to rsync to a remote server using a non-standard SSH port and 2FA which I use via Authy app. The SSH works with this command:
ssh -2 -p 9999 -i /Users/Me/.ssh/id_rsa user#9.9.9.9
This brings up a "Verification Code" prompt in the shell. Which I enter from Authy, and I'm in.
Given the discussion on this a StackOverflow answers I tried this variation of rsync:
rsync -rvz -e 'ssh -p 9999 -i /Users/Me/.ssh/id_rsa \
--progress /src/ user#9.9.9.9.9:/dest/
(Put here on two lines just for legibility, it's one line in my shell command).
This does bring up the Verification Code prompt, which I enter correctly, but then it produces this error:
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(185) [sender=3.1.3]
How can I use rsync with 2FA? Many thanks.
Because #JGK mentioned the answer in the comment, adding answer here for posterity. This "is your shell clean" stuff is shown when remote server is echoing some output upon login, which in my case .bashrc indeed was. I've added a conditional to that echo only to apply when the shell login is "interactive", as mentioned in this Server Fault thread, and it works. Just for easier clarity, the IF condition reads as follows:
if echo "$-" | grep i > /dev/null; then
[any code that outputs text here]
fi
Many thanks.
After using gsutil for more than 1 year I suddenly have this error:
.....
At destination listing 8350000...
At destination listing 8360000...
CommandException: Caught non-retryable exception - aborting rsync
.....
I tried to locate the files with this sync problem but I am not able to do so. Is there a "skip error" option of is there a way I can have gsutil more verbose?
My command line is like this:
gsutil -V -m rsync -d -r -U -P -C -e -x -x 'Download/*' /opt/ gs://mybucket1/kraanloos/
I have created a script to split the problem. This gives me more info for a solution
!#/bin/bash
array=(
3ware
AirTime
Amsterdam
BigBag
Download
guide
home
Install
Holding
Multimedia
newsite
Overig
Trak-r
)
for i in "${array[#]}"
do
echo Processing : $i
PROCESS="/usr/bin/gsutil -m rsync -d -r -U -P -C -e -x 'Backup/*' /opt/$i/ gs://mybucket1/kraanloos/$i/"
echo $PROCESS
$PROCESS
echo ""
echo ""
done
I've been struggling with the same problem the last few days. One way to make it super verbose is to put the -D flag before the rsync argument, as in:
gsutil -D rsync ...
By doing that, I found that my problem is due to having # characters in filenames, as in this question.
In my case, it was because of a broken link to a directory.
As blambert said, use the -D option to see exactly what file causes the problem.
I had struggled with this problem as well. I figured it out now.
you need to re-authenticate your Google Cloud SDK Shell and set a target project again.
It seems like rsync will not show the correct error message.
try cp instead, it will guide you to authentic and set the correct primary project
gsutil cp OBJECT_LOCATION gs://DESTINATION_BUCKET_NAME/
after that, your gsutil rsync should run fine.
I'm trying to rsync files with my server, which has root login via ssh disabled. I'm trying to use one of the users I've created and added to to the wheel group. Usually - when I log in via ssh I run
su - root
then type the password and can run everything as root user, but not quite sure how to do it with rsync as I need to upload the files outside of the user directory.
When I run:
rsync -avz files/* sshwheeluser#ip_address:/home/newproject/public_html/
I get:
rsync: ERROR: cannot stat destination "/home/newproject/public_html/": Permission denied (13)
I have added the following to the /etc/sudoers file:
rsyncuser ALL= NOPASSWD:/usr/bin/rsync
plus I've un-commented:
%wheel ALL=(ALL) NOPASSWD: ALL
I've also tried to run the command with –rsync-path="sudo rsync" as suggested in some posts, but still no luck:
rsync -avz –rsync-path="sudo rsync" files/* sshwheeluser#ip_address:/home/newproject/public_html/
Next I tried to use it with additional -e "ssh":
rsync -avz -e "ssh" –rsync-path="sudo rsync" files/* sshwheeluser#ip_address:/home/newproject/public_html/
which changed the error message (looks like we're getting somewhere):
stdin: is not a tty
sudo: sorry, you must have a tty to run sudo
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /SourceCache/rsync/rsync-40/rsync/io.c(452) [sender=2.6.9]
This post suggests that require_tty and tty_tickets should be disabled in sudoers, but I can't find these options in the /etc/sudoers file.
I've added (as recommended) Defaults !tty_tickets to the sudoers, but still the same message.
Any idea what else I need to do to make it work?
You should check Selinux logs, rsync is probably been blocked by Selinux.
You can also try to run:
semanage permissive -a rsync_t
I need to rsync log files like this:
rsync --progress -rvze ssh name#host:/path/to/folder/*.log
When I run this command though, I get an error:
rsync: getcwd(): No such file or directory (2)
No such file or directory? That's odd. So I try to ssh directly:
ssh name#host
it prompts to enter my name, I do, then I type
cd /path/to/folder
which works fine (log files are present).
I double checked my ssh keys, everything seems to be in order there, but for some reason I can't ssh into a subfolder on this host, so there's no way I can get rsync working correctly.
EDIT:
Running the identical rsync command on my Mac, it works fine. Running it in my ubuntu EC2 instance is still failing.
Are you sure there are any log files at all? If not this command will fail with the 'No such file or directory'
Rather use:
rsync --progress --include='*.log' -rvze ssh name#host: /path/to/folder/ local_folder
The 'direct' ssh syntax you use in your second test is not supported:
ssh name#host:/path/to/folder/
because it will use host:/path/to/folder/ as the hostname.