I have a Arduino Yun and want setup the server for Yun.
So what I want is to copy a folder that contain a py file and a index.html to my Yun
I used mac terminal to do this operation
the command looks like this
scp -r /Users/gudi/Desktop/LobsterHeartRate root#192.168.240.1:/mnt/sda1
and then terminal asked for the password
after I typed, it shows
scp: /mnt/sda1/LobsterHeartRate: Not a directory
I didn't type /mnt/sda1/LobsterHeartRate why it shows this error
Your code
scp -r /Users/gudi/Desktop/LobsterHeartRate root#192.168.240.1:/mnt/sda1
requires that the remote directory /mnt/sda1 exists. This looks like it is not true in your case. Check it using ssh root#192.168.240.1 ls /mnt/sda1.
scp is simple tool and it does not allow you to rename directories on the fly and the target directory must exists. You might try
scp -r /Users/gudi/Desktop/LobsterHeartRate root#192.168.240.1:/mnt/
ssh root#192.168.240.1 mv /mnt/LobsterHeartRate /mnt/sda1
or so, if it will suit your needs. But copying more files, rsync is usually more suitable. Check its manual page and give it a try next time.
As #Jens Höpken notes, your post is a bit sparse. But trying to read between the lines of your post I suspect that LobsterHeartRate is a DIRECTORY on your local system but a FILE named LobsterHeartRate in your target system. This might be happening right at the top of the directory tree, or perhaps you have directories/files of the same name further down the tree. scp -rv might help resolve any confusions here.
Beware: scp -r resolves symbolic links. If you want to preserve symlinks you need to do something else. For historic reasons I use the following, though cpio with a find front-end opens up interesting possibilities for fine-grained file selections.
( cd /Users/gudi/Desktop && tar -cf - LobsterHeartRate ) |
ssh root#192.168.240.1 'cd /mnt/sda1 && tar -xf -'
For a safe "dry run" you could change the -xf to a -tf. The && chains are required to prevent bad things from happening if any prior command fails.
Disclaimer: any debugging is left as an exercise for the student.
Whenever I try to SCP files (in bash), they end up in a seemingly random(?) order.
I've found a simple but not-very-elegant way of keeping a desired order, described below. Is there a clever way of doing it?
Edit: deleted my early solution from here, cleaned, adapted using other suggestions, and added as an answer below.
To send files from a local machine (e.g. your laptop) to a remote (e.g. your calculation server), you can use Merlin2011's clever solution:
Go into the folder in your local machine where you want to copy files from.
Execute the scp command, assuming you have an access key for the remote server:
scp -r $(ls -rt) user#foo.bar:/where/you/want/them/.
Note: if you don't have a public access key it may be better to do something similar using tar, then send the tar file, i.e. tar -zcvf files.tar.gz $(ls -rt), and then send that tar file on its own using scp.
But to do it the other way around you might not be able to run the scp command directly from the remote server to send files to, say, your laptop. Instead, you may need to, let's say bring files into your laptop. My brute-force solution is:
In the remote server, cd into the folder you want to copy files from.
Create a list of the files in the order you want. For example, for reverse order of creation (most recent copied last):
ls -rt > ../filenames.txt
Now you need to add the path to each file name. Before you go up to the directory where the list is, print the path using pwd. Now do go up: cd ..
You now need to add this path to each file name in the list. There are many ways to do this, here's one using awk:
cat filenames.txt | awk '{print "path/to/files/" $0}' > delete_me.txt
You need the filenames to be in the same line, separated by a space, so change newlines to spaces:
tr '\n' ' ' < delete_me.txt > filenames.txt
Get filenames.txt to the local server, and put it in the folder where you want to copy the files into.
The scp run would be:
scp -r user#foo.bar:"$(cat filenames.txt)" .
Similarly, this assumes you have a private access key, otherwise it's much simpler to tar the file in the remote, and bring that.
One can achieve file transfer with alphabetical order using rsync:
rsync -P -e ssh -r user#remote_host:/some_path local_path
P allows partial downloading, e sets the SSH protocol and r downloads recursively.
You can do it in one line without an intermediate using xargs:
ls -r <directory> | xargs -I {} scp <Directory>/{} user#foo.bar:folder/
Of course, this would require you to type your password multiple times if you do not have public key authentication.
You can also use cd and still skip the intermediate file.
cd <directory>
scp $(ls -r) user#foo.bar:folder/
I am on Windows 7 - Sourcetree 1.4.1.0 - Embedded Mercurial 2.6.1
Target is a private mercurial repo hosted on bitbucket.
How do I enable SSH compression so that my transactions are faster?
A quick Google search yielded this document:
Edit the Mercurial global configuration file (~/.hgrc). Add the following line to the UI section:
ssh = ssh -C
When you are done the file should look similar to the following:
[ui]
# Name data to appear in commits
username = Mary Anthony <manthony#atlassian.com>
ssh = ssh -C
On Windows, the Mercurial settings file is located here:
C:\Users\{username}\AppData\Local\Atlassian\SourceTree\hg_local\Mercurial.ini
The contents of the file are actually not to be changed, as its header explains:
; System-wide Mercurial config file.
;
; !!! Do Not Edit This File !!!
;
; This file will be replaced by the installer on every upgrade.
; Editing this file can cause strange side effects on Vista.
;
; http://bitbucket.org/tortoisehg/stable/issue/135
;
; To change settings you see in this file, override (or enable) them in
; your user Mercurial.ini file, where USERNAME is your Windows user name:
;
; XP or older - C:\Documents and Settings\USERNAME\Mercurial.ini
; Vista or later - C:\Users\USERNAME\Mercurial.ini
I don't have a Mac, so I can't test this, but this Atlassian answer states that the location of this file for Mac is:
/Applications/SourceTree.app/Contents/Resources/mercurial_local/hg_local/
In my case, I'm using TortoiseHg, but the concept should be the same.
Here is my original c:\somerepo\.hg\hgrc file:
[paths]
default = ssh://hg#bitbucket.org/someuser/somerepo
So what's happening with ssh? Let's debug a pull statement, hg pull --debug on the command-line. I noticed it is running C:\Program Files\TortoiseHg\lib\TortoisePlink.exe instead of ssh to make the call:
PS C:\somerepo> hg pull --debug
pulling from ssh://hg#bitbucket.org/someuser/somerepo
running "C:\Program Files\TortoiseHg\lib\TortoisePlink.exe" -ssh -2 hg#bitbucket.org "hg -R someuser/somerepo serve --stdio"
sending hello command
sending between command
abort: no suitable response from remote hg!
So let's just reuse the call, add compression (yay!), non-interactive (batch) and our key:
[paths]
default = ssh://hg#bitbucket.org/someuser/somerepo
[ui]
ssh = "C:\Program Files\TortoiseHg\lib\TortoisePlink.exe" -ssh -2 -C -batch -i "c:\keys\somekey.ppk"
I am running into this error of:
$ git push heroku master
Warning: Permanently added the RSA host key for IP address '50.19.85.132' to the list of known hosts.
! Your key with fingerprint b7:fd:15:25:02:8e:5f:06:4f:1c:af:f3:f0:c3:c2:65 is not authorized to access bitstarter.
I tried to add the keys and I get this error below:
$ ssh-add ~/.ssh/id_rsa.pub
Could not open a connection to your authentication agent.
Did You Start ssh-agent?
You might need to start ssh-agent before you run the ssh-add command:
eval `ssh-agent -s`
ssh-add
Note that this will start the agent for msysgit Bash on Windows. If you're using a different shell or operating system, you might need to use a variant of the command, such as those listed in the other answers.
See the following answers:
ssh-add complains: Could not open a connection to your authentication agent
Git push requires username and password (contains detailed instructions on how to use ssh-agent)
How to run (git/ssh) authentication agent?.
Could not open a connection to your authentication agent
To automatically start ssh-agent and allow a single instance to work in multiple console windows, see Start ssh-agent on login.
Why do we need to use eval instead of just ssh-agent?
SSH needs two things in order to use ssh-agent: an ssh-agent instance running in the background, and an environment variable set that tells SSH which socket it should use to connect to the agent (SSH_AUTH_SOCK IIRC). If you just run ssh-agent then the agent will start, but SSH will have no idea where to find it.
from this comment.
Public vs Private Keys
Also, whenever I use ssh-add, I always add private keys to it. The file ~/.ssh/id_rsa.pub looks like a public key, I'm not sure if that will work. Do you have a ~/.ssh/id_rsa file? If you open it in a text editor, does it say it's a private key?
I tried the other solutions to no avail. I made more research and found that the following command worked. I am using Windows 7 and Git Bash.
eval $(ssh-agent)
More information in: https://coderwall.com/p/rdi_wq (web archive version)
The following command worked for me. I am using CentOS.
exec ssh-agent bash
Could not open a connection to your authentication agent
To resolve this error:
bash:
$ eval `ssh-agent -s`
tcsh:
$ eval `ssh-agent -c`
Then use ssh-add as you normally would.
Hot Tip:
I was always forgetting what to type for the above ssh-agent commands, so I created an alias in my .bashrc file like this:
alias ssh-agent-cyg='eval `ssh-agent -s`'
Now instead of using ssh-agent, I can use ssh-agent-cyg
E.g.
$ ssh-agent-cyg
SSH_AUTH_SOCK=/tmp/ssh-n16KsxjuTMiM/agent.32394; export SSH_AUTH_SOCK;
SSH_AGENT_PID=32395; export SSH_AGENT_PID;
echo Agent pid 32395;
$ ssh-add ~/.ssh/my_pk
Original Source of fix:
http://cygwin.com/ml/cygwin/2011-10/msg00313.html
MsysGit or Cygwin
If you're using Msysgit or Cygwin you can find a good tutorial at SSH-Agent in msysgit and cygwin and bash:
Add a file called .bashrc to your home folder.
Open the file and paste in:
#!/bin/bash
eval `ssh-agent -s`
ssh-add
This assumes that your key is in the conventional ~/.ssh/id_rsa location. If it isn't, include a full path after the ssh-add command.
Add to or create file ~/.ssh/config with the contents
ForwardAgent yes
In the original tutorial the ForwardAgent param is Yes, but it's a typo. Use all lowercase or you'll get errors.
Restart Msysgit. It will ask you to enter your passphrase once, and that's it (until you end the session, or your ssh-agent is killed.)
Mac/OS X
If you don't want to start a new ssh-agent every time you open a terminal, check out Keychain. I'm on a Mac now, so I used the tutorial ssh-agent with zsh & keychain on Mac OS X to set it up, but I'm sure a Google search will have plenty of info for Windows.
Update: A better solution on Mac is to add your key to the Mac OS Keychain:
ssh-add -K ~/.ssh/id_rsa
Simple as that.
Run
ssh-agent bash
ssh-add
To get more details you can search
ssh-agent
or run
man ssh-agent
ssh-add and ssh (assuming you are using the openssh implementations) require an environment variable to know how to talk to the ssh agent. If you started the agent in a different command prompt window to the one you're using now, or if you started it incorrectly, neither ssh-add nor ssh will see that environment variable set (because the environment variable is set locally to the command prompt it's set in).
You don't say which version of ssh you're using, but if you're using cygwin's, you can use this recipe from SSH Agent on Cygwin:
# Add to your Bash config file
SSHAGENT=/usr/bin/ssh-agent
SSHAGENTARGS="-s"
if [ -z "$SSH_AUTH_SOCK" -a -x "$SSHAGENT" ]; then
eval `$SSHAGENT $SSHAGENTARGS`
trap "kill $SSH_AGENT_PID" 0
fi
This will start an agent automatically for each new command prompt window that you open (which is suboptimal if you open multiple command prompts in one session, but at least it should work).
I faced the same problem for Linux, and here is what I did:
Basically, the command ssh-agent starts the agent, but it doesn't really set the environment variables for it to run. It just outputs those variables to the shell.
You need to:
eval `ssh-agent`
and then do ssh-add. See Could not open a connection to your authentication agent.
Instead of using ssh-agent -s, I used eval `ssh-agent -s` to solve this issue.
Here is what I performed step by step (step 2 onwards on Git Bash):
Cleaned up my .ssh folder at C:\user\<username>\.ssh\
Generated a new SSH key:
ssh-keygen -t rsa -b 4096 -C "xyz#abc.com"
Check if any process id(ssh agent) is already running.
ps aux | grep ssh
(Optional) If found any in step 3, kill those
kill <pids>
Started the SSH agent
$ eval `ssh-agent -s`
Added SSH key generated in step 2 to the SSH agent
ssh-add ~/.ssh/id_rsa
Try to do the following steps:
Open Git Bash and run: cd ~/.ssh
Try to run agent: eval $(ssh-agent)
Right now, you can run the following command: ssh-add -l
In Windows 10 I tried all answers listed here, but none of them seemed to work. In fact, they give a clue. To solve a problem, simply you need three commands. The idea of this problem is that ssh-add needs the SSH_AUTH_SOCK and SSH_AGENT_PID environment variables to be set with the current ssh-agent sock file path and pid number.
ssh-agent -s > temp.txt
This will save the output of ssh-agent in a file. The text file content will be something like this:
SSH_AUTH_SOCK=/tmp/ssh-kjmxRb2764/agent.2764; export SSH_AUTH_SOCK;
SSH_AGENT_PID=3044; export SSH_AGENT_PID;
echo Agent pid 3044;
Copy something like "/tmp/ssh-kjmxRb2764/agent.2764" from the text file and run the following command directly in the console:
set SSH_AUTH_SOCK=/tmp/ssh-kjmxRb2764/agent.2764
Copy something like "3044" from the text file and run the following command directly in the console:
set SSH_AGENT_PID=3044
Now when environment variables (SSH_AUTH_SOCK and SSH_AGENT_PID) are set for the current console session, run your ssh-add command and it will not fail again to connect to ssh agent.
One thing I came across was that eval did not work for me using Cygwin, what worked for me was ssh-agent ssh-add id_rsa.
After that I came across an issue that my private key was too open, the solution I managed to find for that (from here):
chgrp Users id_rsa
as well as
chmod 600 id_rsa
finally I was able to use:
ssh-agent ssh-add id_rsa
For Windows users, I found cmd eval `ssh-agent -s` didn't work, but using Git Bash worked a treat:
eval `ssh-agent -s`; ssh-add KEY_LOCATION
And making sure the Windows service "OpenSSH Key Management" wasn't disabled.
To amplify on n3o's answer for Windows 7...
My problem was indeed that some required environment variables weren't set, and n3o is correct that ssh-agent tells you how to set those environment variables, but doesn't actually set them.
Since Windows doesn't let you do "eval," here's what to do instead:
Redirect the output of ssh-agent to a batch file with
ssh-agent > temp.bat
Now use a text editor such as Notepad to edit temp.bat. For each of the first two lines:
Insert the word "set" and a space at the beginning of the line.
Delete the first semicolon and everything that follows.
Now delete the third line. Your temp.bat should look something like this:
set SSH_AUTH_SOCK=/tmp/ssh-EorQv10636/agent.10636
set SSH_AGENT_PID=8608
Run temp.bat. This will set the environment variables that are needed for ssh-add to work.
I just got this working. Open your ~/.ssh/config file.
Append the following-
Host github.com
IdentityFile ~/.ssh/github_rsa
The page that gave me the hint Set up SSH for Git
said that the single space indentation is important... though I had a configuration in here from Heroku that did not have that space and works properly.
If you follow these instructions, your problem would be solved.
If you’re on a Mac or Linux machine, type:
eval "$(ssh-agent -s)"
If you’re on a Windows machine, type:
ssh-agent -s
I had the same problem on Ubuntu and the other solutions didn't help me.
I finally realized what my problem was. I had created my SSH keys in the /root/.ssh folder, so even when I ran ssh-add as root, it couldn't do its work and kept saying:
Could not open a connection to your authentication agent.
I created my SSH public and private keys in /home/myUsername/ folder and I used
ssh-agent /bin/sh
Then I ran
ssh-add /home/myUsername/.ssh/id_rsa
And problem was solved this way.
Note: For accessing your repository in Git, add your Git password when you are creating SSH keys with ssh-keygen -t rsa -C "your Git email here".
Let me offer another solution. If you have just installed Git 1.8.2.2 or thereabouts, and you want to enable SSH, follow the well-writen directions.
Everything through to Step 5.6 where you might encounter a slight snag. If an SSH agent is already be running you could get the following error message when you restart bash
Could not open a connection to your authentication agent
If you do, use the following command to see if more than one ssh-agent process is running
ps aux | grep ssh
If you see more than one ssh-agent service, you will need to kill all of these processes. Use the kill command as follows (the PID will be unique on your computer)
kill <PID>
Example:
kill 1074
After you have removed all of the ssh-agent processes, run the px aux | grep ssh command again to be sure they are gone, then restart Bash.
Voila, you should now get something like this:
Initializing new SSH agent...
succeeded
Enter passphrase for /c/Users/username/.ssh/id_rsa:
Now you can continue on Step 5.7 and beyond.
This will run the SSH agent and authenticate only the first time you need it, not every time you open your Bash terminal. It can be used for any program using SSH in general, including ssh itself and scp. Just add this to /etc/profile.d/ssh-helper.sh:
ssh-auth() {
# Start the SSH agent only if not running
[[ -z $(ps | grep ssh-agent) ]] && echo $(ssh-agent) > /tmp/ssh-agent-data.sh
# Identify the running SSH agent
[[ -z $SSH_AGENT_PID ]] && source /tmp/ssh-agent-data.sh > /dev/null
# Authenticate (change key path or make a symlink if needed)
[[ -z $(ssh-add -l | grep "/home/$(whoami)/.ssh/id_rsa") ]] && ssh-add
}
# You can repeat this for other commands using SSH
git() { ssh-auth; command git "$#"; }
Note: this is an answer to this question, which has been merged with this one.
That question was for Windows 7, meaning my answer was for Cygwin/MSYS/MSYS2. This one seems for some Unix, where I wouldn't expect the SSH agent needing to be managed like this.
The basic solution to run ssh-agent is answered in many answers. However runing ssh-agent many times (per each opened terminal or per remote login) will create a many copies ot ssh-agent running in memory. The scripts which is suggested to avoid that problem is long and need to write and/or copy separated file or need to write too many strings in ~/.profile or ~/.schrc. Let me suggest simple two string solution:
For sh, bash, etc:
# ~/.profile
if ! pgrep -q -U `whoami` -x 'ssh-agent'; then ssh-agent -s > ~/.ssh-agent.sh; fi
. ~/.ssh-agent.sh
For csh, tcsh, etc:
# ~/.schrc
sh -c 'if ! pgrep -q -U `whoami` -x 'ssh-agent'; then ssh-agent -c > ~/.ssh-agent.tcsh; fi'
eval `cat ~/.ssh-agent.tcsh`
What is here:
search the process ssh-agent by name and by current user
create appropriate shell script file by calling ssh-agent and run ssh-agent itself if no current user ssh-agent process found
evaluate created shell script which configure appropriate environment
It is not necessary to protect created shell script ~/.ssh-agent.tcsh or ~/.ssh-agent.sh from another users access because: at-first communication with ssh-agent is processed through protected socket which is not accessible to another users, and at-second another users can found ssh-agent socket simple by enumeration files in /tmp/ directory. As far as about access to ssh-agent process it is the same things.
In Windows 10, using the Command Prompt terminal, the following works for me:
ssh-agent cmd
ssh-add
You should then be asked for a passphrase after this:
Enter passphrase for /c/Users/username/.ssh/id_rsa:
Try the following:
ssh-agent sh -c 'ssh-add && git push heroku master'
Use parameter -A when you connect to server, example:
ssh -A root#myhost
from man page :
-A Enables forwarding of the authentication agent connection.
This can also be specified on a per-host basis in a configuration file.
Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's
UNIX-domain socket) can access the local agent through the forwarded
connection. An attacker cannot obtain key material from the agent,
however they can perform operations on the keys that enable them to
authenticate using the identities loaded into the agent.
I had this problem, when I started ssh-agent, when it was already running. It seems that the multiple instances conflict with each other.
To see if ssh-agent is already running, check the value of the SSH_AGENT_SOCK environment variable with:
echo $SSH_AGENT_SOCK
If it is set, then the agent is presumably running.
To check if you have more than one ssh-agent running, you can review:
ps -ef | grep ssh
Of course, then you should kill any additional instances that you created.
Read user456814's answer for explanations. Here I only try to automate the fix.
If you using a Cygwin terminal with Bash, add the following to the $HOME/.bashrc file. This only starts ssh-agent once in the first Bash terminal and adds the keys to ssh-agent. (I am not sure if this is required on Linux.)
###########################
# start ssh-agent for
# ssh authentication with github.com
###########################
SSH_AUTH_SOCK_FILE=/tmp/SSH_AUTH_SOCK.sh
if [ ! -e $SSH_AUTH_SOCK_FILE ]; then
# need to find SSH_AUTH_SOCK again.
# restarting is an easy option
pkill ssh-agent
fi
# check if already running
SSH_AGENT_PID=`pgrep ssh-agent`
if [ "x$SSH_AGENT_PID" == "x" ]; then
# echo "not running. starting"
eval $(ssh-agent -s) > /dev/null
rm -f $SSH_AUTH_SOCK_FILE
echo "export SSH_AUTH_SOCK=$SSH_AUTH_SOCK" > $SSH_AUTH_SOCK_FILE
ssh-add $HOME/.ssh/github.com_id_rsa 2>&1 > /dev/null
#else
# echo "already running"
fi
source $SSH_AUTH_SOCK_FILE
Don’t forget to add your correct keys in the "ssh-add" command.
I had a similar problem when I was trying to get this to work on Windows to connect to the stash via SSH.
Here is the solution that worked for me.
Turns out I was running the Pageant ssh agent on my Windows box - I would check what you are running. I suspect it is Pageant as it comes as default with PuTTY and WinSCP.
The ssh-add does not work from command line with this type of agent
You need to add the private key via the Pageant UI window which you can get by double-clicking the Pageant icon in the taskbar (once it is started).
Before you add the key to Pageant you need to convert it to PPK format. Full instructions are available here How to convert SSH key to ppk format
That is it. Once I uploaded my key to stash I was able to use Sourcetree to create a local repository and clone the remote.
For Bash built into Windows 10, I added this to file .bash_profile:
if [ -z $SSH_AUTH_SOCK ]; then
if [ -r ~/.ssh/env ]; then
source ~/.ssh/env
if [ `ps -p $SSH_AGENT_PID | wc -l` = 1 ]; then
rm ~/.ssh/env
unset SSH_AUTH_SOCK
fi
fi
fi
if [ -z $SSH_AUTH_SOCK ]; then
ssh-agent -s | sed 's/^echo/#echo/'> ~/.ssh/env
chmod 600 ~/.ssh/env
source ~/.ssh/env > /dev/null 2>&1
fi
Using Git Bash on Windows 8.1 E, my resolution was as follows:
eval $(ssh-agent) > /dev/null
ssh-add ~/.ssh/id_rsa
I resolved the error by force stopping (killed) git processes (ssh agent), then uninstalling Git, and then installing Git again.
This worked for me.
In the CMD window, type the following command:
cd path-to-Git/bin # (for example,cd C:\Program Files\Git\bin)
bash
exec ssh-agent bash
ssh-add path/to/.ssh/id_rsa
I need to reinstall one of ours servers, and as a precaution, I want to move /home, /etc, /opt, and /Services to backup server.
However, I have a problem: because of plenty of symbolic links a lot of files are copied multiple times.
Is it possible to make scp ignore the symbolic links (or actually to copy link as a link not as a directory or file)? If not, is there another way to do it?
I knew that it was possible, I just took wrong tool. I did it with rsync
rsync --progress -avhe ssh /usr/local/ XXX.XXX.XXX.XXX:/BackUp/usr/local/
I found that the rsync method did not work for me, however I found an alternative that did work on this website (www.docstore.mik.ua/orelly).
Specifically section 7.5.3 of "O'Reilly: SSH: The Secure Shell. The Definitive Guide".
7.5.3. Recursive Copy of Directories
...
Although scp can copy directories, it isn't necessarily the best method. If your directory contains hard links or soft links, they won't be duplicated. Links are copied as plain files (the link targets), and worse, circular directory links cause scp1 to loop indefinitely. (scp2 detects symbolic links and copies their targets instead.) Other types of special files, such as named pipes, also aren't copied correctly.A better solution is to use tar, which handles special files correctly, and send it to the remote machine to be untarred, via SSH:
$ tar cf - /usr/local/bin | ssh server.example.com tar xf -
Using tar over ssh as both sender and receiver does the trick as well:
cd $DEST_DIR
ssh user#remote-host 'cd $REMOTE_SRC_DIR; tar cf - ./' | tar xvf -
One solution is to use a shell pipe. I have a situation where I got some *.gz files and symbolic links generated by some software to link to the same *.gz files with a slightly shorter name. If I simply use scp, then the symbolic links will be copied as regular files and resulting in duplicates. I know rsync can ignore symbolic links, but my gz files are not compressed with syncable options, and sync is very slow in copying these gz files. So I simply use the following script to copy over the files:
find . -type f -exec scp {} target_host:/directory/name/data \;
The -f option will only find regular files and ignore symbolic links. You need to give this command on the source host. Hope this may help some user in my situation. Let me know if I missed anything.
A one liner solution which can be executed at client to copy folder from server using tar + ssh command.
ssh user#<Server IP/link> 'mkdir -p <Remote destination directory;cd <Remote destination directory>; tar cf - ./' | tar xf - C <Source destination directory>
Note: mkdir is must, if the remote destination directory is not present then the command will simply compress the entire home of the remote server and extract it to client.