Capistrano fails SSH public key authentication but all commands still succeed - ssh

Question

I am having trouble deploying using Capistrano using public key authentication. On windows, I have it configured to start an SSH agent automatically when I open my terminal.
Agent pid 4476
Enter passphrase for /c/Users/Lea/.ssh/id_rsa:
Identity added: /c/Users/Lea/.ssh/id_rsa (/c/Users/Lea/.ssh/id_rsa)
id_rsa is in my authorized_keys file on the server, and I use it all the time to ssh into it using ssh lea#web.3.
My Capfile is as follows:
require 'rubygems'
require 'railsless-deploy'
# application name
set :application, "site.com"
# multi-stage deploy
task :production do
set :branch, "master"
set :app_environment, "production"
role :web, "web.3", :primary => true
set :deploy_to, "/var/www/vhosts/site/site.com/"
end
task :dev do
set :branch, `git rev-parse HEAD`
set :app_environment, "development"
role :web, "web.3", :primary => true
set :deploy_to, "/var/www/vhosts/site/dev.site.com/"
end
# deploys remotely on SSH using deploy only key
set :repository, "git#bitbucket.org:us/site.git"
set :scm, :git
set :git_enable_submodules, 1
set :deploy_via, :remote_cache
# release configuration
set :use_sudo, false
set :keep_releases, 2
after "deploy:update", "deploy:cleanup"
# the web server user
set :user, "lea"
namespace :deploy do
task :migrate do
# do nothing
end
task :finalize_update, :except => { :no_release => true } do
transaction do
#run "chmod -R g+w #{release_path}"
run "echo '#{app_environment}' > #{release_path}/ENVIRONMENT"
end
end
task :restart, :except => { :no_release => true } do
# don't need to restart
end
end
When I run the deployment, it asks again for my id_rsa passphrase. Why does it ask when I already have the ssh agent running and the passphrase entered?
Following is the log of the cap dev deploy command. You can see where it asks my passphrase. Also note when I ssh into the server, it starts an ssh-agent there as well and loads a deployment_rsa key used for git (you can see these messages in the log).
$ cap dev deploy
DL is deprecated, please use Fiddle
* 2013-09-12 13:19:30 executing `dev'
* 2013-09-12 13:19:30 executing `deploy'
* 2013-09-12 13:19:30 executing `deploy:update'
** transaction: start
* 2013-09-12 13:19:30 executing `deploy:update_code'
updating the cached checkout on all servers
* executing "if [ -d /var/www/vhosts/site/dev.site.com/shared/cache
d-copy ]; then cd /var/www/vhosts/site/dev.site.com/shared/cached-cop
y && git fetch -q origin && git fetch --tags -q origin && git reset -q --hard 33
09af4ac302a6c2dc46bcf36e877abbd8472988\\\n && git submodule -q init && git submo
dule -q sync && export GIT_RECURSIVE=$([ ! \"`git --version`\" \\< \"git version
1.6.5\" ] && echo --recursive) && git submodule -q update --init $GIT_RECURSIVE
&& git clean -q -d -x -f; else git clone -q git#bitbucket.org:us/v
entek.git /var/www/vhosts/site/dev.site.com/shared/cached-copy && cd
/var/www/vhosts/site/dev.site.com/shared/cached-copy && git checkout
-q -b deploy 3309af4ac302a6c2dc46bcf36e877abbd8472988 && git submodule -q init &
& git submodule -q sync && export GIT_RECURSIVE=$([ ! \"`git --version`\" \\< \"
git version 1.6.5\" ] && echo --recursive) && git submodule -q update --init $GI
T_RECURSIVE; fi"
servers: ["web.3"]
Enter passphrase for c:/Users/Lea/.ssh/id_rsa:
[web.3] executing command
** [web.3 :: out] Agent pid 11336
** [web.3 :: err] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 2300ms
copying the cached version to /var/www/vhosts/site/dev.site.com/r
eleases/20130912191939
* executing "cp -RPp /var/www/vhosts/site/dev.site.com/shared/cache
d-copy /var/www/vhosts/site/dev.site.com/releases/20130912191939 && (
echo 3309af4ac302a6c2dc46bcf36e877abbd8472988\\\n > /var/www/vhosts/us/dev.site.com/releases/20130912191939/REVISION)"
servers: ["web.3"]
[web.3] executing command
** [out :: web.3] Agent pid 11442
*** [err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 751ms
* 2013-09-12 13:19:39 executing `deploy:finalize_update'
* executing "echo 'development' > /var/www/vhosts/site/dev.site.com
/releases/20130912191939/ENVIRONMENT"
servers: ["web.3"]
[web.3] executing command
** [out :: web.3] Agent pid 11451
*** [err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 610ms
* 2013-09-12 13:19:40 executing `deploy:create_symlink'
* executing "rm -f /var/www/vhosts/site/dev.site.com/current && ln
-s /var/www/vhosts/site/dev.site.com/releases/20130912191939 /var/www
/vhosts/site/dev.site.com/current"
servers: ["web.3"]
[web.3] executing command
** [out :: web.3] Agent pid 11460
*** [err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 621ms
** transaction: commit
triggering after callbacks for `deploy:update'
* 2013-09-12 13:19:41 executing `deploy:cleanup'
* executing "ls -xt /var/www/vhosts/site/dev.site.com/releases"
servers: ["web.3"]
[web.3] executing command
[err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh/dep
loyment_rsa)
command finished in 1186ms
** keeping 2 of 7 deployed releases
* executing "rm -rf /var/www/vhosts/site/dev.site.com/releases/2013
0906181120 /var/www/vhosts/site/dev.site.com/releases/20130912185329
/var/www/vhosts/site/dev.site.com/releases/20130912185937 /var/www/vhosts/site/dev.site.com/releases/20130912191939 /var/www/vhosts/us/dev.site.com/releases/11469"
servers: ["web.3"]
[web.3] executing command
** [out :: web.3] Agent pid 11476
*** [err :: web.3] Identity added: /home/lea/.ssh/deployment_rsa (/home/lea/.ssh
/deployment_rsa)
command finished in 750ms
$
Now, my major problem is not with the passphrase. Every time I run capistrano it fails to authenticate 2 times for every deployment. I see this in the ssh log on the server, but no indication is given in Capistrano:
11:58:44 web3 sshd[1134]: Failed password for lea from [ip] port 42421 ssh2
11:58:56 web3 sshd[1134]: Failed password for lea from [ip] port 42421 ssh2
The server is running fail2ban which blocks my IP (for 10 minutes) after 5 failed authentications, meaning I get blocked out after running capistrano 3 times. This is a huge an unacceptable problem, and I have no idea why this would occur. Do you have any advice for how to troubleshoot this problem, or a solution?
Thanks!

Answer 1

I ended up solving this problem myself. I was being locked out of the server because Fail2ban was an old version.
When connecting to SSH, sshd does a reverse DNS lookup. My reverse DNS on the office internet was failing, and it was printing an error into the /var/logs/secure log file.
Address x.x.x.x maps to server.domain.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Fail2ban was recognizing this as a failed connection, and blocking my IP because of it. It was never a problem when connecting manually because that is infrequent, but when Capistrano makes several connections in a row it was triggering it.
I used the info here: https://github.com/fail2ban/fail2ban/pull/64 to solve the problem by removing the regular expression from the fail2ban config file.

Answer 2

Step 1:
Do you really need a passphrase for your keys? This kind of risk today is mitigated by full disk encryption products or use of truecrypt-ed USB sticks. Less PITA, and still passes your security manager's best practices.
That said:
http://blog.blenderbox.com/2013/02/20/ssh-agent-forwarding-with-github/
Try adding
ssh_options[:forward_agent] = true
to the capfile, not Deploy.rb