ansible permission denied, but ssh with key works - ssh

Question

UPDATE: Thanks for the feedback guys. As expected, it turned out to be a bonehead mistake on my part. I was not specifying the correct username when I was running the ansible command. If you run ansible without any user (-u), then it will use your current system username. In my case that was swill when I wanted to run it as root.
This is now working:
ansible all -m ping -u root
I am pretty baffled at this point. I can ssh into the box using the public key without any problems, but when I try to use ansible to connect I get no love.
Note: I am using <ip_address> as the IP as to not expose my actual IP address...
My local /etc/ansible/hosts has the following
<ip_address>
I can ssh into the remote box using my default ssh key:
$ ssh root#<ip_address>
I setup ssh on my remote box using:
$ cat ~/.ssh/id_rsa.pub | ssh <user>#<ip_address> "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
The contents of my remote /etc/ssh/sshd_config file is:
# Package generated configuration file
# See the sshd_config(5) manpage for details
# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes
# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 1024
# Logging
SyslogFacility AUTH
LogLevel INFO
# Authentication:
LoginGraceTime 120
PermitRootLogin yes
StrictModes yes
RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile %h/.ssh/authorized_keys
# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes
# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no
# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no
#MaxStartups 10:30:60
#Banner /etc/issue.net
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM no
Here is what I am getting when I try to connect with ansible...
swill:ansible swill$ ansible all -m ping -vvvv
<<ip_address>> ESTABLISH CONNECTION FOR USER: swill
<<ip_address>> REMOTE_MODULE ping
<<ip_address>> EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/swill/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 <ip_address> /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1435804031.59-82833056078245 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1435804031.59-82833056078245 && echo $HOME/.ansible/tmp/ansible-tmp-1435804031.59-82833056078245'
<ip_address> | FAILED => SSH Error: Permission denied (publickey).
while connecting to <ip_address>:22
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.
swill:ansible swill$ ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/swill/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 <ip_address> /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1435804031.59-82833056078245 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1435804031.59-82833056078245 && echo $HOME/.ansible/tmp/ansible-tmp-1435804031.59-82833056078245'
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/swill/.ssh/config
debug1: /Users/swill/.ssh/config line 4: Applying options for <ip_address>
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 53: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/swill/.ansible/cp/ansible-ssh-<ip_address>-22-swill" does not exist
debug2: ssh_connect: needpriv 0
debug1: Connecting to <ip_address> [<ip_address>] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9958 ms remain after connect
debug3: Incorrect RSA1 identifier
debug3: Could not load "/Users/swill/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /Users/swill/.ssh/id_rsa type 1
debug1: identity file /Users/swill/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH*
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "<ip_address>" from file "/Users/swill/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /Users/swill/.ssh/known_hosts:134
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01#openssh.com,ssh-rsa-cert-v00#openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01#openssh.com,ssh-rsa-cert-v00#openssh.com,ssh-rsa,ssh-dss-cert-v01#openssh.com,ssh-dss-cert-v00#openssh.com,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc#lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc#lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-ripemd160-etm#openssh.com,hmac-sha1-96-etm#openssh.com,hmac-md5-96-etm#openssh.com,hmac-md5,hmac-sha1,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160#openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-ripemd160-etm#openssh.com,hmac-sha1-96-etm#openssh.com,hmac-md5-96-etm#openssh.com,hmac-md5,hmac-sha1,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160#openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: zlib#openssh.com,zlib,none
debug2: kex_parse_kexinit: zlib#openssh.com,zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm#openssh.com,aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc#lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm#openssh.com,aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc#lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-ripemd160-etm#openssh.com,hmac-sha1-96-etm#openssh.com,hmac-md5-96-etm#openssh.com,hmac-md5,hmac-sha1,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160#openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-ripemd160-etm#openssh.com,hmac-sha1-96-etm#openssh.com,hmac-md5-96-etm#openssh.com,hmac-md5,hmac-sha1,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160#openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib#openssh.com
debug2: kex_parse_kexinit: none,zlib#openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5-etm#openssh.com
debug1: kex: server->client aes128-ctr hmac-md5-etm#openssh.com zlib#openssh.com
debug2: mac_setup: found hmac-md5-etm#openssh.com
debug1: kex: client->server aes128-ctr hmac-md5-etm#openssh.com zlib#openssh.com
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 129/256
debug2: bits set: 528/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 5c:c8:5f:d3:24:fe:ca:e2:05:03:12:d6:cb:42:d2:91
debug3: load_hostkeys: loading entries for host "<ip_address>" from file "/Users/swill/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /Users/swill/.ssh/known_hosts:134
debug3: load_hostkeys: loaded 1 keys
debug1: Host '<ip_address>' is known and matches the RSA host key.
debug1: Found key in /Users/swill/.ssh/known_hosts:134
debug2: bits set: 505/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /Users/swill/.ssh/id_rsa (0x7f8482500680), explicit
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/swill/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
I would appreciate it if someone could help me see why I am an idiot and can't figure this out. :) Thanks...

Answer 1

I was facing a related issue:
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
The problem was the permissions with the server (ssh). I corrected it with giving the correct permissions to the .ssh and authorized_keys file, as shown below :
chmod 700 .ssh
chmod 600 .ssh/authorized_keys