Very strange SSH issue connecting via Ansible to Vagrant VM - ssh

Question

I have ansible 2.0.0.2 installed on my Ubuntu 16.04.01 host system.
My hosts file is the following
[testservers]
tsrv1 ansible_ssh_private_key_file=/home/pkaramol/Workspace/vagrans/vagrant/precise32/.vagrant/machines/default/virtualbox/private_key ansible_user=vagrant
[dbservers]
dmsdb_1 ansible_ssh_private_key_file=/home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key ansible_user=vagrant
While ansible tsrv1 -m ping is successful, ansible dmsdb_1 -m ping -vvvv yields the following weird output
(note that I run both after deleting ~/.ssh/known_hosts file)
OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g 1 Mar 2016\r
debug1: Reading configuration data /home/pkaramol/.ssh/config\r
debug1: /home/pkaramol/.ssh/config line 1: Applying options for *\r
debug1: Reading configuration data /etc/ssh/ssh_config\r
debug1: /etc/ssh/ssh_config line 19: Applying options for *\r
debug1: auto-mux: Trying existing master\r
debug1: Control socket \"/home/pkaramol/.ansible/cp/ansible-ssh-dmsdb_1-22-vagrant\" does not exist\r
debug2: resolving \"dmsdb_1\" port 22\r
debug2: ssh_connect_direct: needpriv 0\r
debug1: Connecting to dmsdb_1 [192.168.33.11] port 22.\r
debug2: fd 3 setting O_NONBLOCK\r
debug1: fd 3 clearing O_NONBLOCK\r
debug1: Connection established.\r
debug3: timeout: 10000 ms remain after connect\r
debug1: key_load_public: No such file or directory\r
debug1: identity file /home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key type -1\r
debug1: key_load_public: No such file or directory\r
debug1: identity file /home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key-cert type -1\r
debug1: Enabling compatibility mode for protocol 2.0\r
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1\r
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1\r
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000\r
debug2: fd 3 setting O_NONBLOCK\r
debug1: Authenticating to dmsdb_1:22 as 'vagrant'\r
debug3: send packet: type 20\r
debug1: SSH2_MSG_KEXINIT sent\r
debug3: receive packet: type 20\r
debug1: SSH2_MSG_KEXINIT received\r
debug2: local client KEXINIT proposal\r
debug2: KEX algorithms: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c\r
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ssh-ed25519-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa\r
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc\r
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc\r
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1\r
debug2: compression ctos: zlib#openssh.com,zlib,none\r
debug2: compression stoc: zlib#openssh.com,zlib,none\r
debug2: languages ctos: \r
debug2: languages stoc: \r
debug2: first_kex_follows 0 \r
debug2: reserved 0 \r
debug2: peer server KEXINIT proposal\r
debug2: KEX algorithms: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r
debug2: host key algorithms: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519\r
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm#openssh.com,aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc#lysator.liu.se\r
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm#openssh.com,aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc#lysator.liu.se\r
debug2: MACs ctos: hmac-md5-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-ripemd160-etm#openssh.com,hmac-sha1-96-etm#openssh.com,hmac-md5-96-etm#openssh.com,hmac-md5,hmac-sha1,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160#openssh.com,hmac-sha1-96,hmac-md5-96\r
debug2: MACs stoc: hmac-md5-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-ripemd160-etm#openssh.com,hmac-sha1-96-etm#openssh.com,hmac-md5-96-etm#openssh.com,hmac-md5,hmac-sha1,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160#openssh.com,hmac-sha1-96,hmac-md5-96\r
debug2: compression ctos: none,zlib#openssh.com\r
debug2: compression stoc: none,zlib#openssh.com\r
debug2: languages ctos: \r
debug2: languages stoc: \r
debug2: first_kex_follows 0 \r
debug2: reserved 0 \r
debug1: kex: algorithm: curve25519-sha256#libssh.org\r
debug1: kex: host key algorithm: ecdsa-sha2-nistp256\r
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: zlib#openssh.com\r
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: zlib#openssh.com\r
debug3: send packet: type 30\r
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY\r
debug3: receive packet: type 31\r
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:RS3yHSobFXsfY/LLfBdMIoWNibW8qvoB/Ff9MBkL9/c\r
Warning: Permanently added 'dmsdb_1,192.168.33.11' (ECDSA) to the list of known hosts.\r
debug3: send packet: type 21\r
debug2: set_newkeys: mode 1\r
debug1: rekey after 134217728 blocks\r
debug1: SSH2_MSG_NEWKEYS sent\r
debug1: expecting SSH2_MSG_NEWKEYS\r
debug3: receive packet: type 21\r
debug2: set_newkeys: mode 0\r
debug1: rekey after 134217728 blocks\r
debug1: SSH2_MSG_NEWKEYS received\r
debug2: key: pkaramol#ISA9813 (0x55751be641f0), agent\r
debug2: key: /home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key ((nil)), explicit\r
debug3: send packet: type 5\r
debug3: receive packet: type 6\r
debug2: service_accept: ssh-userauth\r
debug1: SSH2_MSG_SERVICE_ACCEPT received\r
debug3: send packet: type 50\r
debug3: receive packet: type 51\r
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password\r
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r
debug3: authmethod_lookup gssapi-with-mic\r
debug3: remaining preferred: gssapi-keyex,hostbased,publickey\r
debug3: authmethod_is_enabled gssapi-with-mic\r
debug1: Next authentication method: gssapi-with-mic\r
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available
\r
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available
debug1: Unspecified GSS failure. Minor code may provide more information
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available
debug2: we did not send a packet, disable method\r
debug3: authmethod_lookup gssapi-keyex\r
debug3: remaining preferred: hostbased,publickey\r
debug3: authmethod_is_enabled gssapi-keyex\r
debug1: Next authentication method: gssapi-keyex\r
debug1: No valid Key exchange context\r
debug2: we did not send a packet, disable method\r
debug3: authmethod_lookup publickey\r
debug3: remaining preferred: ,publickey\r
debug3: authmethod_is_enabled publickey\r
debug1: Next authentication method: publickey\r
debug1: Offering RSA public key: pkaramol#ISA9813\r
debug3: send_pubkey_test\r
debug3: send packet: type 50\r
debug2: we sent a publickey packet, wait for reply\r
debug3: receive packet: type 51\r
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r
debug1: Trying private key: /home/pkaramol/Workspace/vagrans/vagrant/rhel72-db2/.vagrant/machines/default/virtualbox/private_key\r
debug3: sign_and_send_pubkey: RSA SHA256:b2ubdyVwK253Fsewqq5WjEjzEeeKADonx4L9xYPRLfI\r
debug3: send packet: type 50\r
debug2: we sent a publickey packet, wait for reply\r
debug3: receive packet: type 51\r
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r
debug2: we did not send a packet, disable method\r
debug1: No more authentication methods to try.\r
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r
",
"unreachable": true
}
tsrv1 is a precise32 vm
dmsdb_1 is a rhel72 vm (to which however I used to be able to connect before some random change occurred that messed things up)
I have also tried vagrand destroy --force && vagrant up for the problematic vm.
update: here is my ~/.ssh/config
Host *
StrictHostKeyChecking no
the /etc/ssh/ssh_config file
# This is the ssh client system-wide configuration file. See
# ssh_config(5) for more information. This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.
# Configuration data is parsed as follows:
# 1. command line options
# 2. user-specific file
# 3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.
# Site-wide defaults for some commonly used options. For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.
Host *
# ForwardAgent no
# ForwardX11 no
# ForwardX11Trusted yes
# RhostsRSAAuthentication no
# RSAAuthentication yes
# PasswordAuthentication yes
# HostbasedAuthentication no
# GSSAPIAuthentication no
# GSSAPIDelegateCredentials no
# GSSAPIKeyExchange no
# GSSAPITrustDNS no
# BatchMode no
# CheckHostIP yes
# AddressFamily any
# ConnectTimeout 0
# StrictHostKeyChecking ask
# IdentityFile ~/.ssh/identity
# IdentityFile ~/.ssh/id_rsa
# IdentityFile ~/.ssh/id_dsa
# IdentityFile ~/.ssh/id_ecdsa
# IdentityFile ~/.ssh/id_ed25519
# Port 22
# Protocol 2
# Cipher 3des
# Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
# MACs hmac-md5,hmac-sha1,umac-64#openssh.com,hmac-ripemd160
# EscapeChar ~
# Tunnel no
# TunnelDevice any:any
# PermitLocalCommand no
# VisualHostKey no
# ProxyCommand ssh -q -W %h:%p gateway.example.com
# RekeyLimit 1G 1h
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
GSSAPIDelegateCredentials no
I just use a local configuration file for ansible which just points to the inventory
[defaults]
inventory = ./myfolder
The weird thing is that I CAN connect by typing vagrant ssh to that particular machine!
Any help is highly appreciated I don't know even how to start with this ...

Answer 1

Although this is more of a practical workaround than an actual answer to the above problem, here it goes:
The issue seems to be solved by using a different private network ip address for the specific vm, i.e. by changing the following line in Vagrantfile from
config.vm.network "private_network", ip: "192.168.33.11"
to
config.vm.network "private_network", ip: "192.168.33.13"
as also the subsequent mapping in /etc/hosts file from
192.168.33.11 dmsdb_1
to
192.168.33.13 dmsdb_1