Ansible fails with "Permission denied (keyboard-interactive)" - ssh

I have two different machines I can connect to in SSH, using a corporate VPN and proxy.
For that, my ~/.ssh/config has two hosts defined like this:
Host foobar
User alice
HostName XXX.XXX.XXX.XXX
ProxyCommand nc -x proxy-socks.foobar.com:4242 %h %p
The ProxyCommand is identical for both hosts, only the HostName and User are different. Each host has my public key, I can connect to each one simply by typing ssh foobar, no password is asked.
Now, I tried to use ansible to act on these machines. I have an inventory.cfg like that listing the name of both machines as defined in my ~/.ssh/config:
[vpn]
foobar
barfoo
I tried this simple command:
ansible -i inventory.cfg vpn -m ping
This worked for one of the machines, but not the other. Here is the redacted output:
foobar | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/libexec/platform-python"
},
"changed": false,
"ping": "pong"
}
barfoo | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: Welcome on barfoo [...] Permission denied (keyboard-interactive).",
"unreachable": true
}
Since I got the "welcome" message from the host in the output, it means that I did reach the host (so the proxy configuration should be ok).
But I cannot understand the error "Permission denied (keyboard-interactive).". I can connect to this machine by SSH without password (in fact, the admins even disabled password authentication, I had to send them my public key by e-mail).
I tried to explicitely specify my SSH key by adding --private-key $HOME/.ssh/id_rsa, but the error message was exactly the same.
By curiosity, I tried with Python's package fabric, but this worked fine for both machines:
import fabric
fabric.Connection('barfoo').run('hostname')
So it seems there is something weird going on between ansible and this machine configuration. Any clue?
EDIT
Using the advice from #GeralexGR, I added -vvvv in my ansible command to have more output.
In the output, I could see that ansible is making this call to SSH :
ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/home/alice/.ansible/cp/dbd7338475"' barfoo '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
By trial and error, I could reduce this command to this one :
ssh -vvv -o PreferredAuthentications=publickey barfoo 'hostname'
This command fails on one machine but not the other. However, if I remove the PreferredAuthentications option, it works fine with both machines.
When it fails, it outputs something like that:
debug1: Next authentication method: publickey
debug1: Offering public key: [...]
[...]
debug1: Server accepts key: [...]
[...]
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug3: start over, passed a different list keyboard-interactive
debug3: preferred publickey
debug1: No more authentication methods to try.
If I remove this option and do a simple ssh -vvvv, I get this in the output:
debug1: Next authentication method: publickey
debug1: Offering public key: [...]
[...]
debug1: Server accepts key: [...]
[...]
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
[...]
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
[...]
debug1: Authentication succeeded (keyboard-interactive).
So, my guess is that keyboard-interactive is required for this machine. And indeed, if I put both publickey and keyboard-interactive in the option, this works:
ssh -vvv -o preferredauthentications=publickey,keyboard-interactive barfoo 'hostname'
So, back to ansible, I tried this:
ansible -vvvv --ssh-extra-args='-o preferredauthentications=publickey,keyboard-interactive' -i inventory.cfg -m ping vpn
But it still failed, I guess that this authentication mecanism is not possible with ansible? I am not sure why/how keyboard-interactive is needed here, since I am never prompted for a password. I will ask the admins of the machine.

The solution was kindly given by #Zeitounator in comments.
I added these two lines in my inventory.cfg file (with a dummy password):
[vpn:vars]
ansible_ssh_pass=foo
Now the initial ansible command works well.
Alternatively, using the --ask-pass option on the command line works as well:
ansible --ask-pass -i inventory.cfg -m ping vpn
For the context I asked a bit more information to the admins of the machine. This machine does not ask me for a password or anything, however, it asks to some external users (not me) to accept some terms and conditions. This is why the keyboard-interactive mechanism is needed.

Duo requires keyboard-interactive authentication, but ansible turned it off by -o KbdInteractiveAuthentication=no as seen from the ansible -vvv output as shown in the original post. I added ssh_args -o KbdInteractiveAuthentication=yes in ansible.cfg:
ssh_args = -o PreferredAuthentications=publickey,keyboard-interactive -o ControlMaster=auto -o ControlPersist=300s -o KbdInteractiveAuthentication=yes
and now it works fine.

Related

Remote connection using ssh to oracle-solaris-114-sru29 not working with -o option

Remote ssh connection not working with Solaris 11.4.29 with "-o PreferredAuthentications=password" argument, while the same works fine without it.
ssh -2 -vvv -l - works
ssh -2 -vvv -l -o PreferredAuthentications=password - doesn't work
From any Linux server when below command is executed to remote Solaris 11.4.29 :
ssh -2 -vvv -l -o PreferredAuthentications=password - doesn't work
Verbose messages -
debug1: PAM: setting PAM_TTY to "ssh"
debug1: PAM: password authentication failed for : Authentication failed

How do I configure Ansible to jump through two bastion hosts?

I want to write an Ansible playbook (using Ansible 2.7.5) that will jump through two hosts before reaching the intended server to do things such as install docker and python, etc.
I'm able to get Ansible to jump through one host into server1 by adding this to my hosts file:
[server1:vars]
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q bastion"'
I have also updated my ~/.ssh/config file:
Host bastion
Hostname YY.YY.YY.YY
User user
IdentityFile ~/.ssh/bastion_private_key
Host server1
Hostname XX.XX.XX.XX
User user
IdentityFile ~/.ssh/private_key
ProxyJump bastion
However, I now also need to do this through two hosts. I've added the following to ~/.ssh/config:
Host server2
Hostname ZZ.ZZ.ZZ.ZZ
User user
IdentityFile ~/.ssh/private_key_3
ProxyJump server1
This allows me to type ssh server2 and open a shell inside server2. So that seems to be working.
But, I do not know how to change the hosts file to jump through both of these hosts. I've tried:
ansible_ssh_common_args='-o ProxyCommand="ssh -J bastion,server1"'
and
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q bastion ssh -W %h:%p -q server1"'
Neither work, and both result in a timeout. What should I do to make Ansible jump through bastion and then server1 so that it can reach server2?
This is the result when I run -vvvv (with some path and names obfuscated):
ansible-playbook 2.7.5
config file = /path/to/dir/ansible.cfg
configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]
Using /path/to/dir/ansible.cfg as config file
setting up inventory plugins
/path/to/dir/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
/path/to/dir/hosts did not meet script requirements, check plugin documentation if this is unexpected
/path/to/dir/hosts inventory source with ini plugin
[WARNING]: Found both group and host with same name: server2
statically imported: /path/to/dir/tasks/ansible.yml
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/default.pyc
PLAYBOOK: enable-ansible.yml *********************************************************************************************************************************
1 plays in enable-ansible.yml
PLAY [server2] ****************************************************************************************************************************************
TASK [Gathering Facts] ***************************************************************************************************************************************
task path: /path/to/dir/enable-ansible.yml:2
<server2> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<server2> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o 'ProxyCommand=ssh -W %h:%p -q bastion ssh -W %h:%p -q server1' -o ControlPath=/home/user/.ansible/cp/460e3f86d3 server2 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1546192323.33-48994637286535 `" && echo ansible-tmp-1546192323.33-48994637286535="` echo /tmp/ansible-tmp-1546192323.33-48994637286535 `" ) && sleep 0'"'"''
<server2> (255, '', 'OpenSSH_7.6p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /home/user/.ssh/config\r\ndebug1: /home/user/.ssh/config line 70: Applying options for server2\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket "/home/user/.ansible/cp/460e3f86d3" does not exist\r\ndebug1: Executing proxy command: exec ssh -W SERVER2_IP_ADDRESS:22 -q bastion ssh -W SERVER2_IP_ADDRESS:22 -q server1\r\ndebug3: timeout: 10000 ms remain after connect\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/user/.ssh/bastion type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/user/.ssh/bastion-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.1\r\ndebug1: permanently_drop_suid: 1000\r\nConnection timed out during banner exchange\r\n')
fatal: [server2]: UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: OpenSSH_7.6p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /home/user/.ssh/config\r\ndebug1: /home/user/.ssh/config line 70: Applying options for server2\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/home/user/.ansible/cp/460e3f86d3\" does not exist\r\ndebug1: Executing proxy command: exec ssh -W SERVER2_IP_ADDRESS:22 -q bastion ssh -W SERVER2_IP_ADDRESS:22 -q server1\r\ndebug3: timeout: 10000 ms remain after connect\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/user/.ssh/bastion type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /home/user/.ssh/bastion-cert type -1\r\ndebug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.1\r\ndebug1: permanently_drop_suid: 1000\r\nConnection timed out during banner exchange\r\n",
"unreachable": true
}
to retry, use: --limit #/home/user/Documents/repos/cloud-devops/enable-ansible.retry
PLAY RECAP ***************************************************************************************************************************************************
server2 : ok=0 changed=0 unreachable=1 failed=0
For some added context, this playbook is logging into the remote server as a non-root account and creating the ansible user in it. And to reiterate, this playbook works when I am only jumping through one host.
Just use
ansible_ssh_common_args='-J bastion,server1'

Configure ssh config file for certain command

My problem is the following. I wish to configure the .ssh/config as such, that when I write
ssh exampleX
It is the same as if I wrote
ssh -i /path/to/key.pem user#address
Note that the above command works.
Following the answers here I tried to create the file as
Host exampleX
HostName address
User user
IdentityFile /path/to/key.pem
Taken from
ssh -i /path/to/key.pem user#address
Yet when I run
ssh exampleX
I get the error
ssh: Could not resolve hostname exampleX: Name or service not known
But if I manually run the command
ssh -i /path/to/key.pem user#address
everything works. Where am I making the mistake in creating the file?
Edit
If I run
sudo ssh exampleX -v
I get the output
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
ssh: Could not resolve hostname exampleX: Name or service not known
but if I run it without sudo i get a longer stream, that ends with
debug1: Authentications that can continue: publickey
debug1: Trying private key: /path/to/key.pem
Load key "/path/to/key.pem": Permission denied
debug1: No more authentication methods to try.
Permission denied (publickey).
Edit 2
Due to some confusion , I restate my question
What does the config file has to look like, so that running
ssh exampleX
will work the same as running
ssh -i /path/to/key.pem user#address
When you run your command through sudo, you are using the .ssh/config file that corresponds to the user that sudo runs as. If you really need to run this ssh command as root, you need the configuration added to ~root/.ssh/config instead of ~/.ssh/config.
If possible, run your ssh as a normal user, not as root.
(Since the question was edited, I edited accordingly my answer)
Check the permissions of the file ~/.ssh/config: it must have strict permissions: read/write for the user, and not accessible by others, as explained in the man page.
Check also you have read access (as a user) to the file /path/to/key.pem. The debug option you used with ssh suggests you don't have.

we did not send a packet, disable method

I have tried to setup public key connection between AIX to AIX and that's working and I am facing problem while I setup same public key connection between AIX and DataPower appliance.
I could see in logs that's connection established but while using public I am not able to login and everytime expecting to enter password manually.
Can someone please help on this?
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: ./***_rsa.pub
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
There are many, many reasons why client-side debugging (ssh -vvv ...) shows:
debug2: we did not send a packet, disable method
Many of these are listed in the answers to SSH public key won't send to server on Unix & Linux, but, unfortunately, the client does not give any indication as to which one applies.
When I was struggling with this, my main problem was getting server-side logging/debugging to show. Of great help was the following (paraphrased) suggestion taken from Ciro Santilli 冠状病毒审查六四事件法轮功's answer to How to check sshd log? on ServerFault:
Start a new SSH Server instance on a new port in debug mode with:
/usr/sbin/sshd -d -p 2222
then connect to it from the client with:
ssh -p 2222 user#host
If you have sudo access to the destination server, check the SSH logs (/var/log/secure), it may give you some clue. For my case it was a bad permission on the home directory.
chmod g-w /home/user
chmod 700 /home/user/.ssh
chmod 600 /home/user/.ssh/authorized_keys
Reference: https://chemicloud.com/kb/article/ssh-authentication-refused-bad-ownership-or-modes-for-directory/
I solved the same problem
(when I checked in my system there is .ssh/authorized_keys : the filename was wrong)
this way:
cp -p .ssh/authorised_keys .ssh/authorized_keys
Check the ssh configuration on remote server to find if remote system is accepting type of key being passed.
Check the file /etc/ssh/sshd_config and look for property PubkeyAcceptedKeyTypes to confirm that it accepts key type being passed. For instance, here the key type seems to be ssh-rsa, so that should be added to the list of PubkeyAcceptedKeyTypes
(or for maximium compatibility PubkeyAcceptedKeyTypes *)
Once added, restart sshd daemon
EX:
sudo sed -i "/.*PubkeyAcceptedKeyTypes.*/d" /etc/ssh/sshd_config
sudo sed -i "/.*RSAAuthentication.*/d" /etc/ssh/sshd_config
sudo sed -i "/.*PubkeyAuthentication.*/d" /etc/ssh/sshd_config
sudo sed -i "/.*PasswordAuthentication.*/d" /etc/ssh/sshd_config
sudo sed -i "/.*PermitRootLogin.*/d" /etc/ssh/sshd_config
echo "PubkeyAcceptedKeyTypes *" | sudo tee -a /etc/ssh/sshd_config
echo "RSAAuthentication yes" | sudo tee -a /etc/ssh/sshd_config
echo "PubkeyAuthentication yes" | sudo tee -a /etc/ssh/sshd_config
echo "PasswordAuthentication no" | sudo tee -a /etc/ssh/sshd_config
echo "PermitRootLogin no" | sudo tee -a /etc/ssh/sshd_config
sudo systemctl restart sshd.service
sudo systemctl status sshd.service | grep -i success
Refer to this answer SSH public key won't send to server.
In your log
Offering RSA public key: ./***_rsa.pub
...
we did not send a packet, disable method
It means that the client didn't send the key file. So make sure the key file exists.
Check the key file is in your local path
Make sure the public key content is copied into ~/.ssh/authorized_keys in remote server.
I've just had the same problem. Two raspberry Pis on my local network - one connected, one didn't (with the 'debug2: we did not send a packet, disable method' error)
after 4 hours I found the problem
Being used to uk spelling I created the 'authorised_users' file not 'authorized_users'
Doh !!
So what happened for me is that I have 2 VMs to access from my local machine (2 keys id_rsa.pub and id_rsa2.pub). I realized that my ssh connection is using id_rsa.pub by default for any ssh user#xx.xx.xx.xx connection. I solved my issue by adding a config file and specify the identity to be used for every host like the following :
vi ~/.ssh/config
Add both hostnames and their identity file as follows:
Host server1.nixcraft.com
IdentityFile ~/Users/.ssh/id_rsa1
Host server2.nixcraft.com
IdentityFile ~/Users/.ssh/id_rsa2
This happened to me and, after several hours banging my head against the wall, I realized there was a typo in the public key file. Instead of beginning with
ssh-rsa
there was a + sign at the beginning:
+ssh-rsa
I know it's obvious in retrospect, but it doesn't hurt to double check your file.
It looks like there can be a lot of things causing this problem. In my case I forgot to specified the User name in my ~/.ssh/config file. Apparently, it depends on your current login name.
Host dev
HostName 1.2.3.4
IdentityFile ~/.ssh/id_rsa
User John
This can happen if in your /etc/ssh/sshd_config you have the following entry
PubkeyAcceptedKeyTypes=+ssh-dss
as opposed to
PubkeyAcceptedKeyTypes +ssh-dss #this is the correct entry
After updating this, be sure to issue a
sudo systemctl restart sshd #Ubuntu
Think about selinux, if is enabled,do:
# sudo ausearch -c "sshd" --raw | audit2allow -M my-sshd
# sudo semodule -i my-sshd.pp
I checked and tried every instruction given here and after several hours of troubleshooting, I noticed the SSH server (which is my router) does not have a support for ed25519. Dropbear was spamming the following log: Pubkey auth attempt with unknown algo...
Using an RSA type fixed my issue.
Just for those who have the log "we did not send a packet, disable method" when trying to use a public key.
Check your sshd_config file on the remote server and force "PubkeyAuthentication" to "yes", even if it should be "yes" by default.
For me it solved my connection problem.
In my case the private key must have been added to the agent with:
ssh-add ~/.ssh/private_id_rsa

Can't get SSH ProxyCommand to work (ssh_exchange_identification: Connection closed by remote host)

I'm unsuccessfully trying to use SSH ProxyCommand to connect to a server via a jump box. My config is below, I'm running this command:
ssh 10.0.2.54 -F ssh.config -vv
Host x.x.x.x
User ec2-user
HostName x.x.x.x
ProxyCommand none
IdentityFile /Users/me/.ssh/keys.pem
BatchMode yes
PasswordAuthentication no
Host *
ServerAliveInterval 60
TCPKeepAlive yes
ProxyCommand ssh -W %h:%p -q ec2-user#x.x.x.x
ControlMaster auto
ControlPersist 8h
User ec2-user
IdentityFile /Users/me/.ssh/keys.pem
The result is:
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data ssh.config
debug1: ssh.config line 9: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/me/.ssh/mux-ec2-user#10.0.2.54:22" does not exist
debug2: ssh_connect: needpriv 0
debug1: Executing proxy command: exec ssh -W 10.0.2.54:22 -q ec2-user#x.x.x.x
debug1: identity file /Users/me/.ssh/keys.pem type -1
debug1: identity file /Users/me/.ssh/keys.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: permanently_drop_suid: 501
How can I get this to work/troubleshoot the issue?
Thanks,
ControlPersist in combination with ProxyCommand is not effective and you miss ControlPath option. But it is not a problem here.
First of all, if you are using non-standard config file and you want it to be used even by the proxy command, you need to specify it even there. The -q option makes the connection quiet so you have no idea what is going on under the hood of the proxy command. LogLevel DEBUG3 option is quite useful.
This line:
ProxyCommand ssh -W %h:%p -q ec2-user#x.x.x.x
needs to be (and you don't need the username as it is already specified above):
ProxyCommand ssh -W %h:%p -F ssh.config x.x.x.x
You have also wrong order of parameters in your command:
ssh 10.0.2.54 -F ssh.config -vv
needs to be:
ssh -F ssh.config 10.0.2.54
as you can read from manual page. And -vv is not needed if you use LogLevel option.
Then it should work for you (at least it did for me, otherwise investigate the log).