Timeout waiting for privilege escalation prompt - ssh

Question

Context
I'm asking this question because those two questions/answerse (one and two) have little context, and I'd liket o expand on it.
I'm trying to provision CentOS7 production servers with ansible client 2.8 . My environment consist one master node (NIS Server) and one compute node (NIS Client). Before provisioning live servers, I setted up a virtual lab (vlab) from VirtualBoxes, which mimics production environment. I copied my public keys to both production and vlab environment.
Problem
My playbooks work with vlab. Unfortunately, the same commands/playbooks are failing against production nodes to which I have ssh password less access and sudo rights. The only feedback that I get is this error "Timeout (7s) waiting for privilege escalation prompt: \u001b[?1h\u001b=\r\r"
Question
Does anyone know what causes this behaviour and how to fix it? I tried SeLinux permissions on home directory, I ended up disabling it (sudo setenforce 0). I'm arriving to a conclusion that there has to be something external that stops ansible but?
Debugging, logging
Ansible Config
# config file for ansible -- https://raw.githubusercontent.com/ansible/ansible/devel/examples/ansible.cfg
[defaults]
timeout = 5
inventory = ./config/hosts
remote_user = lukas
SSHD Config
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
UseDNS no
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem sftp /usr/libexec/openssh/sftp-server
Ansible log
# checking kernel version
$ uname -a
Linux compute01 3.10.0-514.26.2.el7.x86_64
# pinging compute node
$ ansible -m ping compute01
123.123.123.123 | SUCCESS => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": false,
"ping": "pong"
}
# installing vim
$ ansible compute01 -m yum -a 'name=vim state=installed' -b -K -u lukas
ansible 2.8.4
config file = /home/lukas/Coding/projects/nebula-provision/ansible/producion/ansible.cfg
configured module search path = [u'/home/lukas/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]
Using /home/lukas/Coding/projects/nebula-provision/ansible/producion/ansible.cfg as config file
BECOME password:
setting up inventory plugins
host_list declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
script declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
auto declined parsing /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts as it did not pass it's verify_file() method
Not replacing invalid character(s) "set([u'-'])" in group name (kubernetes-master)
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
....
Parsed /home/lukas/Coding/projects/nebula-provision/ansible/producion/config/hosts inventory source with ini plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python2.7/dist-packages/ansible/plugins/callback/minimal.pyc
META: ran handlers
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861 `" && echo ansible-tmp-1566570853.88-235437120093861="` echo /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861 `" ) && sleep 0'"'"''
<123.123.123.123> (0, 'ansible-tmp-1566570853.88-235437120093861=/home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<123.123.123.123> Attempting python interpreter discovery
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<123.123.123.123> (0, 'PLATFORM\nLinux\nFOUND\n/usr/bin/python\n/usr/bin/python2.7\n/usr/bin/python\nENDFOUND\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<123.123.123.123> (0, '{"osrelease_content": "NAME=\\"CentOS Linux\\"\\nVERSION=\\"7 (Core)\\"\\nID=\\"centos\\"\\nID_LIKE=\\"rhel fedora\\"\\nVERSION_ID=\\"7\\"\\nPRETTY_NAME=\\"CentOS Linux 7 (Core)\\"\\nANSI_COLOR=\\"0;31\\"\\nCPE_NAME=\\"cpe:/o:centos:centos:7\\"\\nHOME_URL=\\"https://www.centos.org/\\"\\nBUG_REPORT_URL=\\"https://bugs.centos.org/\\"\\n\\nCENTOS_MANTISBT_PROJECT=\\"CentOS-7\\"\\nCENTOS_MANTISBT_PROJECT_VERSION=\\"7\\"\\nREDHAT_SUPPORT_PRODUCT=\\"centos\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\"7\\"\\n\\n", "platform_dist_result": ["centos", "7.3.1611", "Core"]}\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/system/setup.py
<123.123.123.123> PUT /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn TO /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py
<123.123.123.123> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 '[123.123.123.123]'
<123.123.123.123> (0, 'sftp> put /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py\n', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename#openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs#openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs#openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink#openssh.com" revision 1\r\ndebug2: Server supports extension "fsync#openssh.com" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/lukas size 0\r\ndebug3: Looking up /home/lukas/.ansible/tmp/ansible-local-12685VGhDEA/tmpwDBaIn\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:8 O:131072 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:9 O:163840 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:10 O:196608 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:11 O:229376 S:23124\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 32768 bytes at 98304\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 8 32768 bytes at 131072\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 9 32768 bytes at 163840\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 10 32768 bytes at 196608\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 11 23124 bytes at 229376\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 123.123.123.123 '/bin/sh -c '"'"'chmod u+x /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/ /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py && sleep 0'"'"''
<123.123.123.123> (0, '', 'OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 12697\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<123.123.123.123> ESTABLISH SSH CONNECTION FOR USER: lukas
<123.123.123.123> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="lukas"' -o ConnectTimeout=5 -o ControlPath=/home/lukas/.ansible/cp/779c431db0 -tt 123.123.123.123 '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=kuswqyltevcovqytnefnxinbrwvcydkq] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-kuswqyltevcovqytnefnxinbrwvcydkq ; /usr/bin/python /home/lukas/.ansible/tmp/ansible-tmp-1566570853.88-235437120093861/AnsiballZ_setup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
123.123.123.123 | FAILED! => {
"msg": "Timeout (7s) waiting for privilege escalation prompt: \u001b[?1h\u001b=\r\r"
sshd log
Aug 26 13:36:19 123.123.123.123 sudo: pam_unix(sudo:auth): conversation failed
Aug 26 13:36:19 123.123.123.123 sudo: pam_unix(sudo:auth): auth could not identify password for [lukas]

Answer 1

I believe the problem you are having is that you haven't set the permission escalation user password. In Ansible, when we need root permissions, we set the become variable. The method for becoming root may vary, so you can set the way using the ansible_become_method.
Regarding your problem, I think you need to set up the ansible_become_user and ansible_become_password before running your playbook. You can do it in your inventory, or wherever it feels right for your case.
Here is the link to the variable list you can configure to modify how Ansible will connect to the hosts.
You can get more information about privilege escalation on the "Understanding Privilege Escalation" of Ansible Docs.
I hope it helps.

Answer 2

Turned out that the problem relates to PAM auth module. In short, I added auth sufficient pam_permit.so line to /etc/pam.d/sudo file. In long this is how I arrived to the solution.
Be aware that there can be a million of reasons for PAM to fail, thus this solution might not work for you.
$ cat /etc/pam.d/sudo
#%PAM-1.0
# Fixing "auth could not identify password for [username]" ssh problem.
auth sufficient pam_permit.so
# Original config below
auth include system-auth
account include system-auth
password include system-auth
session optional pam_keyinit.so revoke
session required pam_limits.so
session include system-auth

Answer 3

Just now I had the same issue and my command was
./ansible-playbook playbook.yml -i hosts -b --become-user root --extra-vars "ansible_become_pass= myPass" --become-method su
after removing the space in
"ansible_become_pass=[space]myPass" --> "ansible_become_pass=myPass"
I tried executing again and got
The authenticity of host 'localhost (::1)' can't be established.
ECDSA key fingerprint is SHA256:[random string].
Are you sure you want to continue connecting (yes/no/[fingerprint])?
And I was like wt fun?
and then I saw that when executing this playbook I already was a root user
so I've changed to a different user and tried executing this command again and thank God it worked.