Ansible SSH user change during playbook execution causes freeze - ssh

Question

I have a playbook that is launched from AWX with machine credentials, ie. with ssh_user_A. In this playbook, i need to perform a couple of tasks with another SSH user, ie. ssh_user_B.
In terms of credentials, I have:
machine credentials for ssh_user_A that are OK, I can do whatever I need
a public/private keys couple for ssh_user_B:
the public key is present on the remote host in the ~ssh_user_B/.ssh/authorized_keys file
if I test this public/private key with a temp machine credentials (not possible in final target) and a dummy playbook or an ad-hoc module call, it works I can do whatever I need
I change the SSH user in the playbook with these instructions:
- name: Switch to ssh_user_B user
ansible.builtin.set_fact:
ansible_user: "ssh_user_B"
ansible_ssh_pass: "{{ssh_user_B_passphrase}}"
ansible_ssh_private_key_file: "{{file_that_contains_the_private_key}}"
delegate_to: localhost
And the user change is triggered by a simple wait_for_connection:
- name: Just here to validate the user change
ansible.builtin.wait_for_connection:
timeout: 10
But unfortunately, the only thing that I get is a behavior that I do not understand:
with valid public/private keys, the task that is supposed to trigger the user change seems frozen at the first SSH connection attempt, nothing more happens, and nothing in the target remote host SSH logs (in these logs, I have the connection for the ssh_user_A, but absolutely nothing for the ssh_user_B one)
AWX output:
PLAYBOOK: test.yml *************************************************************
Positional arguments: test.yml
verbosity: 4
remote_user: ssh_user_A
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/tmp/awx_14019_pqypcw0i/tmpm_98ffss',)
subset: target_remote_host
extra_vars: ('#/tmp/awx_14019_pqypcw0i/tmp7493qmoq', '#/tmp/awx_14019_pqypcw0i/env/extravars')
forks: 5
1 plays in test.yml
Read vars_file './vars/main.yml'
Read vars_file './vars/main.yml'
Read vars_file './vars/main.yml'
PLAY [MultiUserTest] ***********************************************************
Read vars_file './vars/main.yml'
TASK [Gathering Facts] *********************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:2
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_A
... Connection OK
TASK [Another task OK] ***************************************************
...
TASK [Task done on localhost] ********************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:57
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
... Connection OK
TASK [Another locahost task OK] ***************************************************
...
TASK [switch to ssh_user_B user] *****************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:83
ok: [s4850ads -> localhost] => {
"ansible_facts": {
"ansible_ssh_pass": "dummy_passphrase",
"ansible_ssh_private_key_file": ".dummy_pk_random_name",
"ansible_user": "ssh_user_B"
},
"changed": false
}
Read vars_file './vars/main.yml'
TASK [Wait 300 seconds, but only start checking after 60 seconds] **************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:91
wait_for_connection: attempting ping module test
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_B
<target_remote_host> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile=".dummy_pk_random_name"' -o 'User="ssh_user_B"' -o ConnectTimeout=10 -o ControlPath=/tmp/awx_14019_pqypcw0i/cp/bc63f1e109 target_remote_host '/bin/sh -c '"'"'echo ~ssh_user_B && sleep 0'"'"''
... Here it's frozen, nothing more in the AWX output, nothing in the remote host SSH logs...
with an invalid public/private keys (ie. with additional characters, no LF at the end of the private key file, etc.), the task that is supposed to trigger the user works and ends in error. I can see that every SSH connection method is tested, and it ends in error:
TASK [switch to ssh_user_B user] *****************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:83
ok: [s4850ads -> localhost] => {
"ansible_facts": {
"ansible_ssh_pass": "dummy_passphrase",
"ansible_ssh_private_key_file": ".dummy_pk_random_name",
"ansible_user": "ssh_user_B"
},
"changed": false
}
Read vars_file './vars/main.yml'
TASK [Wait 300 seconds, but only start checking after 60 seconds] **************
task path: /tmp/awx_14021_ypwngn_h/project/test.yml:91
wait_for_connection: attempting ping module test
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_B
<target_remote_host> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile=".dummy_pk_random_name"' -o 'User="ssh_user_B"' -o ConnectTimeout=10 -o ControlPath=/tmp/awx_14021_ypwngn_h/cp/bc63f1e109 target_remote_host '/bin/sh -c '"'"'echo ~ssh_user_B && sleep 0'"'"''
<target_remote_host> (5, b'', b'OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021\
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for \\'final all\\' host target_remote_host originally target_remote_host
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched \\'final\\'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for \\'final all\\' host target_remote_host originally target_remote_host
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched \\'final\\'
debug2: match found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: auto-mux: Trying existing master
debug1: Control socket "/tmp/awx_14021_ypwngn_h/cp/bc63f1e109" does not exist
debug2: resolving "target_remote_host" port 22
debug2: ssh_connect_direct
debug1: Connecting to target_remote_host [<IP>] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9984 ms remain after connect
debug1: identity file .dummy_pk_random_name type -1
debug1: identity file .dummy_pk_random_name-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to target_remote_host:22 as \\'ssh_user_B\\'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from target_remote_host
debug3: order_hostkeyalgs: have matching best-preference key type ecdsa-sha2-nistp256-cert-v01#openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ssh-ed25519-cert-v01#openssh.com,rsa-sha2-512-cert-v01#openssh.com,rsa-sha2-256-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#openssh.com,hmac-sha2-512
debug2: compression ctos: zlib#openssh.com,zlib,none
debug2: compression stoc: zlib#openssh.com,zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib#openssh.com
debug2: compression stoc: none,zlib#openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm#openssh.com MAC: <implicit> compression: zlib#openssh.com
debug1: kex: client->server cipher: aes256-gcm#openssh.com MAC: <implicit> compression: zlib#openssh.com
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:YYY/ZZZ
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from target_remote_host
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <IP>
debug1: Host \\'target_remote_host\\' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /tmp/awx_14021_ypwngn_h/artifacts/14021/ssh_key_data RSA SHA256:XXX agent
debug1: Will attempt key: .dummy_pk_random_name explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 53
debug3: input_userauth_banner
###################################################################
# SERVER MESSAGE. #
###################################################################
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /tmp/awx_14021_ypwngn_h/artifacts/14021/ssh_key_data RSA SHA256:XXX agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: .dummy_pk_random_name
Load key ".dummy_pk_random_name": invalid format
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can\\'t open /dev/tty: No such device or address
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
ssh_user_B#target_remote_host: Permission denied (publickey,password).
')
... Other SSH connection method attempt
I also see the connection attempts in the remote host SSH logs, they all look like this:
Feb 28 14:25:10 target_remote_host sshd[4108095]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<IP> user=ssh_user_B
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Connection closed by authenticating user ssh_user_B <IP> port 12345 [preauth]
What I've checked/done/excluded:
as said previously, the public/private key couple is OK, tested though AWX on a separate credentials machine
I've tested to copy the private key file on the remote folder in order to ckeck its content and see if it was well formed, and it is (here $ represent the LF):
-----BEGIN OPENSSH PRIVATE KEY-----$
...
-----END OPENSSH PRIVATE KEY-----$
I've created a new public/private key couple with a passphrase that only contains letters and digits, in order to exlude a problem from a special character in the passphrase (like a `), it does not change anything
If I do not provide a passphrase, I get a no passphrase given error from the remote host as expected
Regarding these elements, I assume that my SSH connection parameters are OK but something stucks the playbook, but I really do not understand what... Any clue/help would be appreciated.
Note that I do not (and won't) have access on the host where AWX is installed and the playbook runs.
Part of the playbook that contain the credentials change:
- name: Main private key block
block:
- name: Creating temp filenames
ansible.builtin.set_fact:
filename_private_key: ".dummy_pk_random_name"
# Just for testing purposes, target is a key in a Vault
temp_copy_file: "./files/temp_pk_file.txt"
- name: Sub part 1
block:
##########
- name: Creating container file for private key
ansible.builtin.file:
path: "{{filename_private_key}}"
state: touch
mode: '600'
changed_when: false
- name : Putting private key into needed file
ansible.builtin.copy:
src: "{{temp_copy_file}}"
dest: "{{filename_private_key}}"
mode: '600'
changed_when: false
##########
delegate_to: localhost
- name: Sub part 2
block:
##########
- name: Switch to ssh_user_B user
ansible.builtin.set_fact:
ansible_user: "ssh_user_B"
# The passphrase comes from a Vault
ansible_ssh_pass: "{{ssh_user_B_passphrase}}"
ansible_ssh_private_key_file: "{{filename_private_key}}"
delegate_to: localhost
- name: Just here to validate the user change
ansible.builtin.wait_for_connection:
timeout: 10
##########
##########
always:
- name: Ensure private key is not present anymore
file:
path: "{{ filename_private_key }}"
state: absent
changed_when: false
delegate_to: localhost
##########

Answer 1

I've made a lot of tests and found the problem: by default, to connect to the target hosts, Ansible uses the smart connection plugin. In my case, the smart plugin leads to the use of the native OpenSSH.
Forcing manually the use of the paramiko connection plugin solves the problem, everything is OK (paramiko is a Python implementation of OpenSSH). Just need to add the instruction connection: paramiko at the needed level:
---
- name: MultiUserTest part 1
hosts: all
gather_facts: yes
connection: paramiko