Aync shell script on Ansible to handle connection reset - ssh

Question

Despite looking at many posts on SO and Ansible's doc, I'm still failing at understanding what Ansible is doing.
My scenario is following: I need to rename the network interface Ansible is connected over to control the remote and restore connection.
My first attempts revolved around something like this:
- name: Hot Rename Main Iface
become: true
shell:
cmd: |
ip link set oldiface down
ip link set oldiface name newiface
ip link set newiface up
async: 0
poll: 0
register: asynchotrename
- name: Wait For Reconnection
wait_for_connection:
delay: 15
timeout: 180
But whatever the values I would set for async or poll, Ansible would hang indefinitely. On the remote, I could see that the interface was brought down and then nothing. So obviously, nothing was done asynchronously, and as soon as the interface was down, the script could not continue. Probably, the process was killed by the termination of the ssh session.
Then I read that when doing this, Ansible had no time to properly spawn the process and disconnect. It needed the process to wait a bit before cutting the connection short. So I modified the playbook:
- name: Hot Rename Main Iface
become: true
shell:
cmd: |
sleep 5 # <-- Wait for Ansible disconnection
ip link set oldiface down
ip link set oldiface name newiface
ip link set newiface up
async: 0
poll: 0
register: asynchotrename
- name: Wait For Reconnection
wait_for_connection:
delay: 15
timeout: 180
But this did nothing. Ansible still hangs indefinitely, while nothing happens on the remote after the ip link down statement.
Then, I figured out that maybe I had to force send the subprocess to the background, even if this would mean not making use of Ansible's asynchronous feature and so not being able to possibly come back later to check if everything went fine (although of course if that's the case, chances are that the remote is unreachable anyway). I still kept the async and poll values, just to ensure that Ansible would disconnect properly, even if obviously it would do this only once the script had returned. At least, this would prevent some errors that I would have to mask with ignore_errors: true.
I may try without someday, to see if I can just remove these async and poll entirely. (Edit: Done, and it works. No errors to mask.)
The complete playbooks steps ended being (for those interrested, although I'm not going to explain in this post why I had to order the statements this way):
- name: Hot Rename Main Iface
become: true
shell:
cmd: |
(
sleep 5 && \
ip link set oldiface down && \
ip link set oldiface name newiface && \
ip link set newiface up && \
nmcli networking off && \
sleep 1 && \
nmcli networking on && \
sleep 5 && \
systemctl restart sshd
)&
async: 90
poll: 0
register: asynchotrename
- name: Wait For Reconnection
wait_for_connection:
delay: 15
timeout: 180
But then I read that if I use poll: 0, I have to manually cleanup the async job cache. So I added this task:
- name: Cleanup Leftover Async Files
async_status:
jid: "{{ asynchotrename.ansible_job_id }}"
mode: cleanup
result: FAILED! => {"ansible_job_id": "603790343886.29503", "changed": false, "finished": 1, "msg": "could not find job", "started": 1}
I'm totally puzzled. Ansible doesn't even seem to consider the task as an async job.
How to spawn an asynchronous task in Ansible??

Answer 1

During research regarding Ansible doesn't return job_id for async task I've setup a small test on a RHEL 7.9.9 system with Ansible 2.9.25 and Python 2.7.5 which seems to be working so far.
- name: Start async job
systemd:
name: network
state: restarted
async: 60 # 1min
poll: 0
register: network_restart
- name: Wait shortly before check
pause:
seconds: 5
- name: Check async status
async_status:
jid: "{{ network_restart.ansible_job_id }}"
changed_when: false
register: job_result
until: job_result.finished
retries: 6
delay: 10
Because of your comment
Ansible had no time to properly spawn the process and disconnect. It needed the process to wait a bit before cutting the connection short.
and the documentation of Run tasks concurrently: poll = 0
If you want to run multiple tasks in a playbook concurrently, use async with poll set to 0. When you set poll: 0, Ansible starts the task and immediately moves on to the next task without waiting for a result.
I've included the
- name: Wait shortly before check
pause:
seconds: 5
resulting into an execution of
TASK [Start async job] *****************************************************************************************************************************************
changed: [test1.example.com]
Saturday 06 November 2021 17:20:43 +0100 (0:00:02.287) 0:00:10.228 *****
Pausing for 5 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
TASK [Wait shortly] ********************************************************************************************************************************************
ok: [test1.example.com]
Saturday 06 November 2021 17:20:48 +0100 (0:00:05.057) 0:00:15.285 *****
TASK [Check async status] **************************************************************************************************************************************
ok: [test1.example.com]
As you can see the pausing message came almost instantly and seconds before task name message.
On the test host it is seen that the network interface restarted
sudo systemctl status network
● network.service - LSB: Bring up/down networking
Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled)
Active: active (exited) since Sat 2021-11-06 17:20:46 CET; 874ms ago
Regarding
... as soon as the interface was down, the script could not continue. Probably, the process was killed by the termination of the ssh session.
I am too renaming interfaces, frequently during baseline setups like
- name: Make sure main network interface is named correctly
shell:
cmd: nmcli conn mod "ens192" connection.id "eth0"
- name: Gather current interface configuration
shell:
cmd: nmcli conn show eth0
register: nmcli_conn
- name: STDOUT nmcli_conn
debug:
msg: "{{ nmcli_conn.stdout_lines }}"
I have only to make sure before that the interfaces can be managed by NetworkManager. An asynchronous task isn't necessary in my setups to have a reliable restart of the network interfaces, also not for restarting sshd.
By using NetworkManager more advanced task are possilbe later like
- name: Configure DNS resolver
nmcli:
conn_name: eth0
type: ethernet
dns4_search:
- dns.example.com
state: present