I have some difficulities on understanding how ssh works behind the scene when I run it with a command string.
Normally, I type ssh rick#1.2.3.4 then I am logged into the remote machine and run some commands. If I don't use nohup or disown, once I close the session, all running processes started by ssh will stop. That's the ordinary case.
But if I run ssh with a command string, things become different. The process started by ssh won't stop if I close the session.
For example:
Run from local command line: ssh rick#1.2.3.4 "while true; do echo \"123test\" >> sshtest.txt"; done
Run a remote script ssh rick#1.2.3.4 './remoteScript_whichDoTheSameAsAbove.sh'
After closing the session by Ctrl + C or kill pid on the local machine, on the remote machine I see the process still running with ps -ef .
So my question is, could someone make a short introduction on how ssh works when I run it with a command string like above?
Also, I get very confused when I see these 2 related questions during searching:
Q1: Getting ssh to execute a command in the background on target machine . What is this question asking for? Isn't ssh rick#1.2.3.4 'some command' already run as a seperate shell/pts? I don't understand what "in the background" is for.
Q2: Keep processes running after SSH session disconnects Isn't simply running a remote script meets his requirement? ssh rick#1.2.3.4 "./remoteScript.sh. I get very confused when I see so many "magic" answers under that question.
Related
Here is the problem statement:
Suppose I am on an EC2 instance A, and run an Ansible script which does the following tasks:
1. Create an EC2 instance B
2. SSH into it
3. Trigger an Ansible script which is on B, with the simple `ansible-playbook <pb_on_B>.yml` [B is being provisioned from an AMI]
So, what will happen if the instance A gets terminated after task 3 gets started?
Will the Ansible script which is triggered in B, finish to completion?
[W]hat will happen if the instance A gets terminated after task 3 gets started?
Will the Ansible script which is triggered in B, finish to completion?
You can't tell what would happen with 100% certainty.
It depends on the shell configuration (for example TMOUT in bash), SSH daemon configuration (TCPKeepAlive, ClientAliveInterval parameters), timing, network conditions and whether A will close the session with (FIN) or drop without notifying A.
Most likely the playbook execution would get interrupted.
If SSH daemon on B cannot contact the SSH client on A (for example to print out Ansible execution log) and it gets the TCP RST packet, it will drop the session killing the SSH session's child processes, including the shell and ansible-playbook. However the session might also remain active until timeout and the playbook might finish before it occurs.
If ansible-playbook executable was be called through the nohup command (or in a screen or tmux session), it won't be interrupted upon SSH session disconnect (and shell session closure).
Note: when you use nohup the standard output will be redirected to a file nohup.out. Refer to the answers under this question to learn the options.
Also check this answer on Unix.SE which describes the technicalities behind the command.
Can Ansible task provisioned from Ansible run on a remote host without SSH?
Yes, with ansible-pull:
Should you want to invert the architecture of Ansible, so that nodes check in to a central location, instead of pushing configuration out to them, you can.
The ansible-pull is a small script that will checkout a repo of configuration instructions from git, and then run ansible-playbook against that content.
If the idea is to preserve an SSH session in instance B, without worrying about the life/death of instance A, you could try and run your ansible plays in tmux on instance B. Your workflow will be modified like this
Create an EC2 instance B
SSH into it
Instal tmux - apt-get install tmux
Start a tmux session tmux new -s ansible
Trigger an Ansible script which is on B, with the simple ansible-playbook <pb_on_B>.yml
I have a rundeck job that reboots a server, it sends the command "sudo reboot". This works and the server is rebooting.
The problem is that rundeck doesn't get a signal back so the job fails.
Is there a way to make this work and get a complete signal back in rundeck?
Perhaps wrap your command in a script, background the reboot operation, and return 0? I'm doing something similar with a set of development VMs, but I'm using virsh. I don't see why this couldn't be done with a physical server:
#!/bin/bash
ssh rundeck#yourserver sudo reboot &
exit 0
You may need to experiment a bit with the ssh options (perhaps '-f' and/or '-n') to get this to work properly.
Well playing around now I just used as Local Command step:
ssh ${node.username}#${node.hostname} "reboot & exit"
The return code is ZERO and everybody is happy.
I use SSH to connect to my AWS EC2 instances and run code that takes a long time to complete. I find that if my local computer sleeps (or even if I leave it unattended for a bit) the SSH connection hangs up (which is not fatal in itself) but this seems to terminate the code on the EC2 instance that I launched using SSH.
Also, I use SSH to locally monitor the exception of my remote code, so even if there's a way to tell the remote process to stay alive after SSH has gone, I still need a way to locally see the output of the process as it continues to run (without SSH).
How do I keep code running on my AWS EC2 instance after SSH has hung up; how can I monitor the output of such a process?
When you close your tty (ssh close in your case) your process gets a SIGHUP and the default action on SIGHUP is to terminate. To avoid that you can use the command nohup to trap and not send the SIGHUP to your command, or trap the SIGHUP in your code and ignore it.
There are a bunch of ways to track a background process, but perhaps the easiest is to have it write to a file and in that other ssh you can read that file. If your process is really a command on the command line you can redirect its standard output and standard error to a file. When such a file keeps getting new content, it may be annoying to keep reading it to refresh, in which case the command "tail -f" handy.
Here is how you can config your ssh connection to stay alive :
vi ~/.ssh/config # on your client side
add this line to engage sending a "null packet" every 120 seconds :
ServerAliveInterval 120
If you own the server side do a similar change :
vi /etc/ssh/sshd_config
add these lines at bottom of config file
ClientAliveInterval 120
ClientAliveCountMax 720
this is for linux YMMV on other OS settings
Use screen
local> ssh ...
remote> screen
remote+screen> python long_running.py ...
You can then detach from screen and even disconnect from SSH, and when you return by SSHing back in again, you can
remote> screen -r
to reconnect to your running code.
tasks:
name: Connect to Windows hpst machine through an ssh script
script : windows_connect.ssh
name: Run the bat file
script : C:\OV\sentenv.bat
name: To exit from the remote machine
shell: exit
This is what my playbook looks like.
Windows_connect.sh contains a script to connect to a windows machine via ssh.
ssh root#host
So ideally shouldnt ansible prompt me for a password?
Instead it gets stuck.
Please help me with the same
Ansible opens at least one new ssh connection for every task in the play.
So the idea to open a ssh connection with the first task and then reuse the connection in the later tasks will not work. Actually Ansible won't be able to execute the first task because the script module you are trying to use will already need a working connection to the remote host.
I have found little information on how many ssh connection Ansible will use to execute a task so I might be wrong here.
You should consult the Ansible documentation for Window support and implement a solution based on the winrm Python module.
I am trying to run a long java program on a server via ssh, I tried using:
command &
Which would work fine on my local machine, but when I terminate my local ssh connection, the program also terminates. Is there a way I can run this and have the output go to some text file so I can login and check completion at a later date?
You can use nohup
nohup command &