How do I reattach to a detached mosh session or otherwise get rid of
Mosh: You have a detached Mosh session on this server (mosh [XXXX]).
i.e. what's the mosh equivalent of
screen -D -R
or possibly
screen -wipe
Furthermore, where can this answer be found in documentation?
For security reasons, you can not reattach, see https://github.com/keithw/mosh/issues/394
To kill the detached session, use the PID number displayed in that message (that's the 'XXXX' part.) For example, if you see --
Mosh: You have a detached Mosh session on this server (mosh [12345]).
And can run this command:
kill 12345
Also, to close all mosh connections you can:
kill `pidof mosh-server`
Note that if you are currently connected via mosh, this last command will also disconnect you.
To my amazement, I used CRIU (https://criu.org) to checkpoint and restart a mosh client and it worked.
Shocking.
Find your mosh-client's PID:
$ ps -ef | grep mosh
Then, install CRIU according to their instructions.
Then, checkpoint it like this:
$ mkdir checkpoint
$ sudo ./criu dump -D checkpoint -t PID --shell-job
Then, restore it:
$ sudo ./criu restore -D checkpoint --shell-job
And, there it is. Your mosh client is back.
One thing to note, however, is that if your laptop reboots (which is the whole point of what we're trying to protect against), mosh uses a monotonic clock to track time on the client side, which doesn't work across reboots. This will NOT work, however, if your laptop just flat out crashes it won't work because mosh sequence numbers will be out of sync with the version that was checkpointed (the binary will resume, but communication will stop).
In order to fix this, you need to tell mosh to stop doing that and download the mosh source code. Then, edit this file:
cd mosh
vim configure.ac
Then, search for GETTIME and comment out that line.
Then do:
autoreconf # or ./autogen.sh if you've just cloned it for the first time
./configure
make
make install
After that, your CRIU-checkpointed mosh client sessions will survive reboots.
(Obviously you'd need to write something to perform the checkpoints regularly enough to be useful. But, that's an exercise for the reader).
I realize this is an old post, but there is a very simple solution to this, as suggested by Keith Winstein, mosh author, here: https://github.com/mobile-shell/mosh/issues/394
"Well, first off, if you want the ability to attach to a session from multiple clients (or after the client dies), you should use screen or tmux. Mosh is a substitute (in some cases) for SSH, not for screen. Many Mosh users use it together with screen and like it that way."
Scenario: I'm logged into a remote server via mosh. I've then run screen and have a process running in the screen session, htop, for example. I lose connection (laptop battery dies, lose network connection, etc.). I connect again via mosh and get that message on the server,
Mosh: You have a detached Mosh session on this server (mosh [XXXX]).
All I have to do is kill the prior mosh session
kill XXXX
and reattach to the screen session, which still exists.
screen -r
Now, htop (or whatever process was running) is back just as it was without interruption.This is especially useful for running upgrades or other processes that would leave the server in a messy, unknown state if suddenly interrupted. I assume you can do the same with tmux, though I have not tried it. I believe this is what Annihilannic and eskhool were suggesting.
As an addition to Varta's answer, I use the following command to close all mosh connections except the current one:
pgrep mosh-server | grep -v $(ps -o ppid --no-headers $$) | xargs kill
As #varta pointed out, the mosh owners are very against reattaching from different clients for security reasons. So if your client is gone (e.g. you restarted your laptop) your only option is to kill the sessions.
To kill only detached sessions you can use the following line (which I have as an alias in my .bashrc).
who | grep -v 'via mosh' | grep -oP '(?<=mosh \[)(\d+)(?=\])' | xargs kill
That command depends on the fact that who lists connected users including mosh sessions, only attached mosh sessions have "via mosh", and that mosh sessions have their pid in square brackets. So it finds the pids for just the detached mosh sessions and passes them to kill using xargs.
Here is an example who result for reference:
$ who
theuser pts/32 2018-01-03 08:39 (17X.XX.248.9 via mosh [193891])
theuser pts/17 2018-01-03 08:31 (17X.XX.248.9 via mosh [187483])
theuser pts/21 2018-01-02 18:52 (mosh [205286])
theuser pts/44 2017-12-21 13:58 (:1001.0)
An alternative is to use the mosh-server environment variable MOSH_SERVER_SIGNAL_TMOUT. You can set it to something like 300 in your .bashrc on the server side. Then if you do a pkill -SIGUSER1 mosh-server it will only kill mosh-servers that have not been connected in the last 300 seconds (the others will ignore the SIGUSER1). More info in the mosh-server man page. I am using the command above because, once aliased, it seems simpler to me.
Note, as mentioned by #Annihilannic, if you are using tmux/screen inside your mosh sessions then those tmux/screen sessions are still around after you kill the mosh sessions. So you can still attach to them (so you really don't lose much by killing the mosh sessions themselves).
The answers here claiming that killing mosh-server is the only option are largely obsolete, as we can use criu and reptyr to recover and reattach arbitrary processes.
Not to mention that nowadays we can kill -USR1 mosh-server to only kill detached sessions in a clean and safe way, without resorting to unsafe who output or cumbersome commands to avoid killing our own session.
Next to the criu answer from Michael R. Hines, there is the slightly more "light-weight" reptyr which can be used to reattach processes started by mosh-server (i.e. not the mosh-server itself). I typically use
pstree -p <mosh-server PID>
to list the tree of processes under the detached mosh-server, and then
reptyr PID
to reattach the desired process to my current terminal. After repeating the procedure for all processes I care about, I
kill -USR1 <mosh-server PID>
whereas I take care to only kill sessions I know are mine (shared system).
Use ps command for getting the list of running task or use ps -ef | grep mosh
Kill the mosh PID using this command:
kill <pid>
Also, to close all mosh connections you can:
Note that if you are currently connected via mosh, then this also disconnect you
kill `pidof mosh-server`
Related
I have some difficulities on understanding how ssh works behind the scene when I run it with a command string.
Normally, I type ssh rick#1.2.3.4 then I am logged into the remote machine and run some commands. If I don't use nohup or disown, once I close the session, all running processes started by ssh will stop. That's the ordinary case.
But if I run ssh with a command string, things become different. The process started by ssh won't stop if I close the session.
For example:
Run from local command line: ssh rick#1.2.3.4 "while true; do echo \"123test\" >> sshtest.txt"; done
Run a remote script ssh rick#1.2.3.4 './remoteScript_whichDoTheSameAsAbove.sh'
After closing the session by Ctrl + C or kill pid on the local machine, on the remote machine I see the process still running with ps -ef .
So my question is, could someone make a short introduction on how ssh works when I run it with a command string like above?
Also, I get very confused when I see these 2 related questions during searching:
Q1: Getting ssh to execute a command in the background on target machine . What is this question asking for? Isn't ssh rick#1.2.3.4 'some command' already run as a seperate shell/pts? I don't understand what "in the background" is for.
Q2: Keep processes running after SSH session disconnects Isn't simply running a remote script meets his requirement? ssh rick#1.2.3.4 "./remoteScript.sh. I get very confused when I see so many "magic" answers under that question.
I'm curious about how does scp handles a situation when a binary file contains escape sequences - and, in particular, the Ctrl+C ("\0x03") character from the programmer's side of view.
I have already tried starting it in sink mode and sending it a "\0x03" character, but it clearly exited upon receiving it:
$ perl -e 'print "\x03"'|xsel
$ scp -t /tmp/somefile.txt
^C
$
However, transfering of a binary file that contains the same character doesn't fails, though I believe that it should.
I have also tried to read the scp.c:source function's source code to see if it attempts to perform any characters escape, but to my surprise it doesn't appears so.
The short answer is that the source scp instance communicates with the sink instance through a clean data pipe. Any byte values can be sent through the pipe, and no bytes receive any special treatment. This is the expected behavior for an ssh "shell" or "exec" channel with no PTY.
For the longer answer, I'm going to restrict this to OpenSSH scp running on unix systems. I assume that's the scenario that you're talking about.
The special behavior of keystrokes like Ctrl-C (interrupt the process) or Ctrl-D (end of file) is provided by the TTY interface. Programs like the ssh server use a unix feature called PTYs (pseudo-ttys) to provide a TTY interface for network clients. When you run a command like scp -t ... within an interactive session, you're running it in the context of a TTY (or PTY), and it's the TTY which would convert a typed Ctrl-C into an interrupt signal. Processes can disable this behavior, but the scp program doesn't do that, because it doesn't need to.
When you run scp in the normal way, like scp /some/local/file user#host:/some/remote/dir, the scp process that you launch runs a copy of ssh to make the connection to the remote system. The actual command that it runs will be something like this (simplified):
ssh user#localhost "scp -t /some/remote/dir"
In other words, it starts a copy of ssh which connects to the remote system and runs another copy of scp.
When ssh is run in this fashion, specifying a command to run on the remote system, it doesn't request a PTY for the channel by default. So the two scp instances will communicate with each other through a clean data pipe with no PTY involved.
I use SSH to connect to my AWS EC2 instances and run code that takes a long time to complete. I find that if my local computer sleeps (or even if I leave it unattended for a bit) the SSH connection hangs up (which is not fatal in itself) but this seems to terminate the code on the EC2 instance that I launched using SSH.
Also, I use SSH to locally monitor the exception of my remote code, so even if there's a way to tell the remote process to stay alive after SSH has gone, I still need a way to locally see the output of the process as it continues to run (without SSH).
How do I keep code running on my AWS EC2 instance after SSH has hung up; how can I monitor the output of such a process?
When you close your tty (ssh close in your case) your process gets a SIGHUP and the default action on SIGHUP is to terminate. To avoid that you can use the command nohup to trap and not send the SIGHUP to your command, or trap the SIGHUP in your code and ignore it.
There are a bunch of ways to track a background process, but perhaps the easiest is to have it write to a file and in that other ssh you can read that file. If your process is really a command on the command line you can redirect its standard output and standard error to a file. When such a file keeps getting new content, it may be annoying to keep reading it to refresh, in which case the command "tail -f" handy.
Here is how you can config your ssh connection to stay alive :
vi ~/.ssh/config # on your client side
add this line to engage sending a "null packet" every 120 seconds :
ServerAliveInterval 120
If you own the server side do a similar change :
vi /etc/ssh/sshd_config
add these lines at bottom of config file
ClientAliveInterval 120
ClientAliveCountMax 720
this is for linux YMMV on other OS settings
Use screen
local> ssh ...
remote> screen
remote+screen> python long_running.py ...
You can then detach from screen and even disconnect from SSH, and when you return by SSHing back in again, you can
remote> screen -r
to reconnect to your running code.
I am wanting to rsync files from my home computer to a cloud server. I am able to set up my continuous rsync with the following:
#!/bin/bash
while :
do
rsync -rav * --include=*.bz2 --exclude=*.* --exclude=ZIP.sh --exclude=UPLOAD.sh
--chmod=a+rwx user#server.com:/home/user/date
sleep 180
done
This of course will run continuously if i set up a keygen as here states. I want to run the rsync continuously with me entering the password the first time and after that it will continuously run until I push CTRL+C. Is there a way to do this?
Yes, using SSH connection sharing:
Add this top your ~/.ssh/config file:
ControlMaster auto
ControlPath /tmp/ssh_%r#%n:%p
ControlPersist 8h
Connection sharing means that all your SSH connections to the same server will share the same connection. This means you can skip the authentication process for all but the first connection. The ControlPersist setting controls how long the connection will idle for before being closed (8 hours means I can login in the morning, and the connection will still be active at the end of the day, but will have expired before the next day).
The ControlPath specifies where the cached sockets will live. It can be anywhere you like, and they can be called anything you like, but the /tmp directory will do fine, and the name must be unique to each user, server, and port you wish to use, or else you'll get clashes.
Incidentally, you should probably check out the lsyncd tool as an alternative to continuous active scanning. It uses kernel notifications to watch the file-system, and launches rsync only when something actually changes.
Ansible seems to be sending SIGHUP signals at the end of (certain?) tasks. This is a problem as the tasks call a bash script which in turn starts a server instance.
Now if the closing of Ansibles SSH session sends a SIGHUP, this will actually shutdown the server - the start of which was the key point of the Ansible task in the first place.
So, is there a way I can guarantee that Ansible will use SSH in a way that will not send a SIGHUP signal when closing the task/session?
I could theoretically start the bashscript using nohup. But this seems to be just a dirty workaround, as I know that SSH is capable of doing what I want: if I manually compose and pass the script command via SSH as such:
ssh user#server "scriptToStartMyServer.sh params"
.. then it works fine and no SIGHUP is sent (thus the server survives and is not shutdown immediately after being started).
Edit:
Sadly we cannot avoid using these bash scripts to start servers and we cannot really change them as this is something given to us by the customer.