When I'm teleworking, I need to access some internal web servers. I use ssh -f -N -D 4000 someserver.mywork.com on my home computer to setup local dynamic port forwarding. Then, I configure my web browser to use localhost port 4000 as a SOCKS host, and everything works great, even with HTTPS.
The problem is that the proxy stops working every couple of days. When this happens, the ssh process prints messages like the following:
accept: Too many open files
In this scenario, I have to kill the ssh process and restart it in order to get it working again. Based on my research into this error message, I could increase the limit on the number of open files, but that doesn't seem like a permanent or an ideal solution.
I was hoping autossh might be able to monitor the connection and restart it automatically. Is that possible?
I have tried the following command:
autossh -f -M 0 -N -D 4000 someserver.mywork.com
But it didn't work. The proxy stopped working, and autossh did not restart it. Any suggestions or alternative solutions to automatically restarting my ssh proxy?
Related
I have a powershell script that keeps closing ```ssh jumpbox_user#jumpbox_ip -p 22 -L port:server_address:port -i '~/.ssh/id_rsa' -N
The issue I have is that the tunnel will start but will then, randomly, it will close. I have this issue no matter how I start the SSH tunnel. I get the feeling that it has something to do, potentially, with my home internet? I don't seem to have this issue at work or at other locations. I've purchased a new router and a new modem to no effect, is there a way to check what is going wrong???
I'm trying to setup a an ssh tunnel with remote port forwarding. The idea is the have a VPS act as a means to ssh into remote deployed systems (which currently incorporate a Raspberry Pi). Everything seems to work, but I run into issues when trying to move all arguments into the ~/.ssh/config file.
what does work is the setting of the HostName, User, Port and IdentityFile. However setting the RemoteForward parameter does not seem to work.
The following works:
ssh -R 5555:localhost:22 ssh-tunnel
How ever when using the following line in the config file;
Host ssh-tunnel
...
RemoteForward 5555 localhost:22
The following command returns the message "Bad remote forwarding specification 'ssh-tunnel'"
ssh -R ssh-tunnel
Obvious I found the answer almost immediately after posting the question. Using the -R flag requires you to set the remote forwarding in the command line call. However because remote forwarding is set in the config file you shouldn't add it to the command. However something confusing occurs in that aside from setting up the tunnel you also ssh into the remote server. To avoid this add the -f and the -N flag. This results in the following command:
ssh -f -N ssh-tunnel
I have a client and a server. there is a firewall between them such that the client can ssh to the server, but the server can not ssh to the client.
i'd like to set up an ssh tunnel from the client to the server that would allow the server to make ssh connections back to the client.
I know there are several posts on ssh tunneling, but have not found anything that talks about this. I know its possible I just can't find it.
thanks for your time.
Try on the client
ssh -R 45849:127.0.0.1:22 <server_user#server>
Then on the server you should be able to do
ssh -p 45849 <client_user#127.0.0.1>
That's if I got everything right, otherwise try switching the 45849 and the 22 on the '-R'.
Later edit:
I've just tested it, it's all good (I was on the phone first). For something to go wrong you will have to have at least one of the following problems:
port forwarding disable on either of the servers. Check with sudo grep AllowTcpForwarding /etc/ssh/sshd_config, make sure it says yes
sshd server binding/listening interface on the client. Check with sudo netstat -lntp | grep ssh, you should see "0.0.0.0:22" listed (not sure if ":::22" helps), if you find another IP address use it instead of the "127.0.0.1" in the "-R" argument.
First off: I have read the answers to similar questions on SO, but none of them worked.
IMPORTANT NOTE: The answer below is still valid, but maybe jump to the end for an alternative.
The situation:
App with GUI is running in a docker container (CentOS 7.1) under Arch Linux. (machine A)
Machine A has a monitor connected to it.
I want to access this GUI via X11 forwarding on my Arch Linux client machine. (machine B)
What works:
GUI works locally on machine A (with /tmp/.X11-unix being mounted in the Docker container).
X11 forwarding of any app running outside of docker (X11 forwarding is set up and running properly for non-docker usage).
I can even switch the user while remotely logged in, copy the .Xauthority file to the other user and X11 forwarding works as well.
Some setup info:
Docker networking is 'bridged'.
Container can reach host (firewall is open).
DISPLAY variable is set in container (to host-ip-addr:10.0 because of TCP port 6010 where sshd is listening).
Packets to X forward port (6010) are reaching the host from the container (tcpdump checked).
What does not work:
X11 forwarding of the Docker app
Errors:
X11 connection rejected because of wrong authentication.
xterm: Xt error: Can't open display: host-ip-addr:10.0
Things i tried:
starting client ssh with ssh -Y option on machine B
putting "X11ForwardTrusted yes" in ssh_config on machine B
xhost + (so allow any clients to connect) on machine B
putting Host * in ssh_config on machine B
putting X11UseLocalhost no in sshd_config on machine A (to allow non-localhost clients)
Adding the X auth token in the container with xauth add from the login user on machine A
Just copying over the .Xauthority file from a working user into the container
Making shure .Xauthority file has correct permissions and owner
How can i just disable all the X security stuff and get this working?
Or even better: How can i get it working with security?
Is there at least a way to enable extensive debugging to see where exactly the problem is?
Alternative: The first answer below shows how to effectively resolve this issue. However: I would recommend you to look into a different approach all together, namely VNC. I personally switched to a tigerVNC setup that replaces the X11 forwarding and have not looked back. The performance is just leagues above what X11 forwarding delivered for me. There might be some instances where you cannot use VNC for whatever reason, but i would try it first.
The general setup is now as follows:
-VNC server runs on machine A on the host (not inside a docker container).
-Now you just have to figure out how to get a GUI for inside a docker container (which is a much more trivial undertaking).
-If the docker container was started NOT from the VNC environment, the DISPLAY variable maybe needs ajdusting.
Thanks so much #Lazarus535
I found that for me adding the following to my docker command worked:
--volume="$HOME/.Xauthority:/root/.Xauthority:rw"
I found this trick here
EDIT:
As Lazarus pointed out correctly you also have to set the --net=host option to make this work.
Ok, here is the thing:
1) Log in to remote machine
2) Check which display was set with echo $DISPLAY
3) Run xauth list
4) Copy the line corresponding to your DISPLAY
5) Enter your docker container
6) xauth add <the line you copied>*
7) Set DISPLAY with export DISPLAY=<ip-to-host>:<no-of-display>
*so far so good right?
This was nothing new...however here is the twist:
The line printed by xauth list for the login user looks something like this (in my case):
<hostname-of-machine>/unix:<no-of-display> MIT-MAGIC-COOKIE-1 <some number here>
Because i use the bridged docker setup, the X forwarding port is not listening locally, because the sshd is not running in the container. Change the line above to:
<ip-of-host>:<no-of-display> MIT-MAGIC-COOKIE-1 <some number here>
In essence: Remove the /unix part.
<ip-of-host> is the IP address where the sshd is running.
Set the DISPLAY variable as above.
So the error was that the DISPLAY name in the environment variable was not the "same" as the entry in the xauth list / .Xauthority file and the client could therefor not authenticate properly.
I switched back to an untrusted X11 forwarding setting.
The X11UseLocalhost no setting in the sshd_config file however is important, because the incomming connection will come from a "different" machine (the docker container).
This works in any scenario.
Install xhost if you don't have it. Then, in bash,
export DISPLAY=:0.0
xhost +local:docker
After this run your docker run command (or whatever docker command you are running) with -e DISPLAY=$DISPLAY
It works usually via https://stackoverflow.com/a/61060528/429476
But if you are running docker with a different user than the one used for ssh -X into the server with; then copying the Xauthority only helped along with volume mapping the file.
Example - I sshed into the server with alex user.Then ran docker after su -root and got this error
X11 connection rejected because of wrong authentication.
After copying the .XAuthoirty file and mapping it like https://stackoverflow.com/a/51209546/429476 made it work
cp /home/alex/.Xauthority .
docker run -it --network=host --env DISPLAY=$DISPLAY --privileged \
--volume="$HOME/.Xauthority:/root/.Xauthority:rw" \
-v /tmp/.X11-unix:/tmp/.X11-unix --rm <dockerimage>
More details on wiring here https://unix.stackexchange.com/a/604284/121634
Some clarifying remarks. Host is A, local machine is B
Ive edited this post to note things that I think should work in theory but haven't been tested, vs things I know to work
Running docker non-interactively
If your docker is running not interactively and running sshd, you can use jumphosts or proxycommand and specify the x11 client to run. You should NOT volume share your Xauthority file with the container, and sharing -e DISPLAY likely has no effect on future ssh sessions
Since you essentially have two sshd servers, either of the following should work out of the box
if you have openssh-client greater than version 7.3, you can use the following command
ssh -X -J user-on-host#hostmachine,user-on-docker#dockercontainer xeyes
If your openssh client is older, the syntax is instead
(google says the -X is not needed in the proxy command, but I am suspicious)
ssh -X -o ProxyCommand="ssh -W %h:%p user-on-host#hostmachine" user-on-docker#dockermachine xeyes
Or ssh -X into host, then ssh -X into docker.
In either of the above cases, you should NOT share .Xauthority with the container
Running docker interactively from within the ssh session
The easiest way to get this done is to set --net=host and X11UseLocalhost yse.
If your docker is running sshd, you can open a second ssh -X session on your local machine and use the jumphost method as above.
If you start it in the ssh session, you can either -e DISPLAY=$DISPLAY or export it when you're in. You might have to export it if you attach to an exiting container where this line wasn't used.
Use these docker args for --net host and x11uselocalhost yes
ssh -X to host
-e DISPLAY=$DISPLAY
-v $HOME/.Xauthority:/home/same-as-dash-u-user/.Xauthority
-u user
What follows is explanation of how everything works and other approaches to try
About Xauthority
ssh -X/-Y set up a session key in the hosts Xauthority file, and then sets up a listen port on which it places an x11 proxy that uses the session key, and converts it to be compatible with the key on your local machine. By design, the .Xauthority keys will be different between your local machine and the host machine. If you use jumphosts/proxycommand the keys between the host and the container will yet again be different from each other. If you instead use ssh tunnels or direct X11 connection, you will have to share the host Xauthority with the container, in the case of sharing .Xauthority with the container, you can only have one active session per user, since new sessions will invalidate the previous ones by modifying the hosts .Xauthority such that it only works with that session's ssh x11 proxy
X11UserLocalhost no theory##
Even Though X11UseLocalhost no causes the x server to listen on the wildcard address, With --net host I could not redirect the container display to localhost:X.Y where x and why are from the host $DISPLAY
X11UseLocalhost yes is the easy way
If you choose X11UseLocalhost yes the DISPLAY variable on the host becomes localhost:X:Y, which causes the ssh x11 proxy to listen only on localhost port x.
If X11UseLocalhost is no, the DISPLAY variable on the host becomes the host's hostname:X:Y, which causes the xerver to listen on 0.0.0.0:6000+X and causes xclients to reach out over the network to the hostname specified.
this is theoretical, I don't yet have access to docker on a remote host to test this
But this is the easy way. We bypass that by redirecting the DISPLAY variable to always be localhost, and do docker port mapping to move the data from localhost:X+1.Y on the container, to localhost:X.Y on the host, where ssh is waiting to forward x traffic back to the local machine. The +1 makes us agnostic to running either --net=host or --net=bridge
setting up container ports requires specifying expose in the dockerfile and publishing the ports with the -p command.
Setting everything up manually without ssh -X
This works only with --net host. This approach works without xauth because we are directly piping to your unix domain socket on the local machine
ssh to host without -X
ssh -R6010:localhost:6010 user#host
start docker with -e DISPLAY=localhost:10.1 or export inside
in another terminal on local machine
socat -d -d TCP-LISTEN:6010,fork UNIX-CONNECT:/tmp/.X11-unix/X0
In original terminal run xclients
if container is net --bridged and you can't use docker ports, enable sshd on the container and use the jumphosts method
I'm using SSH to access my university's afs system. I like to use rmate (remote TextMate), which requires SSH tunneling, so I included this alias in my .bashrc.
alias sshr=ssh -R 52698:localhost:52698 username#corn.myschool.edu
It has always worked until now.
I had the same problem. In order to find the port that is already open, you have to issue this command on the 'corn.myschool.edu' computer:
sudo netstat -plant | grep 52698
And then kill all of the processes that come up with this (replace xxxx with the process ids)
sudo kill -9 xxxx
(UPDATED: changed the option to be -plant as it is a nice mnemonic)
I had another SSH connection open. I just needed to close that connection before I opened my SSH tunnel.
Further Explanation:
Once one ssh connection has been established, subsequent connections will produce a message:
Warning: remote port forwarding failed for listen port 52698
This message is harmless, as the forward can only be set up once and one forward will work for all ssh connections to the same machine. The original ssh session that opened the forward will stay open when you exit the shell until all remote editing sessions are finished.
I experienced this problem, but it was while connecting to a server on which I don't have sudo priviliges, so the top response suggesting runing sudo netstat ... wasn't feasible for me.
I eventually figured out it was because there were still instances of rmate running, so I used ps to list the running processes and then kill -9 pid (where pid is the process ID for rmate).
This solved my problem reported here as well. To avoid this notification "AllowTcpForwarding" should be enabled in SSH config.
In my case, the problem was that the remote system didn't have DNS properly set up, and it couldn't even resolve its own hostname. Make sure you have a working DNS in /etc/resolv.conf at the remote system.