I have a Spark installation running under YARN on a remote cluster, with a firewall between me and the head node. I can use a ssh tunnel to access the head node:
> ssh -N -f -L 10000:remotenode:10000 between_machine
and this setup works, for example, to access a HiveServer2 running on remotenote. If Spark was running in cluster mode, I would need to just do the same for the 7077 port and direct the pyspark client to localhost with
> ssh -N -f -L 7077:remotenode:7077 between_machine
> ./pyspark --master spark://localhost:7077
How can I do that with Spark running under the YARN scheduler?
If you are looking for a port to connect, here is a quote from the doc:
You can access this interface by simply opening
http://:4040 in a web browser. If multiple SparkContexts
are running on the same host, they will bind to successive ports
beginning with 4040 (4041, 4042, etc).
If you are just looking for a more universal way to get to the host via ssh "tunnel", you could try ssh working as socks proxy:
ssh user#host -D 20000
And then configuring your browser to connect via socks proxy (host - localhost, port - 20000).
Related
I have the problem that at work I can not connect via network to expo, so I need to use tunnel, which is fine. However sometimes the tunnel is really slow destroying any developer expierience.
Since I can also host expo locally on localhost I had the idea of simply ssh-tunneling to a remote server that has an open port.
my remote host runs ubuntu
so i SSH there like so:
ssh -R 0.0.0.0:19000:0.0.0.0:19000 user#ip
in order for this to work i also added
GatewayPorts clientspecified
to my /etc/ssh/sshd_config
...
sudo netstat -plutn
shows me
tcp 0 0 0.0.0.0:19000 0.0.0.0:* LISTEN 20183/2
so accepting requests (i also tried to forward port 19001 to get something back when i enter it in the browser which worke fine)
However when i enter:
exp://serverip:19000 into the expo client on my android phone he can't connect.
Any ideas on help?
It looks like Expo uses multiple ports 19000, 19001, and 19002. So you will need to forward all of these.
e.g.
$ ssh -f -N -R 19000:localhost:19000 user#ip
$ ssh -f -N -R 19001:localhost:19001 user#ip
$ ssh -f -N -R 19002:localhost:19002 user#ip
Also, you can set the REACT_NATIVE_PACKAGER_HOSTNAME environment variable to use the remote host.
$ export REACT_NATIVE_PACKAGER_HOSTNAME="ip"
$ expo start
Im experiencing a problem replicate my putty ssh tunneling with Cmder bash (on windows machine).
1. I want to access web interface on port 7183 on server_2. To get there I have to go through jump_server first and and tunnel twice, as from the jump_server, only visible port is 22.
Steps with putty:
1. connect to jump_server with tunnel (L22 server_2:22) using username_1
2. connect to localhost with tunnel (L7183 localhost:7183) using username_2
After that, Im able to access the web interface when I type localhost:7183 into browser on my local machine.
Now Im trying to reproduce this in Cmder, but I havent been able to do that with one big command, nor 2 separate commands:
ssh -L 7183:localhost:7183 username_1#jump_server ssh -L 22:localhost:22 -N username_2#server_2 -vvv
This is only the last command I used as I tried interchanging ports and hosts without success.
2. Is the syntax different when I want to open port 12345 on my local machine and have it forwarded to port 21050 on server_2 or that would be remote tunneling?
Finally managed to achieve the 1. question with:
ssh username_1#jump_server -L 22:server_2:22 -N -vvv
ssh -L 7183:localhost:7183 username_2#localhost
Now Im albe to access the web interface from server_2 on my localhost:7183
I am trying to use dynamic port forwarding to ssh to various servers in my university from home.
I used local port forwarding successfully:
ssh -L 10001:server1:22 my_user#proxy_server
ssh -p 127.0.0.1:10001
but I am using 12 servers, and I'd like to create only one tunnel (dynamic).
Is it possible to configure ssh to use a SOCKS proxy?
You can add the '-L' option several times to have more then one forwarding through one ssh connection.
Like
ssh -L 10001:server_1:22 -L10002:server_2:22 -L ... my_user#proxy_server
I've long seeked a solution to tunnel to a machine behind a firewall, passing VNC (or other ports) through. Like explained in this old usenet post, which I'll recap here:
I have to log through an intermediate machine, something like:
local $ ssh interim
interim $ ssh remote
remote $ ...any commands...
This works fine. But now I am trying to tunnel a vnc session from remote to local and I can't find the magic incantation, using either one or two steps.
I recently found a wonderfully simple and adaptable solution: simply tunnel the ssh to the target system through the connection to the firewall. Like such:
local $ ssh -L 2222:remote:22 interim
interim $ ...no need to do anything here...
In another local console you connect to localhost on port 2222, which is actually your remote destination:
local $ ssh -C -p 2222 -L 5900:localhost:5900 localhost
remote $ ...possibly start you VNC server here...
In yet another local console:
local $ xtightvncviewer :0
It's that simple. You can add any port forwarding you want to the 2nd command (-L localport:localhost:remoteport) just like if there wasn't any intermediate firewall. For instance for RDP: -L 3389:localhost:3389
What is the best way to run a command through an ssh tunnel? Not running the command on the remote server, but forwarding locally created traffic to it through a tunnel.
For example, wget. How can I run wget on my local machine so that it tunnels to a VPS that I have ssh access to? Both the local and remote servers are running Arch Linux.
From what i got, your question is about a proxy, specifically, a socks proxy . So the easiest way to do this is to show you some examples. and work from that.
ssh -NT -D 1080 user#remoteserver.net
What this is doing is keeping the port 1080 open as a socks5 proxy -NT keeps the connection alive.
Then what needs to be done is to use the proxy. I dont think wget has the option to use a socks proxy so we will have to use curl as i know that can use them.
curl -O --socks5-hostname 127.0.0.1:1080 url
This connects to the socks proxy and downloads from url.