How to prevent "connection refused" with SOCKS on dataproc? - ssh

Question

I'm trying to set up a SOCKS connection to my dataproc spark cluster following the Google Jupyter guide, but I keep getting "connection refused" errors after launching the browser, Chrome:
channel 4: open failed: connect failed: Connection refused
channel 5: open failed: connect failed: Connection refused
channel 6: open failed: connect failed: Connection refused
channel 7: open failed: connect failed: Connection refused
channel 8: open failed: connect failed: Connection refused
channel 5: open failed: connect failed: Connection refused
channel 5: open failed: connect failed: Connection refused
channel 5: open failed: connect failed: Connection refused
channel 16: open failed: connect failed: Connection refused
channel 16: open failed: administratively prohibited: open failed
channel 17: open failed: administratively prohibited: open failed
channel 18: open failed: administratively prohibited: open failed
channel 16: open failed: connect failed: Connection refused
channel 18: open failed: connect failed: Connection refused
channel 18: open failed: connect failed: Connection refused
channel 18: open failed: connect failed: Connection refused
This happens with both --proxy-server="socks5://localhost:1080" and --proxy-server="socks5://127.0.0.1:1080"

Answer 1

So, I'm not 100% sure where the "administratively prohibited" messages come from, but in my experience those have always been false alarms, and I see those open failed: administratively prohibited: open failed messages even when my socks proxy is functioning correctly as expected.
As for the actual problem, due to the way things like YARN bind their web services, I've gotten Connection refused if trying to access the YARN UI with http://localhost:8088 instead of http://<master-hostname>:8088. This matches the behavior running get inside the cluster:
dhuo#dhuo-jupyter-m:~$ wget http://localhost:8124
...
Saving to: ‘index.html.13’
index.html.13 100%[=============================================================================================================================================================================>] 11.41K --.-KB/s in 0s
2016-07-15 23:26:25 (222 MB/s) - ‘index.html.13’ saved [11686/11686]
dhuo#dhuo-jupyter-m:~$ wget http://localhost:8088
--2016-07-15 23:26:28-- http://localhost:8088/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8088... failed: Connection refused.
Connecting to localhost (localhost)|127.0.0.1|:8088... failed: Connection refused.
dhuo#dhuo-jupyter-m:~$ wget http://`hostname`:8124
...
Saving to: ‘index.html.14’
index.html.14 100%[=============================================================================================================================================================================>] 11.41K --.-KB/s in 0s
2016-07-15 23:26:34 (260 MB/s) - ‘index.html.14’ saved [11686/11686]
dhuo#dhuo-jupyter-m:~$ wget http://`hostname`:8088
...
Saving to: ‘index.html.15’
index.html.15 100%[=============================================================================================================================================================================>] 10.81K --.-KB/s in 0s
2016-07-15 23:26:37 (248 MB/s) - ‘index.html.15’ saved [11067/11067]
As you can see, this is different from the Jupyter behavior (which I ran on port 8124), where the Jupyter webapp works correctly resolving localhost:8124 on the master. Since the name resolution with those linked instructions should be happening on the master, the browser's behavior resolving hosts will be the same as running wget in the node you've tunneled into.
So if you just use your master's hostname instead of localhost it should work.