I'm attempting to move data from one cluster to a new cluster without downtime.
So far, I have one node on the old cluster (this is a test setup) that's on a private subnet. I have a bastion host on that network that I connect through. My new cluster also has one node, and again, is behind a bastion, so inaccessible directly.
I was hoping I'd be able to create an SSH tunnel from my new cluster machine that connects to the machine in the old cluster (so something like localhost:9300 ==> remote:9300), and then be able to set the following in my elasticsearch.yml:
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]
network.publish_host: 127.0.0.1
network.bind_host: 127.0.0.1
I've tried this, but all I'm seeing is bound_address {inet[/127.0.0.1:9301]}, publish_address {inet[/127.0.0.1:9301]}
Has anyone managed to connect two clusters over an SSH tunnel with success?
Related
Currently, I have built a small datacenter environment in OTC with Terraform. based on Ubuntu 20.04 images.
The idea is to have a jump host in the setup phase and for operational purposes that allows spontaneous access to service frontends via ssh proxy jumps without permanently routing them to the public net.
Basic setup works fine so far - I can access the jump host with ssh, and can access the internal machines from there with ssh when I put the private key onto the jump host. So, cloudwise the security seems to be fine. Key pair is generated with ed25519, I use the same key for jump host and internal servers (for now).
What I cannot achieve is the proxy jump as a chained command from my outside machine.
On the jump host, I set AllowTcpForwarding to "yes" in /etc/ssh/sshd_config and restarted ssh and sshd services.
My current local ssh config looks like this:
Host otc
User ubuntu
Hostname <FloatingIP-Address>
Port 22
StrictHostKeyChecking=no
UserKnownHostsFile=/dev/null
IdentityFile= ~/.ssh/ssh_access
ControlPath ~/.ssh/cm-%r#%h:%p
ControlMaster auto
ControlPersist 10m
Host 10.*
User ubuntu
Port 22
IdentityFile=~/.ssh/ssh_access
ProxyJump otc
StrictHostKeyChecking=no
UserKnownHostsFile=/dev/null
I can use this to ssh otc to the jump host.
What I would expect is that I could use e.g. ssh 10.0.0.56 to reach an internal host without further ado. As well I should be able to use commands like ssh -L 8080:10.0.0.56:8080 10.0.0.56 -N to map an internal server's port to a localhost port on my external machine. This is how I managed that successfully on other hosting scenarios in the public cloud.
All I get is:
Stdio forwarding request failed: Session open refused by peer
kex_exchange_identification: Connection closed by remote host
Journal on the Jump host says:
Jul 30 07:19:04 dev-nc-o-bastion sshd[2176]: refused local port forward: originator 127.0.0.1 port 65535, target 10.0.0.56 port 22
What I checked as well:
ufw is off on the Jump Host.
replaced ProxyJump configuration with ProxyCommand
So I am at the end of my knowledge. Has anyone a hint what else could be the reason? Any help welcome!
Ok, cause is found (but not yet fully explained).
My local ssh setting was allowing multiplexed forwards (ControlMaster auto ) which caused the creation of a unix socket file for the Controlpath in ~/.ssh.
I had to login to the jump host to AllowTcpForwarding in the first place.
After rebooting the sshd, I returned to the local machine and the failure occured when trying to forward to the remote internal machine.
After deleting the socket file in ~/.ssh, the connection can now be established as needed. Obviously, the persistent tunnel was not impacted by the restarted daemon on the jump host and simply refused to follow the new directive.
This cost me two days. On the bright side, I learned a lot about ssh :o
I would like to understand the concept of SSH tunneling in detail as I am learning a few things around this topic. I have gone through some details in public forum but still got a few questions.
An SFTP service is running in a remote server and I have been given credentials to connect to it. I am using GUI like WinScp to connect the remote server. What's the role of SSH tunneling here?
Remote SFTP Server admin asked me to generate RSA public key from my machine and its added to the remote server. Now, I can directly connect to the server from SSH terminal without password. What's the role of SSH tunneling here?
Is tunneling implicit or need to be called explicitly for certain circumstances?
Please clarify.
SSH tunneling, SSH console sessions and SFTP sessions are functionally unrelated things.
They can be used simultaneously during single session but usually it is not the case so do not try to find any relation or role of tunneling in ssh/sftp session.
It does not makes sense to mix ssh tunneling with multiple ssh/sftp sessions.
Basically you would use dedicated ssh session for tunneling and extra sessions for console and transfers.
What the heck SSH tunneling is?
Quite often both parties (you and server) reside in different networks where arbitrary network connections between such networks are impossible.
For example server can see on its network workstation nodes and service nodes which are not visible to outside network due to NAT.
The same is valid for the user who initiates connection to the remote server:
so you (ssh client) can see your local resources (worstation nodes and server nodes) but can't see nodes on network of remote server.
Here comes ssh tunneling.
SSH tunnel is NOT a tool to assist ssh related things like remote console ssh sessions and secure file transfers but quite other way around - it is ssh protocol who assists you with building transport to tunnel generic TCP connections the same way TCP proxy works. Once such pipe is built and in action it does not know what is getting transferred via such pipe/tunnel.
Its concept is similar to TCP proxy.
TCP proxy runs on single node so it serves as acceptor of connections and as iniciator of outgoing connections.
In case of SSH tunneling such concept of TCP proxy is split in two halves - one of the nodes (participating in ssh session) performs role of listener(acceptor of connections) and second node performs role of proxy (i.e. initiates outgoing connections).
When you establish the SSH session to the remote server you can configure two types of tunnels which are active while your ssh connection is active.
Multiple ssh clients use notations like
R [IP1 :] PORT1 : IP2 : PORT2
L [IP1 :] PORT1 : IP2 : PORT2
The most confusing/hard part to understand in this ssh tunneling thing are these L and R markers/switches(or whatever).
Those letter L and R can confuse beginners quite a lot because there are actually 6(!!!) parties in this game(each with its own point of view of what is local and what is remote):
you
ssh server
your neighbors who want to expose theirs ports to anyone who sees the server
your neighbors who want to connect to any service server sees
anyone who sees the server and want to connect to any service your
neighbor provides (opposite side/socket of case #3)
any service in a local network of server who wants to be exposed to
your LAN (opposite side/socket of case#4)
In terms of ssh client these tunnel types are:
"R" tunnel (server listens) - YOU expose network services from your LOCAL LAN to remote LAN (you instruct sshd server to start listening ports at remote side and route all incoming connections )
"L" tunnel (you listens) - Server exposes resources of its REMOTE LAN to your LAN (your ssh client starts listening ports on your workstation. your neighbors can access remote server network services by connecting to the ports of your workstation. server makes outgoing connections to local services on behalf of your ssh client)
So SSH tunneling is about providing access to the service which typically is inaccessible due to network restrictions or limitations.
And here is simple conter-intuitive rule to remember while creating tunnels:
to open access to Remote service you use -L switch
and
to open access to Local service you use -R switch
examples of "R" tunnels:
Jack is your coworker(backend developer) and he develops server-side code at his workstation with IP address 10.12.13.14. You are team lead (or sysadmin) who organizes working conditions. You are sitting in the same office with Jack and want to expose his web server to outside world through remote server.
So you connect to ssh server with following command:
ssh me#server1 -g -R 80:ip-address-of-jack-workstation:80
in such case anyone on the Internet can access Jack's current version of website by visiting http://server1/
Suppose there are many IoT Linux devices (like raspberry pi) in the world sitting in multiple home networks and thus not accessible from outside.
They could connect to the home server and expose theirs own port 22 to the server for admin to be able to connect to all those servers.
So RPi devices could connect to the server in a such way:
RPi device #1
ssh rpi1#server -R 10122:localhost:22
RPi device #2
ssh rpi1#server -R 10222:localhost:22
RPi device #3
ssh rpi1#server -R 10322:localhost:22
and sysadmin while being at server could connect to any of them:
ssh localhost -p 10122 # to connecto first device
ssh localhost -p 10222 # to connecto second device
ssh localhost -p 10322 # to connecto third device
admin on remote premises blocked ssh outgoing connections and you want production server to contact bitbucket through your connection...
#TODO: add example
Typical pitfalls in ssh tunneling:
mapping remote service to local priviledged port
ssh me#server -L 123:hidden-smtp-server:25 # fails
#bind fails due to priviledged ports
#we try to use sudo ssh to allow ssh client to bind to local port switches
sudo ssh me#server -L 123:hidden-smtp-server:25 # fails
#this usually results to rejected public keys because ssh looks for the key in /root/.ssh/id_rsa
#so you need to coerce ssh to use your key while running under root account
sudo ssh me#server -i /home/me/.ssh/id_rsa -L 123:hidden-smtp-server:25
exposing some service from local network to anyone through the public server:
typical command would be
ssh me#server -R 8888:my-home-server:80
#quite often noone can't connect to server:8888 because sshd binds to localhost.
#To make in work you need to edit /etc/ssh/sshd_config file to enable GatewayPorts (the line in file needs to be GatewayPorts yes).
my tunnel works great on my computer for me only but I would like my coworkers to access my tunnel as well
typical working command you start with would be
ssh me#server -L 1234:hidden-smtp-server:25
#by default ssh binds to loopback(127.0.0.1) and that is the reason why noone can use such tunnel.
#you need to use switch -g and probably manually specify bind interface:
ssh me#server -g -L 0.0.0.0:1234:hidden-smtp-server:25
Background
I have a machine in production running an elixir application (no access to iex, only to erl) and I am tasked with running an analysis on why we are consuming so much CPU. The idea here would be to launch observer, check the processes tab and see the processes with the most reductions.
How am I connecting?
To connect I am following a tutorial from a blog:
https://sgeos.github.io/elixir/erlang/observer/2016/09/16/elixir_erlang_running_otp_observer_remotely.html 1
Their instructions are as follows:
launch the app in the production machine with a cookie and a name
from local run: ssh user#public_ip "epmd -names" to get the name of the app and the port used
from local create a ssh tunnel to the remote machine: ssh -L 4369:user#public_ip:4369 -L 42877:user#public_ip:42877 user#public_ip (4369 is the epmd port by default, 42877 is the port of the app)
from local connect to the remote machine using the node's name: erl -name "user#app_name" -setcookie "mah_cookie" -hidden -run observer
Problem
And now in theory I should be able to use observer on the machine. Instead however I am greeted with the following error:
Protocol ‘inet_tcp’: register/listen error: epmd_close
So, after scouring the dark side of internet, I decided to use sudo journalctl -f to check all the logs of the machine and I found this:
channel 3: open failed: administratively prohibited: open failed
my_app_name sshd[8917]: error: connect_to flame#99.999.99.999: unknown host (Name or service not known)
/scripts/watchdog.sh")
my_app_name CRON[9985]: pam_unix(cron:session): session closed for user flame
Where:
erlang -name: my_app_name
machine user: flame
machine public ip: 99.999.99.999 (obviously not real)
so it tells me, unknown host ?? I am confused since 99.999.99.999 is the public IP of the machine itself!
Questions
What am I doing wrong?
I read that in older versions of erlang I can’t monitor a machine with observer if they are in different networks (which is the case, because I want to monitor this machine from my localhost) but I didn’t find any information regarding this in modern days.
If this is in fact impossible, what alternatives do I have?
Solution
After 3 days of non-stop searching, I finally found something that works.
To summarize I am putting it here everything I did.
All steps in local machine:
get the ports from the remote server:
> ssh remote-user#remote-ip "epmd -names"
epmd: up and running on port 4369 with data:
name super_duper_app at port 43175
create a ssh tunel with the ports:
ssh remote-user#remote-ip -L4369:localhost:4369 -L43175:localhost:43175
On another terminal in your local machine, run a iex terminal with the cookie the app in your remote server is using. Then connect to it and start observer:
iex --name observer#127.0.0.1 --cookie super_duper_cookie
Node.connect :"super_duper_app#127.0.0.1"
> true
:observer.start
With observer started, select the machine from the Nodes menu.
Possible setbacks
If you have tried this and it didn't work there are a few things you can check for:
Check if the EPMD port on your local machine is free, if not, kill the process using it and free it.
Check your ssh tunneling keys and configurations for permissions. As #Roberto Aloi pointed out this link can be useful: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open
Per the Stackoverflow community's recommendation; I have recently moved my Chef Workstation from a VM to a machine running Ubuntu 16.04 desktop. I have regenerated my .pem file as I'm using the same Hosted Chef user on the new machine.
Knife node list returns the two managed nodes I have setup, and knife search node name:some-name yields a search result for each of the nodes. When I attempt to initiate a Chef Client run on the nodes, I receive an SSH timeout.
The command I am using to initiate the chef client runs is as follows:
knife ssh 'name:managed-node1' 'sudo chef-client' --ssh-user c7-device --identity-file ~/chef/.chef/chucknorris.pem --attribute ipaddress
A bit of information that may be of particular importantance: each one of these managed nodes is a VM with a resolvable hostname. Why after moving my workstation to a new machine, would I now receive an SSH timeout when attempting to initiate a Chef Client run on my managed nodes?
Thanks in advance for any guidance,
As noted above in comments, there is no network connectivity between the hosts in general. This probably means you need to fix your network layout but is outside the scope of knife.
Sorry, should be shot for having to even ask this, but wasted day on this - and feel like I've read everything there is.
I can't create a cluster on my EC2 instances (3) that are spread on three different regions. The hosts:
rabbit#ip-172-31-47-217
rabbit#ip-172-31-1-82
rabbit#ip-172-31-36-111
The initial state before trying to make the cluster:
ubuntu#ip-172-31-47-217:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit#ip-172-31-47-217' ...
[{nodes,[{disc,['rabbit#ip-172-31-47-217']}]},
{running_nodes,['rabbit#ip-172-31-47-217']},
{partitions,[]}]
ubuntu#ip-172-31-36-111:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit#ip-172-31-36-111' ...
[{nodes,[{disc,['rabbit#ip-172-31-36-111']}]},
{running_nodes,['rabbit#ip-172-31-36-111']},
{partitions,[]}]
ubuntu#ip-172-31-1-82:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit#ip-172-31-1-82' ...
[{nodes,[{disc,['rabbit#ip-172-31-1-82']}]},
{running_nodes,['rabbit#ip-172-31-1-82']},
{partitions,[]}]
When I try to check status from one server for another:
sudo rabbitmqctl status -n rabbit#ip-172-31-1-82
Status of node 'rabbit#ip-172-31-1-82' ...
Error: unable to connect to node 'rabbit#ip-172-31-1-82': nodedown
nodes in question: ['rabbit#ip-172-31-1-82']
hosts, their running nodes and ports:
- unable to connect to epmd on ip-172-31-1-82: timeout (timed out)
current node details:
- node name: 'rabbitmqctl3835#ip-172-31-36-111'
- home dir: /var/lib/rabbitmq
- cookie hash: 0tsf/OyQZI7zobmv1Ia97w==
All three servers have the same erlang cookie hash.
I can verify the host names are setup properly:
host ip-172-31-36-111
ip-172-31-36-111.us-west-2.compute.internal has address 172.31.36.111
I know the ports are open:
netstat -plten | grep beam
Because I opened all TCP and UDP at this point as a test, no change.
and finally if this would behave differently given those failures:
sudo rabbitmqctl join_cluster --ram rabbit#ip-172-31-1-82
Clustering node 'rabbit#ip-172-31-47-217' with 'rabbit#ip-172-31-1-82' ...
Error: {cannot_discover_cluster,"The nodes provided are either offline or not running"}
Please help, being driven insane by this.
The problem is that they are in different regions (presumably in EC2-classic - you didn't mention whether you were using a VPC). This means they cannot communicate via their private IPs (see e.g. Can EC2 instances in different regions communicate over their private IP addresses?)
ping 172.31.36.111
will fail from one of the other servers, for example. Pinging using the hostname probably will probably even fail on the DNS lookup.
Your options are:
Put them in separate zones in a single region (in EC2 classic, they will be able to communicate). You could also use a VPC in this case, putting the in separate subnets but allowing interconnections via appropriately set up security groups.
Set up /etc/hosts on each server to point the relevant public IPs of the other servers (you could attach elastic IPs to each server to ensure stability across server restarts). You could also set the hostname of each server for clarity. Set you your security groups to allow access on the relevant ports that rabbitmq uses. There may be security implications of doing this, since the data will be travelling over the public internet.
Set up a VPN between each server in the cluster. Amazon VPC has a VPN facility, but there are ways of setting it up yourself I think.
I think only option 1 is simplest. Option 2 has major security implications (I believe there are ways of securing the connection between the cluster servers, but they aren't documented on the rabbitmq website as far as I can tell). Option 3 is complex but probably the best option if you need multiple regions.
Note that rabbitmq clusters aren't meant to be run over wide geographical areas, since they aren't too reliable in the face of network partitions. See here: https://www.rabbitmq.com/clustering.html