I was able to connect to my VM correctly before. I closed the SSH window and reconnected, it now just loop between "transferring SSH keys to VM" and "connecting to SSH".
This is in the console:
[19864.873131] google_accounts_daemon[720]: Adding user gurvanbk to group google-sudoers
Mar 26 19:07:01 winter-lounge google_accounts_daemon[720]: Adding user gurvanbk to group google-sudoers
Mar 26 19:07:41 winter-lounge google-accounts: INFO Adding user gurvanbk to the Google sudoers group.
It keeps looping that log.
Also here is another post I made on ServerFault.
nmap result:
PS C:\WINDOWS\system32> nmap XX.XX.XXX.XXX
Starting Nmap 7.80 ( https://nmap.org ) at 2020-03-28 12:48 Paris, Madrid
Nmap scan report for 245.151.90.34.bc.googleusercontent.com (34.90.151.245)
Host is up (0.027s latency).
Not shown: 998 filtered ports
PORT STATE SERVICE
22/tcp open ssh
3389/tcp closed ms-wbt-server
Nmap done: 1 IP address (1 host up) scanned in 9.17 seconds
That log of google_accounts_daemon on the serial console is not an ssh or sudoer issue, it looks more like a network one. Following this guide:
So we manage to see this port is open. Now verify again GCP firewall rules (gcloud compute firewall-rules list), if you don't see 22 fw rule you can create it (gcloud compute firewall-rules create default-allow-ssh --allow tcp:22).
Then if all network setup it's ok, try to delete all SSH keys from VM instance metadata and from Compute engine metadata and stop and start your VM Instance, this will force google to create new ssh keys and push them to the vm instance, finally try to connect again.
If this last method don't work, delete again all ssh keys from Compute Engine metadata and VM instance metadata. Then enable OS login feature to try other method of ssh keys injection to your VM instance, once it is activated, stop and start the instance and try to connect again.
Related
Currently, I have built a small datacenter environment in OTC with Terraform. based on Ubuntu 20.04 images.
The idea is to have a jump host in the setup phase and for operational purposes that allows spontaneous access to service frontends via ssh proxy jumps without permanently routing them to the public net.
Basic setup works fine so far - I can access the jump host with ssh, and can access the internal machines from there with ssh when I put the private key onto the jump host. So, cloudwise the security seems to be fine. Key pair is generated with ed25519, I use the same key for jump host and internal servers (for now).
What I cannot achieve is the proxy jump as a chained command from my outside machine.
On the jump host, I set AllowTcpForwarding to "yes" in /etc/ssh/sshd_config and restarted ssh and sshd services.
My current local ssh config looks like this:
Host otc
User ubuntu
Hostname <FloatingIP-Address>
Port 22
StrictHostKeyChecking=no
UserKnownHostsFile=/dev/null
IdentityFile= ~/.ssh/ssh_access
ControlPath ~/.ssh/cm-%r#%h:%p
ControlMaster auto
ControlPersist 10m
Host 10.*
User ubuntu
Port 22
IdentityFile=~/.ssh/ssh_access
ProxyJump otc
StrictHostKeyChecking=no
UserKnownHostsFile=/dev/null
I can use this to ssh otc to the jump host.
What I would expect is that I could use e.g. ssh 10.0.0.56 to reach an internal host without further ado. As well I should be able to use commands like ssh -L 8080:10.0.0.56:8080 10.0.0.56 -N to map an internal server's port to a localhost port on my external machine. This is how I managed that successfully on other hosting scenarios in the public cloud.
All I get is:
Stdio forwarding request failed: Session open refused by peer
kex_exchange_identification: Connection closed by remote host
Journal on the Jump host says:
Jul 30 07:19:04 dev-nc-o-bastion sshd[2176]: refused local port forward: originator 127.0.0.1 port 65535, target 10.0.0.56 port 22
What I checked as well:
ufw is off on the Jump Host.
replaced ProxyJump configuration with ProxyCommand
So I am at the end of my knowledge. Has anyone a hint what else could be the reason? Any help welcome!
Ok, cause is found (but not yet fully explained).
My local ssh setting was allowing multiplexed forwards (ControlMaster auto ) which caused the creation of a unix socket file for the Controlpath in ~/.ssh.
I had to login to the jump host to AllowTcpForwarding in the first place.
After rebooting the sshd, I returned to the local machine and the failure occured when trying to forward to the remote internal machine.
After deleting the socket file in ~/.ssh, the connection can now be established as needed. Obviously, the persistent tunnel was not impacted by the restarted daemon on the jump host and simply refused to follow the new directive.
This cost me two days. On the bright side, I learned a lot about ssh :o
I'm trying to ssh into my cisco ISR router. DHCP is working and I can ping the default gateway (the ISR), and can ssh with other devices on the LAN. so I know the LAN connection isn't the problem. I set up a local user that works with console logins so that's not the problem either.
I set up my ssh connection on the router with the following commands:
(config)#line vty 0 21
(config-line)#login local
(config-line)#exec-timeout 3
(config-line)#rotary 1
(config-line)#transport input ssh
(config)#crypto key generate rsa
(config)#ip ssh version 2
(config)#ip ssh port 2222 rotary 1
(config)#ip ssh authentication-retries 3
Then when I nmap the router it has the following ports open:
PORT STATE SERVICE
22/tcp open ssh
However, every time I try to log in to the router I get a Network is unreachable error. This is using the 2222 and 22 port and testing the normal IP and the hostname#ip formats for ssh and absolutely nothing works. I managed to get into telnet with the default settings earlier, but I'm not sure how to get in with SSH.
Thank you all for the help, I know it was very open ended so just let me know anything that could be helpful and I'll provide it.
One logical test step to do, would be switch back to port 22.
Network unreachable usually indicates there's no packet response coming from the host.
It could be because of multiple reasons, but since you've mentioned that ping and telnet went fine. I'd suggest you revert the port config, restart the unit once. See how it goes.
Other possible reasons could be ACL block and/or firewall block on your machine but I think it's unlikely.
I am attempting to connect (via SSH) one GCE VM instance to another GCE VM instance (which will be referred to as Machine 1 and Machine 2 from now one).
So far I have generated (via ssh-keygen -t rsa -f ~/.ssh/ssh_key) a public and private key on Machine 1, and have added the contents of ssh_key.pub to the ~/.ssh/authorized_keys file on Machine 2.
However, whenever I try to connect them via ssh using the following command: gcloud compute ssh --project [PROJECT_ID] --zone [ZONE] [Machine_2_Name] it simply times out (Connection timed out. ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].)
I have doubled checked that each VM instance has plenty of disk space, and their firewall settings are permissive, and OS Login is not enabled. I have read through the answer here but nothing is working.
What am I doing wrong? How do I properly SSH from one GCE VM instance to another?
The problem I was having was that each VM was using a different network/sub-network with different firewall configurations. After making one using the same network/sub-network, I was able to easily ssh into one from the other via
username#machine1:~$ ssh machine2
I tested the same scenario on my side and I got the same result as you said. Then I ran this command inside the machine to debug the SSH process to try to narrow down the issue:
gcloud compute ssh YOUR_INSTANCE_NAME --zone ZONE --ssh-flag="-vvv"
Then I got this result:
debug1: connect to address 35.x.x.x port 22: Connection timed out
ssh: connect to host 35.x.x.x port 22: Connection timed out
So, means the instance 1 is unable to connect to the external IP address of instance 2. I only added a new firewall rule and it works.
After running above mentioned command, if you see any permission denied message, it means you did not copy the public key to the source machine properly.
Background
I have a machine in production running an elixir application (no access to iex, only to erl) and I am tasked with running an analysis on why we are consuming so much CPU. The idea here would be to launch observer, check the processes tab and see the processes with the most reductions.
How am I connecting?
To connect I am following a tutorial from a blog:
https://sgeos.github.io/elixir/erlang/observer/2016/09/16/elixir_erlang_running_otp_observer_remotely.html 1
Their instructions are as follows:
launch the app in the production machine with a cookie and a name
from local run: ssh user#public_ip "epmd -names" to get the name of the app and the port used
from local create a ssh tunnel to the remote machine: ssh -L 4369:user#public_ip:4369 -L 42877:user#public_ip:42877 user#public_ip (4369 is the epmd port by default, 42877 is the port of the app)
from local connect to the remote machine using the node's name: erl -name "user#app_name" -setcookie "mah_cookie" -hidden -run observer
Problem
And now in theory I should be able to use observer on the machine. Instead however I am greeted with the following error:
Protocol ‘inet_tcp’: register/listen error: epmd_close
So, after scouring the dark side of internet, I decided to use sudo journalctl -f to check all the logs of the machine and I found this:
channel 3: open failed: administratively prohibited: open failed
my_app_name sshd[8917]: error: connect_to flame#99.999.99.999: unknown host (Name or service not known)
/scripts/watchdog.sh")
my_app_name CRON[9985]: pam_unix(cron:session): session closed for user flame
Where:
erlang -name: my_app_name
machine user: flame
machine public ip: 99.999.99.999 (obviously not real)
so it tells me, unknown host ?? I am confused since 99.999.99.999 is the public IP of the machine itself!
Questions
What am I doing wrong?
I read that in older versions of erlang I can’t monitor a machine with observer if they are in different networks (which is the case, because I want to monitor this machine from my localhost) but I didn’t find any information regarding this in modern days.
If this is in fact impossible, what alternatives do I have?
Solution
After 3 days of non-stop searching, I finally found something that works.
To summarize I am putting it here everything I did.
All steps in local machine:
get the ports from the remote server:
> ssh remote-user#remote-ip "epmd -names"
epmd: up and running on port 4369 with data:
name super_duper_app at port 43175
create a ssh tunel with the ports:
ssh remote-user#remote-ip -L4369:localhost:4369 -L43175:localhost:43175
On another terminal in your local machine, run a iex terminal with the cookie the app in your remote server is using. Then connect to it and start observer:
iex --name observer#127.0.0.1 --cookie super_duper_cookie
Node.connect :"super_duper_app#127.0.0.1"
> true
:observer.start
With observer started, select the machine from the Nodes menu.
Possible setbacks
If you have tried this and it didn't work there are a few things you can check for:
Check if the EPMD port on your local machine is free, if not, kill the process using it and free it.
Check your ssh tunneling keys and configurations for permissions. As #Roberto Aloi pointed out this link can be useful: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open
I'm having problem accessing one of my VMs (called myvm1 here) after having restored a disk from a snapshot. Here is what I did yesterday (which worked just fine):
I made a snapshot of disk1.
I created a new disk, called disk2, using the snapshot created above.
I attached the disk to myvm1 though Google Console.
I unmounted disk1 and mounted disk2.
I deleted disk1.
Everything worked fine, and database data on disk2 was accessible as desired. There's not much else on that disk.
Today, what I wanted to do, was to "rename" disk2 to disk1 (to avoid future problems with our Terraform setups). I did this by doing the following:
I made a snapshot of disk2.
I created a new disk, calling it disk1, using the snapshot above.
I attached the disk to myvm1 using the terminal:
gcloud compute --project=myproject instances attach-disk myvm1 --disk disk1
After this, when I attempted to ssh into myvm1 (to unmount and mount), I get a
ssh: connect to host myvm1 port 22: Connection refused
I have attempted the following to solve this/investigate:
stopping and starting the VM (takes a considerably longer time than other VMs in the same project) repeatedly
detaching disk1 (and re-attaching it)
Other information:
other VMs in the same project are still accessible via ssh.
I did nothing else to the VM yesterday or today but what I have written above. The system has not been in use between yesterday and now (it was shut down over night to save money).
Using the Google Console SSH does not work, BUT it does not work for the other VMs either, as we connect using private keys.
"The instance is booting up and sshd is not yet running." - It's listed as RUNNING.
"The instance is not running sshd." I have not manually disabled sshd.
"sshd is listening on a port other than the one you are connecting to." I've made no changes to ports.
"There is no firewall rule allowing SSH access on the port." Also, under "Firewall rules and routes details" port 22 is enabled. Also, firewall rules are identical to the other VMs in the same project.
"The firewall rule allowing SSH access is enabled, but is not configured to allow connections from GCP Console services." We don't want to be able to connect via GCP Console so that doesn't matter.
"The instance is shut down." - It's running.
Debug information for the ssh-call:
me#mycomputer:~/project$ ssh myvm1 -vvv
OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /home/me/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "myvm1" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to myvm1 [10.23.0.3] port 22.
debug1: connect to address 10.23.0.3 port 22: Connection refused
ssh: connect to host myvm1 port 22: Connection refused
I've looked at the solution mentioned here
Why Google Cloud Compute Engine instance gives ssh connection refused after restart?
but since I have not yet mounted/unmounted any of the disks I don't see how that could be the same problem.
I would very much appreciate any help you can give me. Solutions involving creating a new instance are not relevant, as I want to know what went wrong in the first place, so that this does not happen in a production environment. Thankfully myvm1 is just a sandbox system.
A port 22 error can come from two sources: firewall not properly set up on GCP or port 22 not accepting SSH connections from within your instance. Assuming that firewall is properly set up since it works on other instances, please try to log in with serial console and check your iptable.
In order to connect to serial console you will have to perform the following:
1). Activate the “Connect to serial console” button.
Go to VM instances, click on your VM, Edit, and active “enable connecting to serial ports” in the Remote access area and click on save.
2). Create a username and password.
Go to Vm instance, click on your Vm again, Edit, and fill up the custom metadata section with:
In key: startup-script
In value:
#!/bin/bash
sudo useradd -G sudo pamela
sudo echo 'pamela:pamela5' | chpasswd
(This is a script that creates a username : pamela and password: pamela5, which you are going to use later. Please use something else for security purposes)
3). A reboot is needed for changes to take effect.
I had the same problem. I think the snapshot file is corrupted.