I was able to connect to my VM correctly before. I closed the SSH window and reconnected, it now just loop between "transferring SSH keys to VM" and "connecting to SSH".
This is in the console:
[19864.873131] google_accounts_daemon[720]: Adding user gurvanbk to group google-sudoers
Mar 26 19:07:01 winter-lounge google_accounts_daemon[720]: Adding user gurvanbk to group google-sudoers
Mar 26 19:07:41 winter-lounge google-accounts: INFO Adding user gurvanbk to the Google sudoers group.
It keeps looping that log.
Also here is another post I made on ServerFault.
nmap result:
PS C:\WINDOWS\system32> nmap XX.XX.XXX.XXX
Starting Nmap 7.80 ( https://nmap.org ) at 2020-03-28 12:48 Paris, Madrid
Nmap scan report for 245.151.90.34.bc.googleusercontent.com (34.90.151.245)
Host is up (0.027s latency).
Not shown: 998 filtered ports
PORT STATE SERVICE
22/tcp open ssh
3389/tcp closed ms-wbt-server
Nmap done: 1 IP address (1 host up) scanned in 9.17 seconds
That log of google_accounts_daemon on the serial console is not an ssh or sudoer issue, it looks more like a network one. Following this guide:
So we manage to see this port is open. Now verify again GCP firewall rules (gcloud compute firewall-rules list), if you don't see 22 fw rule you can create it (gcloud compute firewall-rules create default-allow-ssh --allow tcp:22).
Then if all network setup it's ok, try to delete all SSH keys from VM instance metadata and from Compute engine metadata and stop and start your VM Instance, this will force google to create new ssh keys and push them to the vm instance, finally try to connect again.
If this last method don't work, delete again all ssh keys from Compute Engine metadata and VM instance metadata. Then enable OS login feature to try other method of ssh keys injection to your VM instance, once it is activated, stop and start the instance and try to connect again.
I've an f1-micro instance which I've been testing docker on created as such:
$ gcloud compute instances create dockerbox \
--image container-vm-v20140731 \
--image-project google-containers \
--zone europe-west1-b \
--machine-type f1-micro
This all works fine.
I'm now in the process of upgrading to a larger google compute engine VM. I've taken a snapshot of the fi-micro dockerbox, then used this as the Boot Source for the larger n1-standard-8 VM... this seems to create without problems until I try to ssh onto it.
via the command line:
$ gcloud compute --project "secure-electron-631" ssh --zone "europe-west1-b" "me#biggerbox"
ssh: connect to host xx.xx.xx.xx port 22: Connection timed out
ERROR: (gcloud.compute.ssh) Your SSH key has not propagated to your instance yet. Try running this command again.
via the browser, ssh connection I get:
Connection Failed
We are unable to connect to the VM on port 22. Please check that the VM is healthy and the SSH server is running.
I've tried multiple times but same result
I've confirmed it biggerbox is RUNNING. not sure about sshd
OK, problem seemed to stem from not detaching the micro instance from a mounted persistant disk when I took the snapshot. Detached and unmounted the PD volume and snapshotted the micro-instance again and based a new n1-standard-8 on it. Works ok now.
FYI, also handy for those troubleshooting GCE instance ssh:
https://github.com/GoogleCloudPlatform/compute-ssh-diagnostic-sh
I'm having issues using ssh to log in to a VM created from a custom image.
I followed the steps for creating an image from an existing GCE instance.
I have successfully created the image, uploaded it to Google Cloud Storage and added it as an image to my project, yet when I try to connect to the new image, I get a "Connection Refused".
I can see other applications running on other ports for the new image, so it seems to be just ssh that is affected.
The steps I did are below:
...create an image from existing GCE instance (one I can log into fine via ssh)..then:
gcutil --project="river-ex-217" addimage example2 http://storage.googleapis.com/example-image/f41aca6887c339afb0.image.tar.gz
gcutil --project="river-ex-217" addinstance --image=example2 --machinetype=n1-standard-1 anothervm
gcutil --service_version="v1" --project="river-ex-217" ssh --zone="europe-west1-a" "anothervm"
Which outputs:
INFO: Running command line: ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -i /Users/mark1/.ssh/google_compute_engine -A -p 22 mark1#23.251.133.2 --
ssh: connect to host 23.251.133.2 port 22: Connection refused
I've tried deleting the sshKeys metadata as suggested in another SO answer, and reconnecting which did this:
INFO: Updated project with new ssh key. It can take several minutes for the instance to pick up the key.
INFO: Waiting 120 seconds before attempting to connect.
INFO: Running command line: ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -i /Users/mark1/.ssh/google_compute_engine -A -p 22 mark1#23.251.133.2 --
ssh: connect to host 23.251.133.2 port 22: Connection refused
I then try for the first instance in another zone, it works fine with the new key:
gcutil --service_version="v1" --project="river-ex-217" ssh --zone="europe-west1-b" "image1"
Both instances are running on the same "default" network with port 22 running, and ssh works for the first instance the image is created from.
I tried nc command from the other instance and my local machine, it shows no output:
nc 23.251.133.2 22
...whilst the original VM's ip shows this output:
nc 192.157.29.255 22
SSH-2.0-OpenSSH_6.0p1 Debian-4
I've tried remaking the image again and re-adding the instance, no difference.
I've tried logging in to the first instance, and switching user to one on that machine (which should be the same as the second machine?), and ssh from there.
WARNING: You don't have an ssh key for Google Compute Engine. Creating one now...
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
INFO: Updated project with new ssh key. It can take several minutes for the instance to pick up the key.
INFO: Waiting 300 seconds before attempting to connect.
INFO: Running command line: ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no -i /home/mark/.ssh/google_compute_engine -A -p 22 mark#23.251.133.2 -- --zone=europe-west1-a
ssh: connect to host 23.251.133.2 port 22: Connection refused
I'm out of ideas, any help greatly appreciated :) The maddening thiing is I can see the new VM is live with the application ready, I just need to add a few files to it and set up some cronjobs. I guess I could do this pre-image making, but I would like to be able to log in at a later date and modify it, without needing to take 1hr to create images and launch new instances every time.
Yours faithfully,
Mark
This question appears to be about how to debug SSH connectivity problems with images, so here is my answer to that.
It appears that your instance may not be running the SSH server properly. There may be something amiss with the prepared image.
Possibly useful debugging questions to ask yourself:
Did you use gcimagebundle to bundle up the image or did it manually? Consider using the tool to make sure there isn't something you missed.
Did you change anything about the ssh server configuration before bundling the image?
When the instance is booting, check it's console output for ssh messages - it should mention regenerating the keys, starting sshd daemon and listening on port 22. If it does not or complains about something related to ssh, you should follow up on that.
You covered these, but for sake of completeness, these should also be checked:
Can you otherwise reach the VM after it comes up? Does it respond on webserver ports (if any) or respond to ping?
Double check that the network you VM is on allows SSH (port 22) access from the host you are connecting from.
You can compare your ssh setup to that of a working image:
Create a new disk (disk-mine-1) from your image.
Create a new disk (disk-upstream-1) from any working boot image, for example the debian wheezy one.
Attach both of these to a VM you can access (either on console or from cli).
SSH into the VM.
Mount both of the images (sudo mkdir /mnt/{mine,upstream} && sudo mount /dev/sdb1 /mnt/mine && sudo mount /dev/sdc1 /mnt/upstream). Note that whether your image is sdb or sdc depends on the order you attached the images!
Look for differences between the ssh config (diff -waur /mnt/{mine,upstream}/etc/ssh). There should not be any unless you specifically need them.
Also check if your image has proper /mnt/mine/etc/init.d/{ssh,generate-ssh-hostkeys} scripts. They should also be linked from /mnt/mine/etc/rc{S,2}.d (S10generate-ssh-hostkeys and S02ssh respectively).
I've set up the EC2 instance couple days ago and even last night I was able to SSH to it with no problems. Today morning, I can't ssh to it. Port 22 is already open in the security group and I haven't changed anything since last night.
Error:
ssh: connect to host [ip address] port 22: Connection refused
I had similar issue recently and i couldn't figure out why it was happening, so I had to create a new instance, set it up again, and connect and configure all EBS storages to the new one. Took me couple hours... and now it's happening again. In the previous one, I've installed denyhost, which might have blocked me, but in the current one, there are only apache2, and mysql running.
The current instance has been up for 16 hours now, so I don't think it's because it didn't finish booting... Also, port 22 is open to all sources (0.0.0.0/0) and is using tcp protocol.
Any ideas?
Thanks.
With the help of #abhi.gupta200297, we were able to resolve it.
The issue was the error in /etc/fstab, and sshd was supposed to be started after fstab is successful. But it wasn't, hence, the sshd wouldn't start and that's why it was refusing the connection. Solution was to create a temporary instance, mount the root EBS from the original instance, and comment out stuff from the fstab and voila, it's letting me connect again. And for the future, I just stopped using fstab and created bunch of shell commands to mount the EBS volumes to directories and added them in /etc/init.d/ebs-init-mount file and then run update-rc.d ebs-init-mount defaults to initialize the file and I'm no longer having issues with locked ssh.
UPDATE 4/23/2015
Amazon team created a video tutorial of similar issue and show how to debug using this method: https://www.youtube.com/watch?v=_P29ZHu_feU
Looks like sshd might have stopped for some reason. Is the instance EBS backed? if thats the case, try shutting it down and starting it back up. That should solve the problem.
Also, are you able to ssh from AWS web console? They have a java plugin there to ssh into the instance.
For those of you who came across this post because you are unable to SSH into your EC2 instance after a reboot, this is cross-posted to a similar question at serverfault:
From the AWS Developer Forum post on this topic:
Try stopping the broken instance, detaching the EBS volume, and
attaching it as a secondary volume to another instance. Once you've
mounted the broken volume somewhere on the other instance, check the
/etc/sshd_config file (near the bottom). I had a few RHEL instances
where Yum scrogged the sshd_config inserting duplicate lines at the
bottom that caused sshd to fail on startup because of syntax errors.
Once you've fixed it, just unmount the volume, detach, reattach to
your other instance and fire it back up again.
Let's break this down, with links to the AWS documentation:
Stop the broken instance and detach the EBS (root) volume by going into the EC2 Management Console, clicking on "Elastic Block Store" > "Volumes", the right-clicking on the volume associated with the instance you stopped.
Start a new instance in the same region and of the same OS as the broken instance then attach the original EBS root volume as a secondary volume to your new instance. The commands in step 4 below assume you mount the volume to a folder called "data".
Once you've mounted the broken volume somewhere on the other instance,
check the "/etc/sshd_config" file for the duplicate entries by issuing these commands:
cd /etc/ssh
sudo nano sshd_config
ctrl-v a bunch of times to get to the bottom of the file
ctrl-k all the lines at the bottom mentioning "PermitRootLogin without-password" and "UseDNS no"
ctrl-x and Y to save and exit the edited file
#Telegard points out (in his comment) that we've only fixed the symptom. We can fix the cause by commenting out the 3 related lines in the "/etc/rc.local" file. So:
cd /etc
sudo nano rc.local
look for the "PermitRootLogin..." lines and delete them
ctrl-x and Y to save and exit the edited file
Once you've fixed it, just unmount the volume,
detach by going into the EC2 Management Console, clicking on "Elastic Block Store" > "Volumes", the right-clicking on the volume associated with the instance you stopped,
reattach to your other instance and
fire it back up again.
This happened to me on a Red Hat EC2 instance because these two lines were being automatically appended the end of the /etc/ssh/sshd_config file every time I launched my instance:
PermitRootLogin without-passwordUseDNS no
One of these append operations was done without a line break, so the tail of the sshd_config file looked like:
PermitRootLogin without-passwordUseDNS noPermitRootLogin without-passwordUseDNS no
That caused sshd to fail to start on the next launch. I think this was caused by the bug reported here: https://bugzilla.redhat.com/show_bug.cgi?id=956531 The solution was to remove all the duplicate entries at the bottom of the sshd_config file, and add extra line breaks at the end.
Go to your AWS management console > select instance > right click and select "Get System Logs"
This will list what went wrong.
Had the same issue, but sys logs had this:
Starting sshd: /var/empty/sshd must be owned by root and not group or world-writable.
[FAILED]
Used the same steps described above to detach volume and attach to connectable instance. Then used:
sudo chmod 755 /var/empty/sshd
sudo chown root:root /var/empty/sshd
(https://support.microsoft.com/en-us/help/4092816/ssh-fails-because-var-empty-sshd-is-not-owned-by-root-and-is-not-group)
Then detached and reattached to original EC2 Instance and could now access via ssh.
I got similar ssh locked out by detach an EBS but forgot to modify the /etc/fstab
If your ubuntu has systemd, you can edit /lib/systemd/system/local-fs.target and comment out the last two lines:
#OnFailure=emergency.target
#OnFailureJobMode=replace-irreversibly
I haven't tested this extensively and don't know if there are any risks or side effects involved, but so far it works like a charm. It mounts the root volume and all other volumes (except those that are misconfigured, obviously), then it continues the boot process until SSH is up, so you can connect to the instance and fix the incorrect fstab entries.
In my case, the volume was out of space and a service was failing to start. I used the AWS tutorial (from Sherzod's post) to mount it on a good EC2 instance and clean it up and remove the service from startup before remounting it and verifying that things worked.
For me it was that my IP had changed. Hope this helps someone. Navigate to the security groups and update your My IP in the inbound rules.
I had the same issue not able to connect to the aws instance with permission denied error.
I was able to connect with aws team on screen share call and they guided me to change the folder permission on the aws instance using the following user meta script.
steps :
stop the instance
Actions > Instance setting > Edit user meta
Enter the below script and save
**Content-Type: multipart/mixed; boundary="//" MIME-Version: 1.0
--// Content-Type: text/cloud-config; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config cloud_final_modules:
[scripts-user, always]
--// Content-Type:
text/x-shellscript; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash chown root:root /home chmod 755 /home chmod 700 /home/ubuntu chmod 700 /home/ubuntu/.ssh chmod 600 /home/ubuntu/.ssh/authorized_keys ls -ld /home /home/ubuntu /home/ubuntu/.ssh /home/ubuntu/.ssh/authorized_keys chown ubuntu:ubuntu /home/ubuntu -R
--//**
save and connect to the instance with correct pem key.
Resolved my problem
*change ubuntu to your instance username
I have an EC2 instance which i've got a couple of sites hosted on and it's previously been workin really nice.
Today I try to SSH into it and it just refusing connections on port 22 even though the security group has it open and the security group is set to the instance.
Anyone help me out as to why this could be the case?
I just get this line each time - ssh: connect to host 54.247.99.86 port 22: Connection refused
If your instance is EBS based and not an Instance store then take an ami image first as a backup, then stop the instance, boot a new raw instance.
As your old instance is EBS based, detach the volume and attach it to new instance. Once attached mount it to some directory, change the permission /var/empty/sshd, also do cat /etc/fstab to know where was your / partition mounted. Now umount the volume from new instance attach to the old instance with the exact mount point which you had when in fstab like /dev/sda1 for /.
Once attached, start the old instance and check whether you are able to login or not.
To fix the issue, you can follow the steps below to do so:
Launch a new instance in same region (so we can attach the EBS volume to it later).
Stop your current instance and detach the EBS volume
Attach that volume to the new instance at /dev/sdf.
SSH into the new instance and run the following command: sudo mount /dev/sdf /mnt
Run this command: cat /mnt/etc/rc.d/rc.local
If the output contains these lines please remove it from the rc.local file:
cat <<EOL >> /etc/ssh/sshd_config
UseDNS no
PermitRootLogin without-password
EOL
After you modified the rc.local file you can un-mount the EBS volume,detach it from the temporary instance and attach it back to your original instance at /dev/sda1 . And you should be able to SSH into your instance.
This appears to be a temporary fix. This will reoccur if you reboot. I also looking for permanent fix.