SSH using terminal, cloud shell, or window browser in to Google Cloud VM instance not working - ssh

Question

I am new to google cloud computing. Recently have not been able to SSH into my VM instance, previously it worked by clicking the SSH button and opened a new browser window. I have tried several fixes based on internet searches, mostly summarized below:
When I click the SSH button, it says "Transferring Keys to the VM" but never logs in and eventually times out.
Attempt 1: I attempted to follow another thread Google VM Instance not opening with SSH and I increased the persistent disk size and reset the VM. Now it says "Could not connect, retrying..." when I click the SSH button and eventually times out.
Attempt 2: I also tried SSHing into the VM using Google Cloud Shell using the command
gcloud beta compute ssh --zone "myZONE" "VM_instance_name" --project "PROJECT_ID"
And got the following error
Permission denied (publickey).
ERROR: (gcloud.beta.compute.ssh) [/usr/bin/ssh] exited with return code [255].
I created new keys using
sudo gcloud compute config-ssh
and it gave me this error
ERROR: (gcloud.compute.config-ssh) The project property is set to the empty string, which is invalid.
I ran the following and it seems to work
gcloud config set project myPROJECT_ID
Updated property [core/project].
But when I ran sudo gcloud again it gave me the same empty string error.
Attempt 3: I also tried setting up the log in locally using google-cloud-sdk. I followed interactive installing instructions on Using the Google Cloud SDK installer. I tried obtaining ssh key using
gcloud compute project-info describe --project myPROJECT_ID
and copied in into the ssh key on for the VM thru the cloud console website. I also tried
sudo gcloud compute config-ssh
which seemed to work and gave the following
Updating project ssh metadata...⠼Updated [https://...].
Updating project ssh metadata...done.
You should now be able to use ssh/scp with your instances.
For example, try running:
$ ssh VMinstance.myzone.PROJECT_ID
When I try running the ssh command I get the following error.
ssh: Could not resolve hostname VMinstance.myzone.PROJECT_ID: nodename nor servname provided, or not known
My instance has "enable connecting to serial ports" activated.
Any assistance is greatly appreciated.
Thank you in advance.
Update: for the VM instance in question, I clicked on serial port 1 (console) and here are the first 100 lines
serialport: Connected to PROJECT_ID.Zone.VMInstance port 1 (session ID: ##..., active connections: 1).
Jul 2 12:19:02 INSTANCE google-accounts: INFO Removing user root.
Jul 2 12:19:02 INSTANCE google-accounts: INFO Removing user root from the Google sudoers group.
[1##.##8] google_accounts_daemon[822]: Removing user root from group google-sudoers
Jul 2 12:19:02 INSTANCE google_accounts_daemon[822]: Removing user root from group google-sudoers
[1##.##0] google_accounts_daemon[822]: gpasswd: /etc/group.####: No space left on device
Jul 2 12:19:02 INSTANCE google_accounts_daemon[822]: gpasswd: /etc/group.####: No space left on device
[1##.##6] google_accounts_daemon[822]: gpasswd: cannot lock /etc/group; try again later.
Jul 2 12:19:02 INSTANCE google_accounts_daemon[822]: gpasswd: cannot lock /etc/group; try again later.
Jul 2 12:19:02 INSTANCE google-accounts: WARNING Could not update user root. Command '['gpasswd', '-d', 'root', 'google-sudoers']' returned non-zero exit status 1..
Jul 2 12:19:02 INSTANCE google-accounts: ERROR Exception calling the response handler. [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/'].#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/google_compute_engine/metadata_watcher.py", line 200, in WatchMetadata#012 handler(response)#012 File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_daemon.py", line 285, in HandleAccounts#012 self.utils.SetConfiguredUsers(desired_users.keys())#012 File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_utils.py", line 318, in SetConfiguredUsers#012 mode='w', prefix=prefix, delete=True) as updated_users:#012 File "/usr/lib/python3.6/tempfile.py", line 681, in NamedTemporaryFile#012 prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)#012 File "/usr/lib/python3.6/tempfile.py", line 269, in _sanitize_params#012 dir = gettempdir()#012 File "/usr/lib/python3.6/tempfile.py", line 437, in gettempdir#012 tempdir = _get_default_tempdir()#012 File "/usr/lib/python3.6/tempfile.py", line 372, in _get_default_tempdir#012 dirlist)#012FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']
Jul 2 12:19:41 INSTANCE systemd[1]: snapd.service: Start operation timed out. Terminating.
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/ssh/sedYpckqQ: No space left on device
[1##.##1] google_accounts_daemon[822]: sed: couldn't flush /etc/ssh/sedYpckqQ: No space left on device
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedeXXx0O: No space left on device
[1##.##4] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedeXXx0O: No space left on device
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sed27Z7HO: No space left on device
[1##.##2] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sed27Z7HO: No space left on device
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sed3bBBFO: No space left on device
[1##.##6] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sed3bBBFO: No space left on device
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedUKduxP: No space left on device
[1##.##3] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedUKduxP: No space left on device
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedhtixlP: No space left on device
[1##.##0] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedhtixlP: No space left on device
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedy6zVAS: No space left on device
[1##.##9] google_accounts_daemon[822]: sed: couldn't flush /etc/pam.d/sedy6zVAS: No space left on device
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: Restarting optional services.
[1##.##7] google_accounts_daemon[822]: Restarting optional services.
Jul 2 12:20:03 INSTANCE systemd[1]: Stopping Login Service...
Jul 2 12:20:03 INSTANCE systemd[1]: Stopped Login Service.
Jul 2 12:20:03 INSTANCE systemd[1]: Starting Login Service...
Jul 2 12:20:03 INSTANCE systemd[1]: Started Login Service.
Jul 2 12:20:03 INSTANCE systemd[1]: Stopping Regular background program processing daemon...
Jul 2 12:20:03 INSTANCE systemd[1]: Stopped Regular background program processing daemon.
Jul 2 12:20:03 INSTANCE systemd[1]: Started Regular background program processing daemon.
Jul 2 12:20:03 INSTANCE google_accounts_daemon[822]: Restarting SSHD
[1##.##5] google_accounts_daemon[822]: Restarting SSHD
Jul 2 12:20:03 INSTANCE systemd[1]: Stopping OpenBSD Secure Shell server...
Jul 2 12:20:03 INSTANCE systemd[1]: Stopped OpenBSD Secure Shell server.
Jul 2 12:20:03 INSTANCE systemd[1]: Starting Google Compute Engine Instance Setup...
Jul 2 12:20:03 INSTANCE instance-setup: WARNING [Errno 28] No space left on device
Jul 2 12:20:03 INSTANCE instance-setup: INFO Running google_set_multiqueue.
Jul 2 12:20:03 INSTANCE instance-setup: INFO Setting /proc/irq/31/smp_affinity_list to 0 for device virtio1.
Jul 2 12:20:03 INSTANCE instance-setup: INFO /proc/irq/31/smp_affinity_list: real affinity 0
Jul 2 12:20:03 INSTANCE instance-setup: INFO Setting /proc/irq/32/smp_affinity_list to 0 for device virtio1.
Jul 2 12:20:03 INSTANCE instance-setup: INFO /proc/irq/32/smp_affinity_list: real affinity 0
Jul 2 12:20:03 INSTANCE instance-setup: INFO /usr/bin/google_set_multiqueue: line 139: echo: write error: No such file
or directory
Jul 2 12:20:03 INSTANCE instance-setup: INFO cat: /sys/class/net/ens4/queues/tx-0/xps_cpus: No such file or directory
Jul 2 12:20:03 INSTANCE instance-setup: INFO Queue 0 XPS=/sys/class/net/ens4/queues/tx-0/xps_cpus for
Jul 2 12:20:03 INSTANCE instance-setup: WARNING [Errno 28] No space left on device
Jul 2 12:20:03 INSTANCE systemd[1]: Started Google Compute Engine Instance Setup.
Jul 2 12:20:03 INSTANCE systemd[1]: Starting OpenBSD Secure Shell server...
Jul 2 12:20:03 INSTANCE systemd[1]: Started OpenBSD Secure Shell server.
Jul 2 12:20:03 INSTANCE systemd[1]: Stopping OpenBSD Secure Shell server...
Jul 2 12:20:03 INSTANCE systemd[1]: Stopped OpenBSD Secure Shell server.
Jul 2 12:20:03 INSTANCE systemd[1]: Starting Google Compute Engine Instance Setup...
Jul 2 12:20:04 INSTANCE instance-setup: WARNING [Errno 28] No space left on device
Jul 2 12:20:04 INSTANCE instance-setup: INFO Running google_set_multiqueue.
Jul 2 12:20:04 INSTANCE instance-setup: INFO Setting /proc/irq/31/smp_affinity_list to 0 for device virtio1.
Jul 2 12:20:04 INSTANCE instance-setup: INFO /proc/irq/31/smp_affinity_list: real affinity 0
Jul 2 12:20:04 INSTANCE instance-setup: INFO Setting /proc/irq/32/smp_affinity_list to 0 for device virtio1.
Jul 2 12:20:04 INSTANCE instance-setup: INFO /proc/irq/32/smp_affinity_list: real affinity 0
Jul 2 12:20:04 INSTANCE instance-setup: INFO /usr/bin/google_set_multiqueue: line 139: echo: write error: No such file or directory

Answer 1

If your instance is running and you can restart it you may try logging in via serial console. This method is independed from any firewall / network settings :)
Just add a user with a password; create a startup script like this (it will create a user and add it to google-sudoers group so you can do everything with this account):
#! /bin/bash
adduser username
echo 'sudouser:userspass' | chpasswd
usermod -aG google-sudoers sudouser
And then connect to serial console you can log in. Either using console window or cloud shell: gcloud compute connect-to-serial-port instance-name
This will allow you to log in. Unless there's something really wront with your VM.
But - judging from you desription - you want regular SSH access.
For that make sure that:
you firewall rules don't block port 22 on this machine
your firewall on a VM in question allows traffic on port 22
your ethernet interface is up in the VM
SSH server listens (ps aux | grep sshd) on port 22
Then check if your VM has external IP - if not you will be only able to connect to your VM via another that has one, or via cloud shell (and of course serial console as described above).
However - since you got this Permission denied (publickey) message that means that for some reason your SSH keys don't work (corrupted, deleted etc). When you log in with serial console check if they're there. If not you can add them manually.
----------- UPDATE ------------
Look at this line: Jul 2 12:19:02 INSTANCE google_accounts_daemon[822]: gpasswd: /etc/group.####: No space left on device - it says that your VM ran out of disk space. Stop your VM, try resizing your VM's disk first with gcloud compute disks resize mydiskname --size=100GB --zone=us-central1-a and start it. After the start VM's partition will be resized. –