Not able to login via SSH to GCP Compute Engine VM instance - ssh

My GCP VM instance was setup sometime back and all the services were working fine. However, suddenly from today I am not able to login via SSH and even an instance restart did not fix the issue. When checked the serial log, it seems to have stopped in this line,
[ 8.686861] intel_powerclamp: No package C-state
availableintel_powerclamp: No package C-state available[ 8.705723]
attempt to access beyond end of device [ 8.706721] sda1: rw=0,
want=41927608, limit=41927602 [[32m OK [0m] Started Security
Auditing Service.
Starting Update UTMP about System Boot/Shutdown...
32m OK [0m] Started Update UTMP about System Boot/Shutdown.
Starting Update UTMP about System Runlevel Changes...
32m OK [0m] Started Update UTMP about System Runlevel
ChanWelcome to emergency mode! AfterGive root password for maintenance
(or type Control-D to continue):
It won't proceed further to start SSH and other services, hence I am not able to login to the instance. Any help would be greatly appreciated.
Thanks in advance.

The error
No package C-state available
points to problems to mount the filesystem.
Did you try to follow the steps written in this other post ?
When you see Ctrl+D enter root password. Then enter mount -a to see in what line is the error.
Enter vi /etc/fstab and correct whatever is missing, save and reboot.

What you can do is logon to the serial console so you can interact with it. It will probably ask you for a root password and if you have not set that up, you can set that up through startup scripts.

Related

Cannot ssh into Google-Engine, connecting in a loop

I am unable to connect through SSH to my GCE instance. I was connecting without any problem, the only think I changes was my user name through top right corner of the browser then selected Change Linux Username.
When I try to ssh into my google engine via browser, I keep having following message in a endless loop:
When I try to ssh via cloud shell I also get following error message, (serial console output):
Permission denied (publickey).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
[Q] Is there any way to fix this problem? Since I have no access to the engine now, I don't know what to do.
However you could always get back access through serial console then from there you could internally y troubleshoot user/ssh issue.
1) $ gcloud compute instances add-metadata [INSTANCE_NAME] --metadata=serial-port-enable=1
You can then connect to the instance through the serial port
NOTE:The root password have must been already set in order to use the serial port
2)
$ gcloud compute connect-to-serial-port [INSTANCE_NAME]
If you never set the root password you could set it by adding a startup-script to your instance that will set a password as root by running the below command :
NOTE: the instance must be rebooted in order to run the startup script.
3) $ gcloud compute instances add-metadata [instance name] --metadata startup-script='echo "root:YourPasswdHere" | chpasswd'
Reboot the instance run the command on the step "2)" authenticate your self as root with the password that you set on the startup script in the step "3)" .
I had the same problem, It took me several days to figure out what was happening in my case.
To find out, I created a new instance from scratch and started making all modifications I've done to those that eventually couldn't connect to, one by one, exiting the ssh connection and re entering so as to test it.
I've tried it a couple of times, in both cases, the connection was impossible after uninstalling python (I only needed 3.7 version so I was uninstalling all others and installing that one I needed).
My command for uninstalling it was
sudo apt purge python2.7-minimal
and
sudo apt purge python3.5-minimal
I don't know if it was specifically because of deleting python, or because of using purge (in which case this problem might reproduce if using purge with another program).
I don't even know why would this affect ssh connection.
Could it be that google cloud is somehow using destination python for the ssh web?
In any case, if you are experiencing this problem try to avoid uninstalling anything from the base VM.

New raspberry pi 2 B not able to boot into GUI

I just purchased new raspberry pi and tried to install raspbian using Noobs.
So the installation goes successfully and I boot into raspbian GUI successfully and make changes to raspi-config like to ask for the password when booting into GUI and enable various option like SSH,SCI and more. Then it reboots and gives me following error:
Raspberry pi Fatal server error: could not create lock file in /inp/.1xd-lock Please consult the X.org Foundation support at http://wiki.x.org for help
init: giving up
init: unable to connect to X server: Connection not sucessfull
init: server error nuth: error in locking authority file /home/pi/.xauthority
I have done this process so many time it is giving me same error every time. Any suggestion.......
Check your .xauthority file to make sure it's owned by the user you're logging in as rather than root. If it's owned by root, you can either change the ownership, or simply delete it and it will be recreated next time you attempt a login.
This has happened to me before!
It is likely your .xauthority file is corrupted, or root owns that file which means it won't let you boot into the GUI. Try this code and it seemed to work for me...
sudo chown -R pi:pi /home/pi
Basically what this is doing is it is reverting all the files' permission settings back to you, so the .Xauthority file for your account will be yours again.
BUT...
If the actual file itself is corrupt, then do this:
xauth -b
.Xauthority is not needed on startup, so it is safe to just delete it off your SD card, since a new version will be generated when you boot into the GUI.

ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]

I kept getting kicked out of my compute engine instance after a few seconds of idle with the indicated error (255).
I used 'gcloud compute ssh' to log in.
I am using the default firewall setting, which I believe would be good enough for ssh.
But if I am missing something, please so indicate and suggest the fix for this error.
Basically I can't get any efficient work done at this point having to ssh in so many times.
gcloud denies an ssh connection if there was a change in the setup, e.g.
after you changed your default zone or region, or you created another instance.
Then, you must update the ssh keys in your metadata by
sudo gcloud compute config-ssh
If this complains about different entries in your config file where your ssh key entries are stored, ~/.ssh/config, delete this file and execute the above command again.
If you have installed gcloud without sudo, you can omit sudo.
255 is the interactive ssh exit code for ssh failure - otherwise interactive ssh exits with the exit code of the last command executed in the ssh session.
The next time you get exit code 255 from ssh try running with --ssh-flag="-vvv" (more v's => more debugging output) and see if it helps track down connection problems.
For those who stop by this page. This helped me to solve the problem.
Try to the following:
Go to your Google and remove the SSH key for the server
Go to your google cloud console -> compute engine -> Metadata -> "SSH
keys" tab and click on edit. Here you can delete the ssh keys.
Run the gcloud command again
Click on the "Instances" link on the left side of your google cloud account, which will list down all the instances on the right side. Under
connect column, you will see "SSH" drop-down, click on "View cloud
Command" and this will bring a new dialog. Copy that command and run on your PC's terminal. This will let you SSH into the google compute engine.
It seems a feature/issue from Google Cloud Platform itself, we are going to continue checking it.
If the default network was edited, or if not using the default network, you may need to explicitly enable ssh access by adding a firewall-rule:
$ gcloud compute firewall-rules create --network=YOUR_NETWORK \
default-allow-ssh --allow tcp:22
After that, retry the 'gcloud compute ssh' command.
This is a real problem with very little documentation to dealing with it.
Sometime after creating the instance using the gcloud sdk ssh snippet provided via GCP console stopped working and continually errors with 255 making connecting to ssh on the instance only available through browser via GCP console for the compute instance in question. Not to mention this has happened to me on many different instances some without touching the default account permissions after initial setup and deployment which is overly frustrating. Cause for no reason it just stops working...works, then doesn't...
The only thing that worked for me was creating a new user to connect with through gcloud sdk! Be it Windows/PowerShell or Linux locally, using the following snippet:
gcloud compute ssh newuser-name#instance-name
That all per GCP documentation here: https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-ssh
Everything else passed per suggestions in documentation - port 22 open with access meaning it has to be a a problem with the default users authorization_keys WHICH they provide absolutely no documentation on how to fix that - at least nothing I could find on fixing (not creating or deleting)
I've tried updating the account, tried deleting the user and credentials from the instance, nothing appears to work. using:
gcloud compute --project "project-name" ssh --zone "us-east4-a" "instance-name"
Just doesn't work...
- even tried 'gcloud compute config-ssh --force-key-file-overwrite' NOTHING WORKS...
But creating a new user works every time, and once the user is created you can keep using that user via gcloud sdk
It's a work around, and I hate work around's for things like this but for my sanity this works at least until I can figure out how to reset the default account permissions, so if anyone has any ideas there or can point me in a direction for that I'd more than appreciate it!
IT was my mistake stating that the default firewall would allow all connections into an instance. The contrary turned out to be true. Please refer to an appropriate firewall rule must be set up to allow connection into an instance
Anh-
If you have Identity-Aware Proxy (IAP) enabled for your setup, try adding the --tunnel-through-iap option to the gcloud compute ssh command.
$ gcloud compute ssh --zone <zone> --project <project> --tunnel-through-iap <instance-name>
More information for people landing on this page, if you're using preemptible instances to save some compute costs, that could also be the reason for getting kicked out like this. Your instance may have just randomly stopped.
In my case, the I had created a bootable disk for the VM without adding the information of what source-image it needs to have. Because of this, even though the instance was coming up alright and ssh-allow rule was there, the VM was not booting up.
Finally added the source image to the disk and I was able to ssh into the VM.
Hope this helps for someone.
I had the same error . i restarted the VM instance and ssh workis fine
I had the problem where after clicking on the SSH button it would keep trying to establish a connection and fail. After long struggle I resolved it by adding Service Account User role to myself. If your account was created after the VM instance was created, it might result in this situation.
I know this was opened a long time ago, but for a more recent update on this topic. I had the same trouble connecting via ssh. It was giving the error code 225. Obviously there was a connectivity issue. There was already a firewall rule set under VPC network-> Firewall to allow ssh. However, to fix this problem I had to go to the specific network and create a rule under the network Firewall Rules. VPC network details -> FIREWALL RULES and create an inbound TCP rule for port 22.
if you are having a problem trying to access you g-cloud VM instance from your computer terminal remotely, and are getting the error code 255,the problem is that the ssh protocols in your computer are wrong or not updated.
In this case the best way to fix it is to go to your home directory (in your computer) check the hidden files and find the folder ".ssh" .Just delete this folder and re-open your bash terminal. Then run again your gcloud vm command.
Example:
you#your_computer:~$ gcloud beta compute ssh --zone "us-central1-a" "your_VM_name" --project "your_project_name"
You should this time instead of getting the error 255 code, the messages below:
WARNING: The private SSH key file for gcloud does not exist.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
This tool needs to create the directory [/home/your_name/.ssh] before being able to
generate SSH keys.
Do you want to continue (Y/n)?
Type "Y" and gcloud will setup the new protocols by creating a brand new updated .ssh file.
After that you should be able to access your VM with your gcloud command without any problem.
That should solve the problem
Cheers
https://blackpearlmatrix.com
had the exact same symptoms - in my case the reason appeared to be the following. I was using root user + ssh key whereas root login is by default disabled in /etc/ssh/sshd_config (PermitRootLogin property).
I eventually had to delete my instance and make a new one with the same disk. See https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-ssh#use_your_disk_on_a_new_instance for details.
For me, my other teammates were able to login into the machine, but not me. So I asked them to create a user of my name with sudo rights, logged into serial console and changed passwordAuthentication to yes followed by sudo service ssh restart (for few this could be sudo service sshd restart.)
Post this I was able to login with
ssh -o PreferredAuthentications=password username#publicIP -p 22
This trick worked fine for me.
Reinitializing the gcloud with "gcloud init" and generating new ssh keys resolved the problem for me.
I had same issue.
I had connected the serial control and had checked logs. and there was some error log like "there is no disk space". Then I had resized disk as written in this document.
Now I am able to connect to instance with ssh.
Try switching to a different Internet connection
So, I was getting the same error but in my case I was not able to log in to the instance at all.
(base) girish#girish:~$ gcloud beta compute ssh --zone "asia-east1-b" "fp-1" --project "fp-public"
ssh: connect to host 12.345.678.90 port 22: Resource temporarily unavailable
ERROR: (gcloud.beta.compute.ssh) [/usr/bin/ssh] exited with return code [255].
(base) girish#girish:~$ gcloud beta compute ssh --ssh-flag='-vvv' --zone "asia-east1-b" "fp-1" --project "fp-public"
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "12.345.678.90" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 12.345.678.90 [12.345.678.90] port 22.
[debug1: connect to address 12.345.678.90 port 22: Resource temporarily unavailable
ssh: connect to host 12.345.678.903 port 22: Resource temporarily unavailable
ERROR: (gcloud.beta.compute.ssh) [/usr/bin/ssh] exited with return code [255].
What worked for me:
I tried reinstalling lots of things and re-initializing various config and then landed on a thread which suggest to change the Internet network you are using and it worked!!
It's possible you have a rule that only allows whiltelisted IPs to ssh into a gcloud VM. So you may have forgotten to enable your work VPN or out of your work's office IP.
Try restarting your computer.
I got the same error and tried gcloud config ssh as mentioned previously to no avail. I then checked that the IDs and roles of serviceaccount and developer had 'editor' permissions, and that was fine. I started a new instance and logged out of all of my other google accounts and it still threw the error. Then, I restarted my computer and did not log back into my other google accounts. That fixed it.
When using IAP, GCP stores the key in instance metadata and then propagate
that to the ~/.ssh/authorized_keys file.
You might get the error OP talks about when you remove the key from the ~/.ssh/authorized_keys file and it's still in the instance metadata. Reason being:
GCP check that the user, key combo that you are using to ssh is already in the instance metadata.
It assumes that the exists in the ~/.ssh/authorized_keys file for that user and doesn't propagate the key.
As the key doesn't exist in ~/.ssh/authorized_keys file for whatever reason (you deleted it, someone else deleted it etc. etc.) - you get access denied.
If this is the case with you, then fix is simple: remove the instance metadata entry for that user, key combo (have attached an image for ref, just click X and remove your faulty key) and try ssh again
What worked for me was turning my firewall on. (On a Mac, ssh'ing into a gcp instance).
In another instance of the error, my connection worked fine when I was on ethernet, but not when I was on wifi. Switching back to ethernet allowed me to connect again.
In my case sorted out the issue after restarting the VM.
if you are able to access the VM previously and suddenly giving SSH issues, give it a try by restarting.
Permission wise check whether you have IAP-secured Tunnel User
gcloud compute ssh --zone "your_zone" "instance_name" --tunnel-through-iap --project "project_name"
If this not works check with the GCP built-in SSH client, and click open in browser window.
Hope this help !!!

Access Denied when executing through cygwin openssh

When I execute the command "iisreset" through an ssh terminal on a remote windows machine, I get the following error:
Attempting stop...
Restart attempt failed.
Access denied, you must be an administrator of the remote computer to use this
command. Either have your account added to the administrator local group of
the remote computer or to the domain administrator global group.
When I type whoami, it shows that I am the administrator. My cygwin ssh session is running as the "cyg_server" user who has admin privileges.
My ssh server is configured with privilege separation and allows me to login as administrator.
When I run the command locally, it works fine. The problem is execution through ssh.
I've also used process monitor to see what's going on, but it does not indicate the problem.
That is pretty strange because I am able to do admin-only operations in remote ssh such as:
echo "hi">/cygdrive/c/x.txt
rm /cygdrive/c/x.txt
Turning off UAC did not make a difference.
Any ideas?
I had a similar problem: unable to start/stop services using net start/net stop from a remote password-less (public/private key) SSH user. Attempting to start/stop the service was resulting in a "System Error 5 has occurred. Access is denied." error).
I had to install Cygwin's LSA authentication package (see http://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-setuid-overview) in order for (I presume) setuid to work properly for password-less logins.
The problem should go away once LSA is installed on the Cygwin/SSH host and the machine has been rebooted.
I got scared of the LSA package mentioned in #user3609241's answer because of this sentence in the LSA docs:
as soon as the LSA encounters serious problems (for instance, one of
the protected LSA processes died), it triggers a system reboot.
But, those same docs point to a very easy way to "runas" SYSTEM - just use the at command:
$ date
Mon, Jan 12, 2015 8:17:35 PM
$ at 20:18 iisreset
Added a new job with job ID = 1
$ at
Status ID Day Time Command Line
-------------------------------------------------------------------------------
1 Today 8:18 PM iisreset
It works, at the cost of having to wait up to 59 seconds.
(wrapping the above sequence of commands in a simple-to-call script is left as an exercise to the reader; our management util is written in Perl so it was pretty straightforward).
Run the Cygwin terminal as administrator

What happens to a process in an EC2 instance when I get a 'Broken Pipe' error on ssh?

I am using some EC2 instances to run some large jobs I can not run locally. The issue I am seeing is that after a while (X hours since the process started) my connection on my shell gives me a broken pipe error
ubuntu#ip-10-122-xxx-xxx:~/stratto/ode$ Write failed: Broken pipe
The instance is still there because I can reconnect with no problems, but how can I reconnect and get back at seeing the logs of the process as before the 'Broken Pipe'
Any tip much appreciated,
Thanks!
Redirect your output to a file and then run the program "nohup ..." to ensure the disconnect doesn't kill it. Use "tail -f" to monitor the redirected file.
Note: Originally said to use "tee" but that won't work. I think a straight redirect and then tail on the file works.
You can use screen to run processes in the cloud even when you are not connected to the server.
sudo apt install screen
To specifically address the issue described in the original post (e.g. connecting to AWS EC2 instances) I a basic example and a more advanced example of using screen.
You can use "screen". Detach from it and ping to google.com. So there ssh session will be active through out the installation.