gerrit ssh permission denied (public key) - ssh

I am struggling to authenticate with gerrit.openbmc-project.xyz using ssh.
I am trying to use ssh on windows.
My .ssh\config looks like this:
Host openbmc.gerrit
Hostname gerrit.openbmc-project.xyz
Port 29418
User patelutpal
The contents of my *.ssh* directory look like this:
-rw-r--r-- 1 utpalp 1049089 107 Dec 7 23:04 config
-rw-r--r-- 1 utpalp 1049089 411 Dec 7 21:10 id_ed25519
-rw-r--r-- 1 utpalp 1049089 97 Dec 7 21:10 id_ed25519.pub
-rw-r--r-- 1 utpalp 1049089 3381 Dec 7 22:14 id_rsa
-rw-r--r-- 1 utpalp 1049089 741 Dec 7 22:14 id_rsa.pub
-rw-r--r-- 1 utpalp 1049089 419 Dec 7 22:53 known_hosts
I have logged into gerrit.openbmc-project.xyz with my github account with id patelutpal.
I have added id_rsa.pub ssh key to my settings in https://gerrit.openbmc-project.xyz/settings/#SSHKeys
When I try:
C:\Users\utpalp\git>ssh -p 29418 -vvvt patelutpal#gerrit.openbmc-project.xyz
I get the following output.
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug1: Reading configuration data C:\\Users\\utpalp/.ssh/config
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolving "gerrit.openbmc-project.xyz" port 29418
debug2: ssh_connect_direct
debug1: Connecting to gerrit.openbmc-project.xyz [169.44.58.24] port 29418.
debug1: Connection established.
debug1: identity file C:\\Users\\utpalp/.ssh/id_rsa type 0
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_rsa-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_rsa-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_rsa-cert type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_dsa error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_dsa.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_dsa type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_dsa-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_dsa-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_dsa-cert type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ecdsa error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ecdsa.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_ecdsa type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ecdsa-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ecdsa-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\utpalp/.ssh/id_ed25519 type 3
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ed25519-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ed25519-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_ed25519-cert type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_xmss error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_xmss.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_xmss type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_xmss-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_xmss-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version GerritCodeReview_3.2.6 (APACHE-SSHD-2.4.0)
debug1: no match: GerritCodeReview_3.2.6 (APACHE-SSHD-2.4.0)
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gerrit.openbmc-project.xyz:29418 as 'patelutpal'
debug3: put_host_port: [gerrit.openbmc-project.xyz]:29418
debug3: hostkeys_foreach: reading file "C:\\Users\\utpalp/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file C:\\Users\\utpalp/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from [gerrit.openbmc-project.xyz]:29418
debug3: Failed to open file:C:/Users/utpalp/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01#openssh.com,rsa-sha2-512-cert-v01#openssh.com,rsa-sha2-256-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group18-sha512,diffie-hellman-group17-sha512,diffie-hellman-group16-sha512,diffie-hellman-group15-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm#openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm#openssh.com compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:LCaJB3JewPu5SBvlqsZfvKJD7yOVi7YaLWAEjsHSG60
debug3: put_host_port: [169.44.58.24]:29418
debug3: put_host_port: [gerrit.openbmc-project.xyz]:29418
debug3: hostkeys_foreach: reading file "C:\\Users\\utpalp/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file C:\\Users\\utpalp/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from [gerrit.openbmc-project.xyz]:29418
debug3: Failed to open file:C:/Users/utpalp/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: hostkeys_foreach: reading file "C:\\Users\\utpalp/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file C:\\Users\\utpalp/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from [169.44.58.24]:29418
debug3: Failed to open file:C:/Users/utpalp/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug1: Host '[gerrit.openbmc-project.xyz]:29418' is known and matches the ECDSA host key.
debug1: Found key in C:\\Users\\utpalp/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_rsa RSA SHA256:obA7xm+M+BuNoN/miSnhiIZmPbCIdpvMTj+xu8SRDsI
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_ed25519 ED25519 SHA256:zOx6/ktAHFTrf/yC4zBC55mEbfRZHF290Q29st34SA0
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: C:\\Users\\utpalp/.ssh/id_rsa RSA SHA256:obA7xm+M+BuNoN/miSnhiIZmPbCIdpvMTj+xu8SRDsI
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: C:\\Users\\utpalp/.ssh/id_dsa
debug3: no such identity: C:\\Users\\utpalp/.ssh/id_dsa: No such file or directory
debug1: Trying private key: C:\\Users\\utpalp/.ssh/id_ecdsa
debug3: no such identity: C:\\Users\\utpalp/.ssh/id_ecdsa: No such file or directory
debug1: Offering public key: C:\\Users\\utpalp/.ssh/id_ed25519 ED25519 SHA256:zOx6/ktAHFTrf/yC4zBC55mEbfRZHF290Q29st34SA0
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: C:\\Users\\utpalp/.ssh/id_xmss
debug3: no such identity: C:\\Users\\utpalp/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
patelutpal#gerrit.openbmc-project.xyz: Permission denied (publickey).
I dont know what to do. I think I have done all the right things. I wonder if this is a server side error. Can someone please help?
I will gladly provide more information if needed.
Thank you.
-Utpal


Context
OpenSSH 8.8 disables RSA signatures using the SHA-1 hash algorithm because they are considered cryptographically broken.
Git for Windows 2.33.1 upgrades OpenSSH to 8.8.
Note that this is a client side restriction.
Error
When trying to use a newer SSH client with git to clone or fetch using SSH, the following error might be observed:
git fetch
Fetching origin
dxxxxx#your.gerrit.host: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
error: Could not fetch origin
Done
Solution
You must provide a key type that is accepted by OpenSSH 8.8. OpenSSH recommends e.g. Ed25519:
ssh-keygen -t ed25519 -C "your_email#example.com"
copy content of ".ssh/id_ed25519.pub" to your Gerrit user settings
If your $HOME/.ssh/config file sets the IdentityFile option, then you may need to change the path accordingly, e.g.:
Host your.gerrit.host
IdentityFile ~/.ssh/id_ed25519
Alternatively you can configure the client to accept RSA by adding this ssh config option in ~/.ssh/config:
PubkeyAcceptedKeyTypes +ssh-rsa


I had a similar issue recently on Windows and GitBash. I downgraded to GitBash 2.27 and it worked again.

Related

"ssh localhost" not working without entering login password on Mac

When I fire the command "ssh localhost", I am always getting asked for password.
partha#Parthas-MBP ~ % ssh localhost
(partha#localhost) Password:
I have given proper rights to all files under "~/.ssh/" folder
partha#Parthas-MBP ~ % ls -lrt ~/.ssh/*
-rwxrwxr-x 1 partha staff 2610 May 15 15:17 /Users/partha/.ssh/id_rsa
-rwxrwxr-x 1 partha staff 572 May 15 15:17 /Users/partha/.ssh/id_rsa.pub
-rwxrwxr-x 1 partha staff 572 May 15 15:17 /Users/partha/.ssh/authorized_keys
-rwxrwxr-x 1 partha staff 91 May 15 15:18 /Users/partha/.ssh/known_hosts.old
-rwxrwxr-x 1 partha staff 825 May 15 15:18 /Users/partha/.ssh/known_hosts
The complete verbose log is as below.
partha#Parthas-MBP ~ % ssh -vvv localhost
OpenSSH_8.6p1, LibreSSL 3.3.5
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/partha/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/partha/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to localhost port 22.
debug1: Connection established.
debug1: identity file /Users/partha/.ssh/id_rsa type 0
debug1: identity file /Users/partha/.ssh/id_rsa-cert type -1
debug1: identity file /Users/partha/.ssh/id_dsa type -1
debug1: identity file /Users/partha/.ssh/id_dsa-cert type -1
debug1: identity file /Users/partha/.ssh/id_ecdsa type -1
debug1: identity file /Users/partha/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/partha/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/partha/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/partha/.ssh/id_ed25519 type -1
debug1: identity file /Users/partha/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/partha/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/partha/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/partha/.ssh/id_xmss type -1
debug1: identity file /Users/partha/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.6
debug1: compat_banner: match: OpenSSH_8.6 pat OpenSSH* compat 0x04000000
debug3: fd 6 is O_NONBLOCK
debug1: Authenticating to localhost:22 as 'partha'
debug3: record_hostkey: found key type ED25519 in file /Users/partha/.ssh/known_hosts:1
debug3: record_hostkey: found key type RSA in file /Users/partha/.ssh/known_hosts:2
debug3: record_hostkey: found key type ECDSA in file /Users/partha/.ssh/known_hosts:3
debug3: load_hostkeys_file: loaded 3 keys from localhost
debug1: load_hostkeys: fopen /Users/partha/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01#openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01#openssh.com,ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,sk-ssh-ed25519-cert-v01#openssh.com,sk-ecdsa-sha2-nistp256-cert-v01#openssh.com,rsa-sha2-512-cert-v01#openssh.com,rsa-sha2-256-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519#openssh.com,sk-ecdsa-sha2-nistp256#openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com
debug2: compression stoc: none,zlib#openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:vQ9e/Je46T5XXO4ar2I6IeaJHbZ2lQfJkFm/qy85JHc
debug3: record_hostkey: found key type ED25519 in file /Users/partha/.ssh/known_hosts:1
debug3: record_hostkey: found key type RSA in file /Users/partha/.ssh/known_hosts:2
debug3: record_hostkey: found key type ECDSA in file /Users/partha/.ssh/known_hosts:3
debug3: load_hostkeys_file: loaded 3 keys from localhost
debug1: load_hostkeys: fopen /Users/partha/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'localhost' is known and matches the ED25519 host key.
debug1: Found key in /Users/partha/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/partha/.ssh/id_rsa RSA SHA256:97SoRQEsHc2Yixc4X+QfE1c+PKJMpFbE/vjAqjJUNcM
debug1: Will attempt key: /Users/partha/.ssh/id_dsa
debug1: Will attempt key: /Users/partha/.ssh/id_ecdsa
debug1: Will attempt key: /Users/partha/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/partha/.ssh/id_ed25519
debug1: Will attempt key: /Users/partha/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/partha/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256#openssh.com,webauthn-sk-ecdsa-sha2-nistp256#openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug3: start over, passed a different list publickey,password,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/partha/.ssh/id_rsa RSA SHA256:97SoRQEsHc2Yixc4X+QfE1c+PKJMpFbE/vjAqjJUNcM
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Trying private key: /Users/partha/.ssh/id_dsa
debug3: no such identity: /Users/partha/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /Users/partha/.ssh/id_ecdsa
debug3: no such identity: /Users/partha/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /Users/partha/.ssh/id_ecdsa_sk
debug3: no such identity: /Users/partha/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /Users/partha/.ssh/id_ed25519
debug3: no such identity: /Users/partha/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /Users/partha/.ssh/id_ed25519_sk
debug3: no such identity: /Users/partha/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /Users/partha/.ssh/id_xmss
debug3: no such identity: /Users/partha/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req: entering
debug2: input_userauth_info_req: num_prompts 1
(partha#localhost) Password:

Delete the ssh key you have now and create a new one using ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa to create a passwordless one
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

GitLab Omnibus 14.7.3 ssh "permission denied" error - but specific to "CentOS Stream", works OK on Amazon Linux 2

I have built a brand-new AWS-hosted GitLab Omnibus server (14.7.3-ee) with its own domain name. I have been building this server from Terraform and Ansible for years and never had a problem. But it was based upon CentOS 8, and I’ve just updated it to use "CentOS Stream".
cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
Since then I cannot seem to get ssh working at all. And I'm pretty sure that the OS is a factor, because if I build the server with the same Ansible and Terraform, but specify Amazon Linux as the OS I don't have this problem.
I’ve created an ssh keypair, again using a script that I’ve had for years. Pasted the pub part into https://gitlab#mydomain.com/-/profile/keys and it has been saved OK. But when I try to use ssh I get the dreaded: “Permission denied (publickey,gssapi-keyex,gssapi-with-mic)”. But I’m completely stumped as to what the problem is.
I’ve tried loads of troubleshooting articles and got nowhere; and I can’t even find the ssh log on the GitLab server as /var/log/auth.log doesn’t exist.
So what I'm really after is any advice on how to troubleshoot this problem further?
ssh -vvvv output shown below.
$ ssh -i /home/peter/.ssh/root-mydomain.com_id_rsa -Tvvvv git#gitlab#mydomain.com
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/peter/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "mydomain.com" port 22
debug2: ssh_connect_direct
debug1: Connecting to mydomain.com [3.9.50.219] port 22.
debug1: Connection established.
debug1: identity file /home/peter/.ssh/root-mydomain.com_id_rsa type 0
debug1: identity file /home/peter/.ssh/root-mydomain.com_id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to mydomain.com:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/peter/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/peter/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from mydomain.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.c
om,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,di
ffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nist
p256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256-cert-v01#openssh.com,ssh-ed25519-cert-v01#openssh.com,sk-ssh-ed25519-cert-v01#openssh.com,rsa-sha2-
512-cert-v01#openssh.com,rsa-sha2-256-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,sk-ecdsa-sha2-nistp256#openssh.com,ssh-ed25519,sk-ssh-ed25519#openssh.com,rsa-s
ha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#opens
sh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#opens
sh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,di
ffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#open
ssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#open
ssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib#openssh.com
debug2: compression stoc: none,zlib#openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:ruhWAao5k3Y9OCJVphXBc1I8WkqRXeRu6p5LQmdbXCU
debug3: hostkeys_foreach: reading file "/home/peter/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/peter/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from mydomain.com
debug3: hostkeys_foreach: reading file "/home/peter/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/peter/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from 3.9.50.219
debug1: Host 'mydomain.com' is known and matches the ECDSA host key.
debug1: Found key in /home/peter/.ssh/known_hosts:3
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/peter/.ssh/root-mydomain.com_id_rsa RSA SHA256:foPG+FSppSKdlcygi8+FuqMsfPJO5VaVs7ZqvuBsaLI explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)
debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/peter/.ssh/root-mydomain.com_id_rsa RSA SHA256:foPG+FSppSKdlcygi8+FuqMsfPJO5VaVs7ZqvuBsaLI explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git#mydomain.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

What is required to achieve GSSAPI authentication in SSH?

I have set up a Kerberos cluster with docker containers as such:
One container runs the KDC.
One container has SSH server.
One container has SSH client.
Server setup
I create a service principal host/krb5-service.myexample.com for the server, create a keytab with that principal and places it in the krb5-server container.
I also created a user in the krb-server container named "yoav" - Don't know if that's necessary..
Client setup
I also create a client principal john. Then I run kinit yoav inside the krb5-client container.
Now I try to connect with SSH from the client to the server
$ ssh yoav#krb5-server.myexample.com
That's what I get as output:
# KRB5_TRACE=/dev/stdout ssh -vvv -o PreferredAuthentications=gssapi-with-mic yoav#krb5-server.myexample.com
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "krb5-server.myexample.com" port 22
debug2: ssh_connect_direct
debug1: Connecting to krb5-server.myexample.com [172.21.0.2] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to krb5-server.myexample.com:22 as 'yoav'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from krb5-server.myexample.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256-cert-v01#openssh.com,ssh-ed25519-cert-v01#openssh.com,sk-ssh-ed25519-cert-v01#openssh.com,rsa-sha2-512-cert-v01#openssh.com,rsa-sha2-256-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,sk-ecdsa-sha2-nistp256#openssh.com,ssh-ed25519,sk-ssh-ed25519#openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com
debug2: compression stoc: none,zlib#openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:5Mill1PYxkf1X/hHoMQfbbx4h/QSsRZv5/xaCAuJbsY
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from krb5-server.myexample.com
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 172.21.0.2
debug1: Host 'krb5-server.myexample.com' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /root/.ssh/id_rsa
debug1: Will attempt key: /root/.ssh/id_dsa
debug1: Will attempt key: /root/.ssh/id_ecdsa
debug1: Will attempt key: /root/.ssh/id_ecdsa_sk
debug1: Will attempt key: /root/.ssh/id_ed25519
debug1: Will attempt key: /root/.ssh/id_ed25519_sk
debug1: Will attempt key: /root/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256#openssh.com>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-with-mic
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred:
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
[50] 1643800420.264240: ccselect module realm chose cache FILE:/tmp/krb5cc_0 with client principal yoav#MYEXAMPLE.COM for server principal host/krb5-server.myexample.com#MYEXAMPLE.COM
[50] 1643800420.264241: Getting credentials yoav#MYEXAMPLE.COM -> host/krb5-server.myexample.com#MYEXAMPLE.COM using ccache FILE:/tmp/krb5cc_0
[50] 1643800420.264242: Retrieving yoav#MYEXAMPLE.COM -> host/krb5-server.myexample.com#MYEXAMPLE.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[50] 1643800420.264244: Creating authenticator for yoav#MYEXAMPLE.COM -> host/krb5-server.myexample.com#MYEXAMPLE.COM, seqnum 138014910, subkey aes256-cts/1E83, session key aes256-cts/5566
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
[50] 1643800420.264251: ccselect module realm chose cache FILE:/tmp/krb5cc_0 with client principal yoav#MYEXAMPLE.COM for server principal host/krb5-server.myexample.com#MYEXAMPLE.COM
[50] 1643800420.264252: Getting credentials yoav#MYEXAMPLE.COM -> host/krb5-server.myexample.com#MYEXAMPLE.COM using ccache FILE:/tmp/krb5cc_0
[50] 1643800420.264253: Retrieving yoav#MYEXAMPLE.COM -> host/krb5-server.myexample.com#MYEXAMPLE.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[50] 1643800420.264255: Getting credentials yoav#MYEXAMPLE.COM -> host/krb5-server.myexample.com#MYEXAMPLE.COM using ccache FILE:/tmp/krb5cc_0
[50] 1643800420.264256: Retrieving yoav#MYEXAMPLE.COM -> host/krb5-server.myexample.com#MYEXAMPLE.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[50] 1643800420.264258: Creating authenticator for yoav#MYEXAMPLE.COM -> host/krb5-server.myexample.com#MYEXAMPLE.COM, seqnum 1058679402, subkey aes256-cts/87DD, session key aes256-cts/5566
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
yoav#krb5-server.myexample.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
NOTE: I have a server and client C applications that perform Kerberos authentication, and that works fine, so the problem is not with the setup of the cluster...
Reference: You can look at the code here: https://github.com/yoav-klein/kerberos

Gitlab SSH: Permission Denied, Despite correct key setup

I'm trying to use Gitlab for the first time, and have been trying to set up my account to automatically ssh in. I've followed instructions and tried many different times, but no matter what, I can't ssh to gitlab properly.
I've tried deleting my id_rsa and id_rsa.pub files and regenerating keys, using the command line to copy the contents of id_rsa.pub to my Gitlab user settings (both with a newline and no newline at the end of the paste). I added my key using ssh-add. I'm running ssh -vvvv git#gitlab.com for verbosity and I just cannot connect succesfully for the life of me.
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "gitlab.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to gitlab.com [35.231.145.151] port 22.
debug1: Connection established.
debug1: identity file /home/jihan/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jihan/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jihan/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jihan/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jihan/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jihan/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jihan/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jihan/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.com:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/jihan/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/jihan/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from gitlab.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com
debug2: compression stoc: none,zlib#openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256#libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw
debug3: hostkeys_foreach: reading file "/home/jihan/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/jihan/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from gitlab.com
debug3: hostkeys_foreach: reading file "/home/jihan/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/jihan/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from 35.231.145.151
debug1: Host 'gitlab.com' is known and matches the ECDSA host key.
debug1: Found key in /home/jihan/.ssh/known_hosts:3
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/jihan/.ssh/id_rsa (0x55e032d43000), agent
debug2: key: jihany#nvidia.com (0x55e032d363a0), agent
debug2: key: /home/jihan/.ssh/id_dsa ((nil))
debug2: key: /home/jihan/.ssh/id_ecdsa ((nil))
debug2: key: /home/jihan/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/jihan/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: jihany#nvidia.com
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/jihan/.ssh/id_dsa
debug3: no such identity: /home/jihan/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/jihan/.ssh/id_ecdsa
debug3: no such identity: /home/jihan/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/jihan/.ssh/id_ed25519
debug3: no such identity: /home/jihan/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
It has to be some small dumb mistake somewhere, but I just can't figure it out. If it helps, when I do ssh-add -l, my fingerprints look like a hash for some reason.

Do check that:
when you copy the public key in your GitLab settings profile, it is copied as one line
when you generate your RSA SSH key, you use the old legacy profile (in case the new OPENSSH one with 70 chars per line is not well interpreted)
That is
ssh-keygen -m PEM -t rsa -P "

Weird problems with SSH on remote server

I am able to login to my virtual server as root as well as username naveed through password-less SSH. I have to run Hadoop on it, for that purpose I have created private and public keys in my remote ssh directory and also added public key to authorized_keys file.
But when I do ssh comquest it asks for password. Comquest is hostname of my remote server.
Below are my permissions info:
naveed#comquest:~$ stat -c "%a %n" ~/.ssh
755 /home/naveed/.ssh
naveed#comquest:~$ stat -c "%a %n" ~/.ssh/authorized_keys
644 /home/naveed/.ssh/authorized_keys
Below is the error I got with naveed#comquest:$ ssh -vvv comquest
naveed#comquest:/usr/local/hadoop$ ssh -vvv comquest
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "comquest" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to comquest [173.82.212.48] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/naveed/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/naveed/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/naveed/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/naveed/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/naveed/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/naveed/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/naveed/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/naveed/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to comquest:22 as 'naveed'
debug3: hostkeys_foreach: reading file "/home/naveed/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/naveed/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from comquest
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01#openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01#openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ssh-ed25519-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com
debug2: compression stoc: none,zlib#openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256#libssh.org
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SHA256:jad0RkXVMVxVyKD+9eY7abnSnUEz31nn+IMCfVUk7CA
debug3: hostkeys_foreach: reading file "/home/naveed/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/naveed/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from comquest
debug3: hostkeys_foreach: reading file "/home/naveed/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/naveed/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 173.82.212.48
debug1: Host 'comquest' is known and matches the RSA host key.
debug1: Found key in /home/naveed/.ssh/known_hosts:2
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug2: key: /home/naveed/.ssh/id_rsa ((nil))
debug2: key: /home/naveed/.ssh/id_dsa ((nil))
debug2: key: /home/naveed/.ssh/id_ecdsa ((nil))
debug2: key: /home/naveed/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 53
debug3: input_userauth_banner
CloudCone LLC
-----------------------------------------------
KVM Cloud Server Ubuntu 16.04.3 LTS
-----------------------------------------------
Service Status status.cloudcone.com
Client Area & Help Desk app.cloudcone.com
FAQs and Documentation help.cloudcone.com
-- NOTICE ---------------------------------------------------------------
This computer system is for authorized users only. Individuals using this
system without authority or in excess of their authority are subject to
having all their activities on this system monitored and recorded or
examined by any authorized person, including law enforcement, as system
personnel deem appropriate. In the course of monitoring individuals
improperly using the system or in the course of system maintenance, the
activities of authorized users may also be monitored and recorded. Any
material so recorded may be disclosed as appropriate. Anyone using this
system consents to these terms.
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/naveed/.ssh/id_rsa
debug3: no such identity: /home/naveed/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/naveed/.ssh/id_dsa
debug3: no such identity: /home/naveed/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/naveed/.ssh/id_ecdsa
debug3: no such identity: /home/naveed/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/naveed/.ssh/id_ed25519
debug3: no such identity: /home/naveed/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
naveed#comquest's password:
Below are the contents of my remote ~/.ssh folder:
naveed#comquest:~$ ls ~/.ssh
authorized_keys known_hosts local local.pub
local and local.pub are keys generated by naveed#comquest to access comquest. local.pub is added to authorized_keys.

Your private key is in a file named local, but that's not a name that ssh looks for (by default). Your ssh -vvv output shows that it's only looking for the default key filenames, such as id_rsa and id_dsa.
Either rename your local and local.pub files to match the expected naming convention (e.g. id_rsa or whatever, depending on what type of key it is), or add -o IdentityFile=~/.ssh/local to your ssh command line.