Gitlab SSH: Failed to get username: Internal API error (500) - ssh

I have installed GitLab from ArchLinux repository and added SSH key for user as described here and ArchWiki. First GitLab didn't added key to /var/lib/gitlab/.ssh/authorized_keys (home directory of gitlab user). Packet builders forgot to change authorized_keys_file option in gitlab.yml. I have fixed it. Now when I am trying to connect I get following:
$ ssh -vT gitlab#localhost
OpenSSH_8.1p1, OpenSSL 1.1.1d 10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to localhost [::1] port 22.
debug1: Connection established.
debug1: identity file /home/user1/.ssh/id_rsa type 0
debug1: identity file /home/user1/.ssh/id_rsa-cert type -1
debug1: identity file /home/user1/.ssh/id_dsa type -1
debug1: identity file /home/user1/.ssh/id_dsa-cert type -1
debug1: identity file /home/user1/.ssh/id_ecdsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user1/.ssh/id_ed25519 type -1
debug1: identity file /home/user1/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user1/.ssh/id_xmss type -1
debug1: identity file /home/user1/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.1
debug1: match: OpenSSH_8.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to localhost:22 as 'gitlab'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Z2mmD/K3tW9k25gNrpTgUefT94vbNXaUDcmWfd6o7v0
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /home/user1/.ssh/known_hosts:11
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/user1/.ssh/id_rsa RSA SHA256:Orfy0Kb/fAtiHmKVQJSovjHOg5b2TDSvQQ/dVh97RWQ
debug1: Will attempt key: /home/user1/.ssh/id_dsa
debug1: Will attempt key: /home/user1/.ssh/id_ecdsa
debug1: Will attempt key: /home/user1/.ssh/id_ed25519
debug1: Will attempt key: /home/user1/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user1/.ssh/id_rsa RSA SHA256:Orfy0Kb/fAtiHmKVQJSovjHOg5b2TDSvQQ/dVh97RWQ
debug1: Server accepts key: /home/user1/.ssh/id_rsa RSA SHA256:Orfy0Kb/fAtiHmKVQJSovjHOg5b2TDSvQQ/dVh97RWQ
debug1: Authentication succeeded (publickey).
Authenticated to localhost ([::1]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: exec
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Remote: /var/lib/gitlab/.ssh/authorized_keys:1: key options: command user-rc
debug1: Remote: /var/lib/gitlab/.ssh/authorized_keys:1: key options: command user-rc
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Remote: X11 forwarding disabled by key options.
X11 forwarding request failed on channel 0
remote:
remote: ========================================================================
remote:
remote: Failed to get username: Internal API error (500)
remote:
remote: ========================================================================
remote:
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3176, received 3248 bytes, in 0.2 seconds
Bytes per second: sent 12932.5, received 13225.7
debug1: Exit status 1
# /var/log/gitlab/production.log:
ArgumentError (wrong number of arguments (given 3, expected 1..2)):
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:48:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:32:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Looks like a bug, but I doesn't find anything related in Google.
gitlab 12.6.2-3, git 2.24.1-4, gitlab-gitaly 1.78.0-2, gitlab-shell 2:10.3.0-2, ruby 2.7.0-1, openssh 8.1p1-2

Related

ssh, scp works, rsync over ssh doesn't

I try to run an rsync sync on a Synology from a linux machine.
I have access to the Synology with SSH with key authentication. SSH SCP works, permissions are correct.
~# scp /tmp/foo.bar backupch:/tmp
foo.bar 100% 9 0.5KB/s 00:00
Rsync fails:
rsync /tmp/foo.bar backupch:/tmp -e 'ssh -v'
Rsync output with debug:
OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 14: Applying options for backupch
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to myhost.mydomain [*.*.*.*] port 2222.
debug1: Connection established.
debug1: identity file /root/.ssh/ipdch.backup.key type -1
debug1: identity file /root/.ssh/ipdch.backup.key-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2
debug1: compat_banner: match: OpenSSH_8.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to myhost.mydomain:2222 as 'backup'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:bbl6asZ3Qc+DnOV8hKPDWdlnJtE2yXMQxb1jyInlc+c
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[myhost.mydomain]:2222' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /root/.ssh/ipdch.backup.key explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256#openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/ipdch.backup.key
Authenticated to myhost.mydomain ([*.*.*.*]:2222) using "publickey".
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts for [myhost.mydomain]:2222 / (none)
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts2 for [myhost.mydomain]:2222 / (none)
debug1: client_input_hostkeys: hostkeys file /root/.ssh/known_hosts2 does not exist
debug1: Remote: /var/services/homes/backup/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug1: Sending command: rsync --server -e.LsfxCIvu . /tmp
debug1: client_global_hostkeys_private_confirm: server used untrusted RSA signature algorithm ssh-rsa for key 0, disregarding
debug1: update_known_hosts: known hosts file /root/.ssh/known_hosts2 does not exist
Permission denied, please try again.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
Transferred: sent 3412, received 2716 bytes, in 0.1 seconds
Bytes per second: sent 56899.8, received 45293.1
debug1: Exit status 1
rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]
Any idea what is going wrong here??
It seems that there is a problem with newer versions of rsync on Synology.
https://community.synology.com/enu/forum/17/post/114547
The solution is to specify the --rsync-path=/bin/rsync with the rsync command.
rsync --rsync-path=/bin/rsync /tmp/foo.bar backupch:/tmp -e 'ssh -v'

Git Bash issue when 'Entering interactive session'. (Windows 10)

SSH used to work nicely with Git Bash before. Don't know what happened recently.
SSH version on CMD:
C:\Users\ my-user>ssh -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
Debug/errors On CMD:
C:\Users\ my-user>ssh -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
C:\Users\ my-user>ssh -v my-user1#server-url
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug1: Connecting to server-url [public-ip] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\ my-user/.ssh/id_rsa type 0
debug1: identity file C:\\Users\\ my-user/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_dsa-cert type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to server-url:22 as 'my-user1'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-ed25519 SHA256:Mw4Mlvqk2MgCynA+8nTUrxDy8wxXONlTAVk2K/FgbcQ
debug1: Host 'server-url' is known and matches the ED25519 host key.
debug1: Found key in C:\\Users\\ my-user/.ssh/known_hosts:167
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_rsa RSA SHA256:OyvnNnesOb+dn3j9NWl9dkDu/Fi52CKlN9Vq8h9LzoA
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
Authorized uses only. All activity may be monitored and reported.
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: C:\\Users\\ my-user/.ssh/id_rsa RSA SHA256:OyvnNnesOb+dn3j9NWl9dkDu/Fi52CKlN9Vq8h9LzoA
debug1: Server accepts key: C:\\Users\\ my-user/.ssh/id_rsa RSA SHA256:OyvnNnesOb+dn3j9NWl9dkDu/Fi52CKlN9Vq8h9LzoA
debug1: Authentication succeeded (publickey).
Authenticated to server-url ([public-ip]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: ENABLE_VIRTUAL_TERMINAL_INPUT is supported. Reading the VTSequence from console
debug1: ENABLE_VIRTUAL_TERMINAL_PROCESSING is supported. Console supports the ansi parsing
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
Last login: Tue Jun 28 09:20:50 2022 from org-ip
Same SSH version on Git Bash:
my-user#my-server MINGW64 /
$ ssh -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
Debug/errors On Git Bash:
my-user#my-server MINGW64 /
$ ssh -v my-user1#server-url
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Connecting to server-url [public-ip] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\ my-user/.ssh/id_rsa type 0
debug1: identity file C:\\Users\\ my-user/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_dsa-cert type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\ my-user/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to server-url:22 as 'my-user1'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-ed25519 SHA256:Mw4Mlvqk2MgCynA+8nTUrxDy8wxXONlTAVk2K/FgbcQ
debug1: Host 'server-url' is known and matches the ED25519 host key.
debug1: Found key in C:\\Users\\ my-user/.ssh/known_hosts:167
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_rsa RSA SHA256:OyvnNnesOb+dn3j9NWl9dkDu/Fi52CKlN9Vq8h9LzoA
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\ my-user/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
Authorized uses only. All activity may be monitored and reported.
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: C:\\Users\\ my-user/.ssh/id_rsa RSA SHA256:OyvnNnesOb+dn3j9NWl9dkDu/Fi52CKlN9Vq8h9LzoA
debug1: Server accepts key: C:\\Users\\ my-user/.ssh/id_rsa RSA SHA256:OyvnNnesOb+dn3j9NWl9dkDu/Fi52CKlN9Vq8h9LzoA
debug1: Authentication succeeded (publickey).
Authenticated to server-url ([public-ip]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
<gets stuck here>
Any help/leads is welcome
Note: Git Bash uses mintty terminal
found a workaround solution: SSH works if I select "Windows cmd" as terminal instead of "mintty" which is the default while installation

cant connect from VM1 to VM2

I have 2 virtual machines on my computer created with Vagrant.
on VM1 I create SSH key with ssh-keygen. Then copied id_rsa.pub into authorized_keys on VM2.
When I try to connect to VM2 using ssh 172.28.128.18, the connection is failed with following error
The authenticity of host '172.28.128.18 (172.28.128.18)' can't be established.
ECDSA key fingerprint is SHA256:YJ0CDFr9UcEBZkY1ajZRFg3Zx9uksZcv/i3fpC8gqdY.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '172.28.128.18' (ECDSA) to the list of known hosts.
vagrant#172.28.128.18: Permission denied (publickey).
With ssh -v vagrant#172.28.128.18 I have an other error:
OpenSSH_8.2p1 Ubuntu-4ubuntu0.3, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 172.28.128.18 [172.28.128.18] port 22.
debug1: Connection established.
debug1: identity file /home/vagrant/.ssh/id_rsa type 0
debug1: identity file /home/vagrant/.ssh/id_rsa-cert type -1
debug1: identity file /home/vagrant/.ssh/id_dsa type -1
debug1: identity file /home/vagrant/.ssh/id_dsa-cert type -1
debug1: identity file /home/vagrant/.ssh/id_ecdsa type -1
debug1: identity file /home/vagrant/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/vagrant/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/vagrant/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/vagrant/.ssh/id_ed25519 type -1
debug1: identity file /home/vagrant/.ssh/id_ed25519-cert type -1
debug1: identity file /home/vagrant/.ssh/id_ed25519_sk type -1
debug1: identity file /home/vagrant/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/vagrant/.ssh/id_xmss type -1
debug1: identity file /home/vagrant/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.3
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.3 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.28.128.18:22 as 'vagrant'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:YJ0CDFr9UcEBZkY1ajZRFg3Zx9uksZcv/i3fpC8gqdY
debug1: Host '172.28.128.18' is known and matches the ECDSA host key.
debug1: Found key in /home/vagrant/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/vagrant/.ssh/id_rsa RSA SHA256:1cq8iCIBMZrwP6BVucSZx4Luk3LfTAf/XOfovgHiW/Y
debug1: Will attempt key: /home/vagrant/.ssh/id_dsa
debug1: Will attempt key: /home/vagrant/.ssh/id_ecdsa
debug1: Will attempt key: /home/vagrant/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/vagrant/.ssh/id_ed25519
debug1: Will attempt key: /home/vagrant/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/vagrant/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256
#openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/vagrant/.ssh/id_rsa RSA SHA256:1cq8iCIBMZrwP6BVucSZx4Luk3LfTAf/XOfovgHiW/Y
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/vagrant/.ssh/id_dsa
debug1: Trying private key: /home/vagrant/.ssh/id_ecdsa
debug1: Trying private key: /home/vagrant/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/vagrant/.ssh/id_ed25519
debug1: Trying private key: /home/vagrant/.ssh/id_ed25519_sk
debug1: Trying private key: /home/vagrant/.ssh/id_xmss
debug1: No more authentication methods to try.
vagrant#172.28.128.18: Permission denied (publickey).
What is the reason of failing connecting to VM2?
on VM2
sudo tail -f /var/log/auth.log
server sshd[2509]: Connection closed by authenticating user vagrant 172.28.128.12 port 56446 [preauth]
Same problem is, when I try to connect from VM2 to VM1.

install arch gitlab and can not git clone using ssh method

I followed, https://wiki.archlinux.org/index.php/GitLab#Configuration, and finally brought gitlab it up. But, I could not use ssh to clone the project.
$ ssh -T gitlab#192.168.48.3
Failed to get username: who='' is invalid
What is this message who='' and how to debug???
Using ssh -Tv, I got
$ ssh -T gitlab#192.168.48.3
Failed to get username: who='' is invalid
[alexlai#N2eMMc128G ~]$ ssh -Tv gitlab#192.168.48.3
OpenSSH_8.1p1, OpenSSL 1.1.1d 10 Sep 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 192.168.48.3 [192.168.48.3] port 22.
debug1: Connection established.
debug1: identity file /home/alexlai/.ssh/id_rsa type 0
debug1: identity file /home/alexlai/.ssh/id_rsa-cert type -1
debug1: identity file /home/alexlai/.ssh/id_dsa type -1
debug1: identity file /home/alexlai/.ssh/id_dsa-cert type -1
debug1: identity file /home/alexlai/.ssh/id_ecdsa type -1
debug1: identity file /home/alexlai/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/alexlai/.ssh/id_ed25519 type -1
debug1: identity file /home/alexlai/.ssh/id_ed25519-cert type -1
debug1: identity file /home/alexlai/.ssh/id_xmss type -1
debug1: identity file /home/alexlai/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.1
debug1: match: OpenSSH_8.1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.48.3:22 as 'gitlab'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:TbXi96Gynm1xqcTglDvS4ajP7hPHMHf2hnOiJ3w5e74
debug1: Host '192.168.48.3' is known and matches the ECDSA host key.
debug1: Found key in /home/alexlai/.ssh/known_hosts:5
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/alexlai/.ssh/id_rsa RSA SHA256:tdOYKwO8CDOzJeJo/6WYbtStpsMg8kuSGLCie5JOY1g
debug1: Will attempt key: /home/alexlai/.ssh/id_dsa
debug1: Will attempt key: /home/alexlai/.ssh/id_ecdsa
debug1: Will attempt key: /home/alexlai/.ssh/id_ed25519
debug1: Will attempt key: /home/alexlai/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/alexlai/.ssh/id_rsa RSA SHA256:tdOYKwO8CDOzJeJo/6WYbtStpsMg8kuSGLCie5JOY1g
debug1: Server accepts key: /home/alexlai/.ssh/id_rsa RSA SHA256:tdOYKwO8CDOzJeJo/6WYbtStpsMg8kuSGLCie5JOY1g
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.48.3 ([192.168.48.3]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: Remote: /var/lib/gitlab/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /var/lib/gitlab/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Failed to get username: who='' is invalid
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3468, received 3288 bytes, in 0.1 seconds
Bytes per second: sent 39481.4, received 37432.2
debug1: Exit status 1

Cannot ssh to compute engine after suggested reboot

The last time I ssh into my compute engine instance, it suggested a system reboot. I rebooted it. After the reboot, I can no longer ssh into it. The instance is still running, the web server is still working. I can even get authentication succeeded. It than it close the connection. The log is a below.
ssh -v -i "privatekey" username#hostname
OpenSSH_7.2p2 Ubuntu-4ubuntu2.4, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to hostname [ip] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file [key location] type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/keyfile-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to hostname:22 as 'username'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256#libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: Server host key
debug1: Host 'hostname' is known and matches the ECDSA host key.
debug1: Found key in /home/username/.ssh/known_hosts:2
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: privatekeyfile
Enter passphrase for key 'privatekeyfile':
debug1: Authentication succeeded (publickey).
Authenticated to hostname ([ip]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: channel 0: free: client-session, nchannels 1
**Connection to hostname closed by remote host.**
**Connection to hostname closed.**
Transferred: sent 2256, received 1380 bytes, in 0.0 seconds
Bytes per second: sent 155684.5, received 95232.6
debug1: Exit status -1
Based on the debug output you have provided, I have found two resolved issues with similar output. Here are the two links found 1st link and 2nd link. They were essentially resolved by changing the ~/.ssh permissions.