Unable to get SSH connections working on On Premise GitLab Instance - ssh

Question

I have deployed an on premise instance of GitLab (using the steps described in https://docs.gitlab.com/ee/install/docker.html), created a user, generated a key pair and uploaded the public key by following the steps in https://docs.gitlab.com/ee/ssh/.
The private key is located in directory ~/.ssh with 600 permissions, and file ~/.ssh/config is as follows:
Host gitlab.mycompany.com
Hostname gitlab.mycompany.com
User myusername
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa
IdentitiesOnly yes
TCPKeepAlive yes
However, I'm getting error 'Permission denied (publickey)' when trying to establish an SSH connection:
$ ssh -vT git#gitlab.mycompany.com
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/myusername/.ssh/config
debug1: /home/myusername/.ssh/config line 1: Applying options for gitlab.mycompany.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to gitlab.mycompany.com [12.345.678.90] port 22.
debug1: Connection established.
debug1: identity file /home/myusername/.ssh/id_rsa type 0
debug1: identity file /home/myusername/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to gitlab.mycompany.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:9FbhbOAyNvky0M+CtyEOT8tBBimwF8aOpDH0zr+6+2Y
debug1: Host 'gitlab.mycompany.com' is known and matches the ECDSA host key.
debug1: Found key in /home/myusername/.ssh/known_hosts:29
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/myusername/.ssh/id_rsa RSA SHA256:uekBVw1MVUz2V4LgS//w1mAP9wBRr1oomLK5uYtfJDE explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256#openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/myusername/.ssh/id_rsa RSA SHA256:uekBVw1MVUz2V4LgS//w1mAP9wBRr1oomLK5uYtfJDE explicit
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
git#gitlab.mycompany.com: Permission denied (publickey).
I believe the issue is on the server side because I was able to establish a successful connection to gitlab.com using the same key pair.
Any ideas on how to troubleshoot this issue?
Additional Details
Something I did not mention in my initial description of the issue is that GitLab was deployed as a container, so I'm thinking what might be happening is that the host is reachable over port 22, but it might not be re-directing to the GitLab container.

Answer 1

Check /var/log/auth.log on your Gitlab host. It tends to have more details on authentication errors. Look for sshd entries.

Answer 2

I managed to resolve the issue :)
Turns out, I missed the --publish 22:22 directive when creating the container.
sudo docker run --detach \
--hostname gitlab.mycompany.com \
--publish 443:443 --publish 80:80 --publish 22:22 \
--name gitlab \
--restart always \
--volume $GITLAB_HOME/config:/etc/gitlab:Z \
--volume $GITLAB_HOME/logs:/var/log/gitlab:Z \
--volume $GITLAB_HOME/data:/var/opt/gitlab:Z \
gitlab/gitlab-ee:latest
The resolution was:
Stop the container
$ sudo docker stop gitlab
Turn off docker
$ sudo systemctl stop docker
Edit file /var/lib/docker/containers/[container id]/hostconfig.json to change PortBindings from
"PortBindings":{"443/tcp":[{"HostIp":"","HostPort":"443"}],"80/tcp":[{"HostIp":"","HostPort":"80"}]}
to
"PortBindings":{"443/tcp":[{"HostIp":"","HostPort":"443"}],"80/tcp":[{"HostIp":"","HostPort":"80"}],"2222/tcp":[{"HostIp":"","HostPort":"22"}]}
(I opted to use port 2222 instead)
Start up docker
$ sudo systemctl start docker
Start the container
$ sudo docker start gitlab
The above allowed me to successfully connect:
$ ssh -T git#gitlab.mycompany.com -p 2222
Enter passphrase for key '/home/<username>/.ssh/id_rsa':
Welcome to GitLab, #<username>!