SSH to remote server refused if done via GitLab CI - ssh

Question

We have a RHEL 7 remote server where I created a dummy user called gitlabci.
While SSH'd into the remote server, I generated a public-private key pair (for use when grabbing files from GitLab)
Uploaded the public key as a deploy key for use later when we get our CI set up
Generated another public-private key pair in my local machine (for use when SSH'ing into the remote server from the GitLab Runner)
Added the public key to the remote server's authorized_keys
Added the private key to the project's CI environment variables
The idea is when the CI runs, the GitLab runner will SSH into the remote server as the gitlabci user I created then fetch the branch into the web directory using the deploy keys.
I thought I have set up the keys properly but whenever the runner tries to SSH, the connection gets refused.
$ which ssh-agent || ( apt-get update -y && apt-get install openssh-client git -y )
...
$ eval $(ssh-agent -s)
Agent pid 457
$ echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
Identity added: (stdin) (GitLab CI)
$ mkdir -p ~/.ssh
$ chmod 700 ~/.ssh
$ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ ssh gitlabci#random.server.com
Pseudo-terminal will not be allocated because stdin is not a terminal.
ssh: connect to host random.server.com port 22: Connection refused
ERROR: Job failed: exit code 1
When I tried to SSH into the remote server via GitBash on my local machine using the key pair I generated it did work.
$ ssh -i ~/.ssh/gitlabci gitlabci#random.server.com
Last login: Mon Nov 4 13:49:59 2019 from machine01.work.server.com

Answer 1

ssh: connect to host random.server.com port 22: Connection refused
"Connection refused" means that the ssh client transmitted a connection request to the named host and port, and it received in response a so-called "reset" packet, indicating that the remote server was refusing to accept the connection.
If you can connect to random.server.com from one host but get connection refused from another host, a few possible explanations come to mind:
You might have an entry in your .ssh/config file which substitutes a different name or address for random.server.com. For example, an entry like the following would cause ssh to connect to random2.server.com when you request random.server.com:
Host random.server.com
Hostname random2.server.com
The IP address lookup for "random.server.com" is returning the wrong address somehow, so ssh is trying to connect to the wrong server. For example, someone might have added an entry to /etc/hosts for that hostname.
Some firewall or other packet inspection software is interfering with the connection attempt by responding with a fake reset packet.