I've recently set up a new GitLab docker container, and though everything else has been working great I can't authenticate to it via ssh.
I followed the instructions here to the letter, with no succes.
Whatever key type I generate, and regardless of the client (Linux, Windows git-bash), The server instantly rejects the publickey and does not prompt for a password.
Debug shows the following:
debug1: Offering public key: /c/Users/[user]/.ssh/id_ed25519 ED25519 SHA256:[SHA-256]
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
Maybe it's something obvious, but I can't quite figure it out and no troubleshooting step managed to help.
As a side note, the ssh port is non standard, though I am accesing via the new port. I've also double checked ssh is enabled on both the server and the clients.
Any help would be greatly appreciated.
Thanks!
Check first if this is similar to gitlab-org/gitlab-foss issue 18371 "docker omnibus gitlab denying ssh public key"
In my case the problem was mismatch between ssh socket that docker container was exposing and my server's one.
It helped to expose it on different port like 10022 and reconfiguring gitlab like this:
gitlab_rails['gitlab_shell_ssh_port'] = 10022
Ideally, you would need to stop, restart the ssh daemon (server side, container side as seen in this thread) with
usr/sbin/sshd -d
That would allow you to check:
if the SSH request is received at all
if it is blocked for any reason
Related
I have a staging server and a production server, and I run identical Bitbucket Pipelines, where I send some commands over SSH. Unfortunately, my pipeline for the production always fails with:
Host key verification failed.
I've tried everything, folder permissions, recreating the keys, nothing works.
Finally with adding -v to my ssh call, I think I'm a step closer, but still lost.
On my staging server, I see something like this:
debug1: Host '$STAGING_SERVER' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:4
debug1: ssh_rsa_verify: signature correct
and the rest of the build follows flawlessly.
On my production server, however, I see the following:
debug1: Host '$PRODUCTION_SERVER' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:5
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: permanently_drop_suid: 0
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Host key verification failed.
So it would look like the key is found on my production server, but for some reason on the production server read_passphrase is being called. I've just created a new id_rsa and id_rsa.pub key, with no password, so why the heck is my production server trying to call read_passphrase? My ssh_config and sshd_config on both servers are identical - checked via diff.
Another way of looking at it is that ssh_rsa_verify is called immediately on the staging server, while on the production server read_passphrase is called.
Any help here would be greatly appreciated, this is driving me crazy!
Hallelujah! Solved! 🥳
Hours wasted for the simplest reason...
I noticed in the full output of the ssh -v on production that Bitbucket was printing out something like this:
debug1: Connecting to $PRODUCTION_SERVER [12.345.567.890] port 22.
where as the staging output was:
debug1: Connecting to $STAGING_SERVER [$STAGING_SERVER] port 22.
Meaning the static IP was the exact value of that repository variable. (Bitbucket parses secret logs out, which is why they appear this way).
I realized I had set the repository variable PRODUCTION_SERVER incorrectly to the alias for the IP address, (i.e. myserver.com) when it should be the IP address exactly. Changing that value in my repository variables to the IP address fixed the issue! Apparently, using the alias name isn't a perfect enough match for SSH to be satisfied.
I had the same issue. I solved this problem by
go to -> Repository settings
go to -> SSH keys ( on the left navigation)
at the known hosts section
input your Bastion host public IP address
then click Fetch button
rerun your pipeline
Please check this for reference
I have added an ecdsa-sha2-nistp256 SSH public key to my user account on a self-hosted GitLab instance. The same public key is used without issue on other services like GitHub.
When I try to clone any repo over SSH (git clone git#gitlab.local:user/project.git), including my own repos I am absolutely an owner of, I can see that the server accepts the public key:
debug1: Offering public key: ecdsa-sha2-nistp256 ECDSA SHA256:LL8b...Onco agent
debug1: Server accepts key: ecdsa-sha2-nistp256 ECDSA SHA256:LL8b...Onco agent
But the connection is immediately terminated and the clone aborted.
I've confirmed that ECDSA keys are enabled on the server, and that I'm connecting to the right server. Removing the public key from my profile results in a different error, so I know that (a) the server's authorized_keys setup is working, and (b) there isn't another user without repo privileges that I'm getting logged in as.
The issue seemed to be that /etc/pam.d/sshd had been modified in a way that prevented authentication.
We added the following to the top of the file:
# local user 'git' needs to be allowed
account sufficient pam_localuser.so
If the GitLab logs don't show any additional clue, I would start the ssh daemon on the server in debug mode: sshd -d
That will trigger a one-time interactive session, where you can see if your client SSH query:
arrives to the server
triggers any error message.
I use Portable Git x64 on Windows. I run everything thought Git Bash. I need to ssh to a server which is reachable only via HTTP proxy. Authentication for server is via pubkey, authentication for proxy is via password, usernames are different. My ~/.ssh/config:
Host server
Hostname server_hostname
User server_username
IdentityFile ~/.ssh/id_rsa
ProxyCommand /c/PortableGit/mingw64/bin/connect.exe -H proxy_username#proxy_ip:12345 %h %p
The problem starts when ssh tries to pop-up the window where you need to enter a password for the HTTP proxy, log from ssh -vvv server:
$ ssh -vvv server
OpenSSH_7.9p1, OpenSSL 1.1.1a 20 Nov 2018
debug1: Reading configuration data /c/Users/username/.ssh/config
debug1: /c/Users/username/.ssh/config line 1: Applying options for server
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Executing proxy command: exec /c/PortableGit/mingw64/bin/connect.exe -H proxy_username#proxy_ip:12345 server_hostname 22
debug1: identity file /c/Users/username/.ssh/id_rsa type 0
debug1: identity file /c/Users/username/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
'C:\PortableGit\mingw64\libexec\git-core\git-gui--askpass' is not recognized as an internal or external command,
operable program or batch file.
FATAL: Cannot decide password for proxy authentication.ssh_exchange_identification: Connection closed by remote host
git-gui--askpass is there, but for some reason it's not picked up by ssh. Running file 'C:\PortableGit\mingw64\libexec\git-core\git-gui--askpass' gives:
$ file 'C:\PortableGit\mingw64\libexec\git-core\git-gui--askpass'
C:\PortableGit\mingw64\libexec\git-core\git-gui--askpass: POSIX shell script, ASCII text executable
Content of the git-gui--askpass is identical to https://github.com/git/git/blob/3bab5d56259722843359702bc27111475437ad2a/git-gui/git-gui--askpass
I tried to run this script via command line, it works fine:
Also, I tried to specify another program as SSH_ASKPASS=/mingw64/libexec/git-core/git-askpass.exe (which I assume a stupid thing to do). This does not work either:
...
fatal: failed to acquire credentials.
I tried to supply a password in ~/.ssh/config as:
ProxyCommand /c/PortableGit/mingw64/bin/connect.exe -H proxy_username:proxy_password#proxy_ip:12345 %h %p
^^^^^^^^^^^^^^^
but this is ignored by ssh.
Besides, I tried to connect via MobaXterm and this works completely fine -- I've been asked for a proxy password and after entering it I am connected. Also, after connecting in MobaXterm I can connect in command line since the proxy does not ask for a password for some time. But for a different reason I cannot use MobaXterm.
Any ideas on how to make it work?
Utility connect.exe works with HTTP_PROXY_USER and HTTP_PROXY_PASSWORD environment variables. Solution found in source code
Try keeping your password in your ~/.ssh/config file and add
unset SSH_ASKPASS
To your .bashprofile
I'm trying to run the following scenario, using TCL script -
Scenario -
Host A runs the TCL script. Host A script connects to Host B through ssh. Then the script invokes an scp file transfer from Host C (server) to Host B (client).
Problem -
The script doesn't actually implement a timeout scenario. However, scp fails with no error message exactly after 10 seconds(probably timeout). If done manually, i.e. Logging in to Host B from Host A, and then scp from Host C to Host B, there is no timeout observed, and the file transfer is successful.
Implemented the ssh connection from tcl script using "expect" package.
What could be the reason? Kindly suggest some solutions.
Thank You.
Did you set
RSAAuthentication yes
on Host C, add the public key of Host B's user to Host C's user authorized_keys file?
See https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2 for more details.
Simple test:
Try to run the scp manually (or try ssh): It shouldn't ask you for a password. Running ssh -v from Host B to Host C should include the following lines:
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/xyz/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 1047
debug1: Authentication succeeded (publickey).
everyday I encounter a very strange phenomenon.
From my university internet connection, sshing to my machine ("ssh example.com") works without any problems.
From my home adsl, "ssh example.com" my console gets stuck with this message:
debug1: Server accepts key: pkalg ssh-rsa blen 533
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
Sometimes it might let me in but in most of the cases not.
The funny thing is that if I execute "ssh example.com bash -i" I get logged in immediately.
I finally found the source of the problem. It has to do with SSH type of service (ToS) TCP packets.
When you ask for a regular ssh teminal, ssh sets the TCP packet type of service (ToS) to "interactive". My router in my residence blocks those packet types!
Using netcat, the tunneled TCP packets get no type of service directives. Thus, if you tunnel all your ssh traffic through netcat, you reset the ToS of the TCP packets to the default ones.
In .ssh/config, you have to set:
Host *.example.com
ProxyCommand nc %h %p
So, each time you try to ssh to example.com, netcat will be called and will tunnel the packets.
As of OpenSSH 5.7, you can just add this to your ssh config file (either ~/.ssh/config or /etc/ssh/ssh_config):
Host *
IPQoS 0x00
This is a more-direct way to work around the problem Asterios identified.
I've just had the same problem. Try logging in with a different ssh client for more information. Whereas the linux command-line client didn't come back with any useful message, Putty came back with "server refused to allocate pty". I fixed it with mkdir /dev/pts and mount -a. How it got that mucked up in the first place I'm less sure about.
BTW, bash -l should act like a login shell so you should be able to prove Peter Westlake's suggestion correct or incorrect in your case fairly easily.
The difference between the two cases is that "bash -i" does not give you a login shell but just running ssh does. You can "man bash" for details of what a "login shell" is, but the main thing is that it runs /etc/profile and your .bash_profile. Have a look in those files for anything that might be causing a problem.
Maybe the server is out of ptys.