SSH raspberrypi kex_exchange_identification: read: Connection reset - ssh

This issue happens after few days of using the pi and i have re flashed the OS multiple times and so far the temporary fix is to reboot the pi so could someone suggest me a permanent solution.
Model: raspberry pi 4
Os: raspbian 64 bit lite
error code
ssh -vvv pi#192.168.0.126
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug1: Reading configuration data C:\\Users\\rexdi/.ssh/config
debug1: C:\\Users\\rexdi/.ssh/config line 6: Applying options for 192.168.0.126
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolve_canonicalize: hostname 192.168.0.126 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.0.126 [192.168.0.126] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\rexdi/.ssh/id_rsa type 0
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_rsa-cert error:2
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_rsa-cert.pub error:2
debug1: identity file C:\\Users\\rexdi/.ssh/id_rsa-cert type -1
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_dsa error:2
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_dsa.pub error:2
debug1: identity file C:\\Users\\rexdi/.ssh/id_dsa type -1
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_dsa-cert error:2
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_dsa-cert.pub error:2
debug1: identity file C:\\Users\\rexdi/.ssh/id_dsa-cert type -1
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_ecdsa error:2
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_ecdsa.pub error:2
debug1: identity file C:\\Users\\rexdi/.ssh/id_ecdsa type -1
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_ecdsa-cert error:2
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_ecdsa-cert.pub error:2
debug1: identity file C:\\Users\\rexdi/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\rexdi/.ssh/id_ed25519 type 3
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_ed25519-cert error:2
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_ed25519-cert.pub error:2
debug1: identity file C:\\Users\\rexdi/.ssh/id_ed25519-cert type -1
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_xmss error:2
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_xmss.pub error:2
debug1: identity file C:\\Users\\rexdi/.ssh/id_xmss type -1
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_xmss-cert error:2
debug3: Failed to open file:C:/Users/rexdi/.ssh/id_xmss-cert.pub error:2
debug1: identity file C:\\Users\\rexdi/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug3: recv - from CB(2) ERROR:108, io:000001C6E5B76FF0
kex_exchange_identification: read: Connection reset

Related

ssh, scp works, rsync over ssh doesn't

I try to run an rsync sync on a Synology from a linux machine.
I have access to the Synology with SSH with key authentication. SSH SCP works, permissions are correct.
~# scp /tmp/foo.bar backupch:/tmp
foo.bar 100% 9 0.5KB/s 00:00
Rsync fails:
rsync /tmp/foo.bar backupch:/tmp -e 'ssh -v'
Rsync output with debug:
OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 14: Applying options for backupch
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to myhost.mydomain [*.*.*.*] port 2222.
debug1: Connection established.
debug1: identity file /root/.ssh/ipdch.backup.key type -1
debug1: identity file /root/.ssh/ipdch.backup.key-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2
debug1: compat_banner: match: OpenSSH_8.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to myhost.mydomain:2222 as 'backup'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:bbl6asZ3Qc+DnOV8hKPDWdlnJtE2yXMQxb1jyInlc+c
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[myhost.mydomain]:2222' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /root/.ssh/ipdch.backup.key explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519#openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256#openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /root/.ssh/ipdch.backup.key
Authenticated to myhost.mydomain ([*.*.*.*]:2222) using "publickey".
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions#openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts for [myhost.mydomain]:2222 / (none)
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts2 for [myhost.mydomain]:2222 / (none)
debug1: client_input_hostkeys: hostkeys file /root/.ssh/known_hosts2 does not exist
debug1: Remote: /var/services/homes/backup/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Sending environment.
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug1: Sending command: rsync --server -e.LsfxCIvu . /tmp
debug1: client_global_hostkeys_private_confirm: server used untrusted RSA signature algorithm ssh-rsa for key 0, disregarding
debug1: update_known_hosts: known hosts file /root/.ssh/known_hosts2 does not exist
Permission denied, please try again.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
Transferred: sent 3412, received 2716 bytes, in 0.1 seconds
Bytes per second: sent 56899.8, received 45293.1
debug1: Exit status 1
rsync error: error in rsync protocol data stream (code 12) at io.c(228) [sender=3.2.3]
Any idea what is going wrong here??

It seems that there is a problem with newer versions of rsync on Synology.
https://community.synology.com/enu/forum/17/post/114547
The solution is to specify the --rsync-path=/bin/rsync with the rsync command.
rsync --rsync-path=/bin/rsync /tmp/foo.bar backupch:/tmp -e 'ssh -v'

Ansible SSH user change during playbook execution causes freeze

I have a playbook that is launched from AWX with machine credentials, ie. with ssh_user_A. In this playbook, i need to perform a couple of tasks with another SSH user, ie. ssh_user_B.
In terms of credentials, I have:
machine credentials for ssh_user_A that are OK, I can do whatever I need
a public/private keys couple for ssh_user_B:
the public key is present on the remote host in the ~ssh_user_B/.ssh/authorized_keys file
if I test this public/private key with a temp machine credentials (not possible in final target) and a dummy playbook or an ad-hoc module call, it works I can do whatever I need
I change the SSH user in the playbook with these instructions:
- name: Switch to ssh_user_B user
ansible.builtin.set_fact:
ansible_user: "ssh_user_B"
ansible_ssh_pass: "{{ssh_user_B_passphrase}}"
ansible_ssh_private_key_file: "{{file_that_contains_the_private_key}}"
delegate_to: localhost
And the user change is triggered by a simple wait_for_connection:
- name: Just here to validate the user change
ansible.builtin.wait_for_connection:
timeout: 10
But unfortunately, the only thing that I get is a behavior that I do not understand:
with valid public/private keys, the task that is supposed to trigger the user change seems frozen at the first SSH connection attempt, nothing more happens, and nothing in the target remote host SSH logs (in these logs, I have the connection for the ssh_user_A, but absolutely nothing for the ssh_user_B one)
AWX output:
PLAYBOOK: test.yml *************************************************************
Positional arguments: test.yml
verbosity: 4
remote_user: ssh_user_A
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/tmp/awx_14019_pqypcw0i/tmpm_98ffss',)
subset: target_remote_host
extra_vars: ('#/tmp/awx_14019_pqypcw0i/tmp7493qmoq', '#/tmp/awx_14019_pqypcw0i/env/extravars')
forks: 5
1 plays in test.yml
Read vars_file './vars/main.yml'
Read vars_file './vars/main.yml'
Read vars_file './vars/main.yml'
PLAY [MultiUserTest] ***********************************************************
Read vars_file './vars/main.yml'
TASK [Gathering Facts] *********************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:2
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_A
... Connection OK
TASK [Another task OK] ***************************************************
...
TASK [Task done on localhost] ********************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:57
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
... Connection OK
TASK [Another locahost task OK] ***************************************************
...
TASK [switch to ssh_user_B user] *****************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:83
ok: [s4850ads -> localhost] => {
"ansible_facts": {
"ansible_ssh_pass": "dummy_passphrase",
"ansible_ssh_private_key_file": ".dummy_pk_random_name",
"ansible_user": "ssh_user_B"
},
"changed": false
}
Read vars_file './vars/main.yml'
TASK [Wait 300 seconds, but only start checking after 60 seconds] **************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:91
wait_for_connection: attempting ping module test
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_B
<target_remote_host> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile=".dummy_pk_random_name"' -o 'User="ssh_user_B"' -o ConnectTimeout=10 -o ControlPath=/tmp/awx_14019_pqypcw0i/cp/bc63f1e109 target_remote_host '/bin/sh -c '"'"'echo ~ssh_user_B && sleep 0'"'"''
... Here it's frozen, nothing more in the AWX output, nothing in the remote host SSH logs...
with an invalid public/private keys (ie. with additional characters, no LF at the end of the private key file, etc.), the task that is supposed to trigger the user works and ends in error. I can see that every SSH connection method is tested, and it ends in error:
TASK [switch to ssh_user_B user] *****************************************************
task path: /tmp/awx_14019_pqypcw0i/project/test.yml:83
ok: [s4850ads -> localhost] => {
"ansible_facts": {
"ansible_ssh_pass": "dummy_passphrase",
"ansible_ssh_private_key_file": ".dummy_pk_random_name",
"ansible_user": "ssh_user_B"
},
"changed": false
}
Read vars_file './vars/main.yml'
TASK [Wait 300 seconds, but only start checking after 60 seconds] **************
task path: /tmp/awx_14021_ypwngn_h/project/test.yml:91
wait_for_connection: attempting ping module test
<target_remote_host> ESTABLISH SSH CONNECTION FOR USER: ssh_user_B
<target_remote_host> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile=".dummy_pk_random_name"' -o 'User="ssh_user_B"' -o ConnectTimeout=10 -o ControlPath=/tmp/awx_14021_ypwngn_h/cp/bc63f1e109 target_remote_host '/bin/sh -c '"'"'echo ~ssh_user_B && sleep 0'"'"''
<target_remote_host> (5, b'', b'OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021\
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for \\'final all\\' host target_remote_host originally target_remote_host
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: not matched \\'final\\'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for \\'final all\\' host target_remote_host originally target_remote_host
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 3: matched \\'final\\'
debug2: match found
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-,gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: auto-mux: Trying existing master
debug1: Control socket "/tmp/awx_14021_ypwngn_h/cp/bc63f1e109" does not exist
debug2: resolving "target_remote_host" port 22
debug2: ssh_connect_direct
debug1: Connecting to target_remote_host [<IP>] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9984 ms remain after connect
debug1: identity file .dummy_pk_random_name type -1
debug1: identity file .dummy_pk_random_name-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to target_remote_host:22 as \\'ssh_user_B\\'
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from target_remote_host
debug3: order_hostkeyalgs: have matching best-preference key type ecdsa-sha2-nistp256-cert-v01#openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ssh-ed25519-cert-v01#openssh.com,rsa-sha2-512-cert-v01#openssh.com,rsa-sha2-256-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#openssh.com,hmac-sha2-512
debug2: compression ctos: zlib#openssh.com,zlib,none
debug2: compression stoc: zlib#openssh.com,zlib,none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm#openssh.com,chacha20-poly1305#openssh.com,aes256-ctr,aes256-cbc,aes128-gcm#openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha2-256,hmac-sha1,umac-128#openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib#openssh.com
debug2: compression stoc: none,zlib#openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm#openssh.com MAC: <implicit> compression: zlib#openssh.com
debug1: kex: client->server cipher: aes256-gcm#openssh.com MAC: <implicit> compression: zlib#openssh.com
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:YYY/ZZZ
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from target_remote_host
debug3: hostkeys_foreach: reading file "/root/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /root/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <IP>
debug1: Host \\'target_remote_host\\' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /tmp/awx_14021_ypwngn_h/artifacts/14021/ssh_key_data RSA SHA256:XXX agent
debug1: Will attempt key: .dummy_pk_random_name explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 53
debug3: input_userauth_banner
###################################################################
# SERVER MESSAGE. #
###################################################################
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /tmp/awx_14021_ypwngn_h/artifacts/14021/ssh_key_data RSA SHA256:XXX agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: .dummy_pk_random_name
Load key ".dummy_pk_random_name": invalid format
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can\\'t open /dev/tty: No such device or address
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
ssh_user_B#target_remote_host: Permission denied (publickey,password).
')
... Other SSH connection method attempt
I also see the connection attempts in the remote host SSH logs, they all look like this:
Feb 28 14:25:10 target_remote_host sshd[4108095]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<IP> user=ssh_user_B
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Failed password for ssh_user_B from <IP> port 12345 ssh2
Feb 28 14:25:12 target_remote_host sshd[4108095]: Connection closed by authenticating user ssh_user_B <IP> port 12345 [preauth]
What I've checked/done/excluded:
as said previously, the public/private key couple is OK, tested though AWX on a separate credentials machine
I've tested to copy the private key file on the remote folder in order to ckeck its content and see if it was well formed, and it is (here $ represent the LF):
-----BEGIN OPENSSH PRIVATE KEY-----$
...
-----END OPENSSH PRIVATE KEY-----$
I've created a new public/private key couple with a passphrase that only contains letters and digits, in order to exlude a problem from a special character in the passphrase (like a `), it does not change anything
If I do not provide a passphrase, I get a no passphrase given error from the remote host as expected
Regarding these elements, I assume that my SSH connection parameters are OK but something stucks the playbook, but I really do not understand what... Any clue/help would be appreciated.
Note that I do not (and won't) have access on the host where AWX is installed and the playbook runs.
Part of the playbook that contain the credentials change:
- name: Main private key block
block:
- name: Creating temp filenames
ansible.builtin.set_fact:
filename_private_key: ".dummy_pk_random_name"
# Just for testing purposes, target is a key in a Vault
temp_copy_file: "./files/temp_pk_file.txt"
- name: Sub part 1
block:
##########
- name: Creating container file for private key
ansible.builtin.file:
path: "{{filename_private_key}}"
state: touch
mode: '600'
changed_when: false
- name : Putting private key into needed file
ansible.builtin.copy:
src: "{{temp_copy_file}}"
dest: "{{filename_private_key}}"
mode: '600'
changed_when: false
##########
delegate_to: localhost
- name: Sub part 2
block:
##########
- name: Switch to ssh_user_B user
ansible.builtin.set_fact:
ansible_user: "ssh_user_B"
# The passphrase comes from a Vault
ansible_ssh_pass: "{{ssh_user_B_passphrase}}"
ansible_ssh_private_key_file: "{{filename_private_key}}"
delegate_to: localhost
- name: Just here to validate the user change
ansible.builtin.wait_for_connection:
timeout: 10
##########
##########
always:
- name: Ensure private key is not present anymore
file:
path: "{{ filename_private_key }}"
state: absent
changed_when: false
delegate_to: localhost
##########

I've made a lot of tests and found the problem: by default, to connect to the target hosts, Ansible uses the smart connection plugin. In my case, the smart plugin leads to the use of the native OpenSSH.
Forcing manually the use of the paramiko connection plugin solves the problem, everything is OK (paramiko is a Python implementation of OpenSSH). Just need to add the instruction connection: paramiko at the needed level:
---
- name: MultiUserTest part 1
hosts: all
gather_facts: yes
connection: paramiko

SFTP on Windows gives error : read_passphrase: can't open /dev/tty: No such file or directory

I am trying to use the windows SFTP command to connect to a server using a .pem openssh key file.
This command works on my own PC and on a LINUX AWS server however when I copy the y.pem file to an AWS windows server and try to connect from there, it gives the error below and asks for a non existent password. There is no passphrase or password to enter, only a .pem file is used to authenticate.
I am also able to connect from this windows server using filezilla so I am at a loss as to why the sftp fails?
sftp -vvv -i y.pem rdg-test-sftp#xxxxx.com
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: y.pem
debug3: sign_and_send_pubkey: RSA SHA256:1hkUUEqCCk5Uy9jTbQ9hclYk7swm+dODpWFE3FXecIc
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
debug3: failed to open file:C:/dev/tty error:3
debug1: read_passphrase: can't open /dev/tty: No such file or directory
rdg-test-fstp#xxxxx.com's password:
Connection closed

gerrit ssh permission denied (public key)

I am struggling to authenticate with gerrit.openbmc-project.xyz using ssh.
I am trying to use ssh on windows.
My .ssh\config looks like this:
Host openbmc.gerrit
Hostname gerrit.openbmc-project.xyz
Port 29418
User patelutpal
The contents of my *.ssh* directory look like this:
-rw-r--r-- 1 utpalp 1049089 107 Dec 7 23:04 config
-rw-r--r-- 1 utpalp 1049089 411 Dec 7 21:10 id_ed25519
-rw-r--r-- 1 utpalp 1049089 97 Dec 7 21:10 id_ed25519.pub
-rw-r--r-- 1 utpalp 1049089 3381 Dec 7 22:14 id_rsa
-rw-r--r-- 1 utpalp 1049089 741 Dec 7 22:14 id_rsa.pub
-rw-r--r-- 1 utpalp 1049089 419 Dec 7 22:53 known_hosts
I have logged into gerrit.openbmc-project.xyz with my github account with id patelutpal.
I have added id_rsa.pub ssh key to my settings in https://gerrit.openbmc-project.xyz/settings/#SSHKeys
When I try:
C:\Users\utpalp\git>ssh -p 29418 -vvvt patelutpal#gerrit.openbmc-project.xyz
I get the following output.
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug1: Reading configuration data C:\\Users\\utpalp/.ssh/config
debug3: Failed to open file:C:/ProgramData/ssh/ssh_config error:2
debug2: resolving "gerrit.openbmc-project.xyz" port 29418
debug2: ssh_connect_direct
debug1: Connecting to gerrit.openbmc-project.xyz [169.44.58.24] port 29418.
debug1: Connection established.
debug1: identity file C:\\Users\\utpalp/.ssh/id_rsa type 0
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_rsa-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_rsa-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_rsa-cert type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_dsa error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_dsa.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_dsa type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_dsa-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_dsa-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_dsa-cert type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ecdsa error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ecdsa.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_ecdsa type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ecdsa-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ecdsa-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\utpalp/.ssh/id_ed25519 type 3
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ed25519-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_ed25519-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_ed25519-cert type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_xmss error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_xmss.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_xmss type -1
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_xmss-cert error:2
debug3: Failed to open file:C:/Users/utpalp/.ssh/id_xmss-cert.pub error:2
debug1: identity file C:\\Users\\utpalp/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version GerritCodeReview_3.2.6 (APACHE-SSHD-2.4.0)
debug1: no match: GerritCodeReview_3.2.6 (APACHE-SSHD-2.4.0)
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gerrit.openbmc-project.xyz:29418 as 'patelutpal'
debug3: put_host_port: [gerrit.openbmc-project.xyz]:29418
debug3: hostkeys_foreach: reading file "C:\\Users\\utpalp/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file C:\\Users\\utpalp/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from [gerrit.openbmc-project.xyz]:29418
debug3: Failed to open file:C:/Users/utpalp/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256#libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01#openssh.com,ecdsa-sha2-nistp384-cert-v01#openssh.com,ecdsa-sha2-nistp521-cert-v01#openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01#openssh.com,rsa-sha2-512-cert-v01#openssh.com,rsa-sha2-256-cert-v01#openssh.com,ssh-rsa-cert-v01#openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: ciphers stoc: chacha20-poly1305#openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm#openssh.com,aes256-gcm#openssh.com
debug2: MACs ctos: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm#openssh.com,umac-128-etm#openssh.com,hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,umac-64#openssh.com,umac-128#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib#openssh.com,zlib
debug2: compression stoc: none,zlib#openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group18-sha512,diffie-hellman-group17-sha512,diffie-hellman-group16-sha512,diffie-hellman-group15-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-sha2-256-etm#openssh.com,hmac-sha2-512-etm#openssh.com,hmac-sha1-etm#openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256-etm#openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256-etm#openssh.com compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:LCaJB3JewPu5SBvlqsZfvKJD7yOVi7YaLWAEjsHSG60
debug3: put_host_port: [169.44.58.24]:29418
debug3: put_host_port: [gerrit.openbmc-project.xyz]:29418
debug3: hostkeys_foreach: reading file "C:\\Users\\utpalp/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file C:\\Users\\utpalp/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from [gerrit.openbmc-project.xyz]:29418
debug3: Failed to open file:C:/Users/utpalp/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug3: hostkeys_foreach: reading file "C:\\Users\\utpalp/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file C:\\Users\\utpalp/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from [169.44.58.24]:29418
debug3: Failed to open file:C:/Users/utpalp/.ssh/known_hosts2 error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts error:2
debug3: Failed to open file:C:/ProgramData/ssh/ssh_known_hosts2 error:2
debug1: Host '[gerrit.openbmc-project.xyz]:29418' is known and matches the ECDSA host key.
debug1: Found key in C:\\Users\\utpalp/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_rsa RSA SHA256:obA7xm+M+BuNoN/miSnhiIZmPbCIdpvMTj+xu8SRDsI
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_ed25519 ED25519 SHA256:zOx6/ktAHFTrf/yC4zBC55mEbfRZHF290Q29st34SA0
debug1: Will attempt key: C:\\Users\\utpalp/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: C:\\Users\\utpalp/.ssh/id_rsa RSA SHA256:obA7xm+M+BuNoN/miSnhiIZmPbCIdpvMTj+xu8SRDsI
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: C:\\Users\\utpalp/.ssh/id_dsa
debug3: no such identity: C:\\Users\\utpalp/.ssh/id_dsa: No such file or directory
debug1: Trying private key: C:\\Users\\utpalp/.ssh/id_ecdsa
debug3: no such identity: C:\\Users\\utpalp/.ssh/id_ecdsa: No such file or directory
debug1: Offering public key: C:\\Users\\utpalp/.ssh/id_ed25519 ED25519 SHA256:zOx6/ktAHFTrf/yC4zBC55mEbfRZHF290Q29st34SA0
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: C:\\Users\\utpalp/.ssh/id_xmss
debug3: no such identity: C:\\Users\\utpalp/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
patelutpal#gerrit.openbmc-project.xyz: Permission denied (publickey).
I dont know what to do. I think I have done all the right things. I wonder if this is a server side error. Can someone please help?
I will gladly provide more information if needed.
Thank you.
-Utpal

Context
OpenSSH 8.8 disables RSA signatures using the SHA-1 hash algorithm because they are considered cryptographically broken.
Git for Windows 2.33.1 upgrades OpenSSH to 8.8.
Note that this is a client side restriction.
Error
When trying to use a newer SSH client with git to clone or fetch using SSH, the following error might be observed:
git fetch
Fetching origin
dxxxxx#your.gerrit.host: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
error: Could not fetch origin
Done
Solution
You must provide a key type that is accepted by OpenSSH 8.8. OpenSSH recommends e.g. Ed25519:
ssh-keygen -t ed25519 -C "your_email#example.com"
copy content of ".ssh/id_ed25519.pub" to your Gerrit user settings
If your $HOME/.ssh/config file sets the IdentityFile option, then you may need to change the path accordingly, e.g.:
Host your.gerrit.host
IdentityFile ~/.ssh/id_ed25519
Alternatively you can configure the client to accept RSA by adding this ssh config option in ~/.ssh/config:
PubkeyAcceptedKeyTypes +ssh-rsa

I had a similar issue recently on Windows and GitBash. I downgraded to GitBash 2.27 and it worked again.

Passwordless access to raspberry pi via ssh

I can't setup passwordless access to Raspberry Pi. I am using similar instructions like this one. But it still asks for password when I
ssh to my raspberry. I made this with other servers and everything worked ok, but not with my raspberry (both of them).
Thanks to Jakuje, become a little bit clearer:
successful debug log:
debug1: Offering RSA public key: /home/tmp/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg ssh-rsa blen 535
debug2: input_userauth_pk_ok: fp SHA256:XPQA6tgcRCam/L18YP4ccPwsOsGtO65ippi7kua4mTM
debug3: sign_and_send_pubkey: RSA SHA256:XPQA6tgcRCam/L18YP4ccPwsOsGtO65ippi7kua4mTM
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
fail log:
debug1: Offering RSA public key: /home/tmp/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/tmp/.ssh/id_dsa
debug3: no such identity: /home/tmp/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/tmp/.ssh/id_ecdsa
debug3: no such identity: /home/tmp/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/tmp/.ssh/id_ed25519
debug3: no such identity: /home/tmp/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
Why it ignores id_rsa key and asks for id_dsa ???

debug1: Offering RSA public key: /home/tmp/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
says that the key was tried but not accepted by server. There will be a problem in server configuration/permissions. Running the server in debug mode identifies where the problem is. The owners and permissions on home directory, .ssh directory should be the following:
chmod g-w /home/your_user
chmod 700 /home/your_user/.ssh
chmod 600 /home/your_user/.ssh/authorized_keys