github actions: SSH into droplet and run code - ssh

Question

I want to deploy a github project automatically through github actions when I push my code to github. My yaml-file looks like this:
name: push-and-deploy-to-server
on:
push:
branches: [ main ]
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: appleboy/scp-action#master
with:
host: ${{ secrets.SSH_HOST }}
port: 22
username: ${{ secrets.SSH_USERNAME }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
source: "."
target: "."
- uses: appleboy/ssh-action#master
with:
host: ${{ secrets.SSH_HOST }}
port: 22
username: ${{ secrets.SSH_USERNAME }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
script: |
npm install
pm2 restart index.js
I have a server with an SSH keypair. The public key is added to the server authorized_keys, and I can SSH through my terminal to the server.
When I push code to the github repo, the action runs. I get the following error:
drone-scp error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
The weird thing is: after this error, I'm not able to SSH into my server anymore, even through my console I get a "Permission denied (publickey)". So before running the github action, everything works, after that it fails.
The ip address of the server is SSH_HOST, the username which I use to SSH into the server is set in SSH_USERNAME and the private key (the same as I use on my local laptop to ssh into the server) is set in SSH_PRIVATE_KEY.
Does anyone have encountered the same problem before? I have really no clue whats going on here.
Edit: extra information: it's a private repository.