Fix JWT::DecodeError (Nil JSON web token) for Gitlab runner / CI job? - gitlab-ci

Question

I am trying to install a simple job in Gitlab-CI. It should just copy most of the files (.php) in the repository to a different folder (from repository to Apache's /var/www/html/...).
The output goes like this:
Using Shell executor...
Preparing environment
Running on my.server...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialisierte existierendes Git-Projektarchiv in /home/gitlab-runner/builds/TyGXtawE/0/sdp-dev/ibo-php/.git/
fatal: unable to access 'https://gitlab-ci-token:[MASKED]#gitlab.sustainable-data-platform.org/sdp-dev/ibo-php.git/':
The requested URL returned error: 500
So I can find the corresponding problem in /var/log/gitlab/gitlab-rails/production.log
Started POST "/api/v4/jobs/request" for ip.ip.ip.ip at 2022-04-26 14:27:41 +0200
Started GET "/sdp-dev/ibo-php.git/info/refs?service=git-upload-pack" for ip.ip.ip.ip at 2022-04-26 14:27:44 +0200
Processing by Repositories::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "repository_path"=>"sdp-dev/ibo-php.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 14ms (Views: 1.3ms | ActiveRecord: 1.5ms | Elasticsearch: 0.0ms | Allocations: 4656)
Started GET "/sdp-dev/ibo-php.git/info/refs?service=git-upload-pack" for ip.ip.ip.ip at 2022-04-26 14:27:44 +0200
Processing by Repositories::GitHttpController#info_refs as */*
Parameters: {"service"=>"git-upload-pack", "repository_path"=>"sdp-dev/ibo-php.git"}
Completed 500 Internal Server Error in 34ms (ActiveRecord: 7.6ms | Elasticsearch: 0.0ms | Allocations: 7564)
JWT::DecodeError (Nil JSON web token):
lib/gitlab/jwt_authenticatable.rb:17:in `decode_jwt_for_issuer'
lib/gitlab/workhorse.rb:210:in `decode_jwt'
# ... stacktrace
I have installed the runner from "https://gitlab-runner-downloads.s3.amazonaws.com/latest/rpm/gitlab-runner_amd64.rpm" and gave it the token from Gitlab's GUI.
Gitlab (v 14.5.2) and the runner (v 14.10.0) are running on the same machine.
All this might be overhead for some cp -r, but I would like to understand the error for upcoming more complex jobs. Any idea how to debug the registration-token, how to find the problem?
Update
403 for Gitlab Runner from Gitlab?
It seems like gitlab-runner just gets a 403 (forbidden) from Gitlab. So I might need to change something in the corresponding (Apache). But how can I debug this (ERROR: Checking for jobs... forbidden)?
How to register a runner?
There is this command: sudo gitlab-runner register --url $URL --registration-token $REGISTRATION_TOKEN. But when I put the right data (URL, REGISTRATION_TOKEN) in this command (not using bash variables!), I am asked for the same data again (URL, TOKEN) and I just do c/p. E.g. Type the URL [MY_URL]:, then I c/p MY_URL and hit enter. Is this OK?
token / unhealthy runner problems
No success after creating a new token in gitlab and registering a new runner, also after updating Gitlab to v14.10.
But I can see some problems in /var/log/messages.
systemd: Started GitLab Runner.
gitlab-runner: Runtime platform #033[0;m arch#033[0;m=amd64 os#033[0;m=linux pid#033[0;m=20848 revision#033[0;m=c6bb62f6 version#033[0;m=14.10.0
gitlab-runner: Starting multi-runner from /etc/gitlab-runner/config.toml...#033[0;m builds#033[0;m=0
gitlab-runner: Running in system-mode. #033[0;m
gitlab-runner: #033[0;m
gitlab-runner: Configuration loaded #033[0;m builds#033[0;m=0
gitlab-runner: listen_address not defined, metrics & debug endpoints disabled#033[0;m builds#033[0;m=0
gitlab-runner: [session_server].listen_address not defined, session endpoints disabled#033[0;m builds#033[0;m=0
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=TyGXtawE
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=iU4sqj-p
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=TyGXtawE
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=iU4sqj-p
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=TyGXtawE
gitlab-runner: #033[31;1mERROR: Runner {URL}/{token} is not healthy and will be disabled!#033[0;m
gitlab-runner: #033[31;1mERROR: Checking for jobs... forbidden #033[0;m #033[31;1mrunner#033[0;m=iU4sqj-p
gitlab-runner: #033[31;1mERROR: Runner {URL}/{token} is not healthy and will be disabled!#033[0;m
I am interested in listen_address not defined, metrics & debug endpoints disabled and Checking for jobs... forbidden.
You can see this problem elsewhere in SO, but I could not find a solution (googling etc.).
Here? https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28965

Answer 1

Some things you might try:
Create a new token
Update Gitlab