x509: certificate signed by unknown authority metrics-server - ssl

Question

I am new to kubernetes and I finally realized how to launch the metrics-server as documented kubernetes-sigs/metrics-server. In case that someone else wonders you need to deploy on Master node and also have minimum one worker in the cluster.
So I get this error:
E0818 15:25:22.835094 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:<hostname-master>: unable to fetch metrics from Kubelet <hostname-master> (<hostname-master>): Get https://<hostname-master>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:<hostname-worker>: unable to fetch metrics from Kubelet <hostname-worker> (<hostname-worker>): Get https://<hostname-worker>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]
I am using my own CAs (not self signed) and I have modified the components.yml file (sample):
args:
- --cert-dir=/tmp/metricsServerCas
- --secure-port=4443
- --kubelet-preferred-address-types=Hostname
I know that I can disable the tls by using this flag --kubelet-insecure-tls I have already tried it. I want to use my own CAs for extra security.
I have see other many relevant questions (few samples) e.g.:
x509 certificate signed by unknown authority- Kubernetes and kubectl unable to connect to server: x509: certificate signed by unknown authority
Although that I have applied chown already my $HOME/.kube/config still I see this error.
Where am I going wrong?
Update: On the worker I am creating a directory e.g. /tmp/ca and I add the ca file(s) in the directory.
I am not really good yet with the mounting points and I assume that I am doing something wrong. The default syntax of the images can be found here kubernetes-sigs/metrics-server/v0.3.7 (see components.yml file).
I tried to create a directory on my worker e.g. /tmp/ca and I modified the flag --cert-dir=/tmp/ca and mountPath: /tmp/ca
When I am deploying the file e.g.:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml
I keep getting the error from the metrics-server-xxxx:
panic: open /tmp/client-ca-file805316981: read-only file system
Although I have given full access to the directory e.g.:
$ ls -la /tmp/ca
total 8
drwxr-xr-x. 2 user user 20 Aug 19 16:59 .
drwxrwxrwt. 18 root root 4096 Aug 19 17:34 ..
-rwxr-xr-x. 1 user user 1025 Aug 19 16:59 ca.crt
I am not sure where I am going wrong.
How is meant to be configured so someone can use non self signed certificates? I can see that most people are using non SSL which I would like to avoid.
Sample of my args in the image:
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp/ca
- --secure-port=4443
- --kubelet-preferred-address-types=Hostname
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- name: tmp-dir
mountPath: /tmp/ca
nodeSelector:
kubernetes.io/os: linux
kubernetes.io/arch: "amd64"
Update 2: Adding curl command from Master to Worker including error output:
$ curl --cacert /etc/kubernetes/pki/ca.crt https://node_hostname:10250/stats/summary?only_cpu_and_memory=true
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

Answer 1

Posting this answer as a community wiki to give better visibility as the solution was posted in the comments.
The version that I used before was 1.18.2 and metrics server v0.3.6. Deployment was through kubeadm. Yes all requirements was exactly as the metrics-server/requirements. The good news is that I got it running by upgrading my k8s version on 1.19.0 and using the latest version v0.3.7. It works with self signed certificates.
The issue was resolved by upgrading:
Kubernetes: 1.18.2 -> 1.19.0
Metrics-server: 0.3.6 -> 0.3.7
This upgrade allowed to run metrics-server with tls enabled (self-signed certificates).
Additional resources that could help when deploying metrics-server with tls:
Github.com: Kubernetes-sigs: Metrics-server: FAQ: How to run metrics-server-securely
How to run metrics-server securely?
Suggested configuration:
Cluster with RBAC enabled
Kubelet read-only port port disabled
Validate kubelet certificate by mounting CA file and providing --kubelet-certificate-authority flag to metrics server
Avoid passing insecure flags to metrics server (--deprecated-kubelet-completely-insecure, --kubelet-insecure-tls)
Consider using your own certificates (--tls-cert-file, --tls-private-key-file)
Github.com: Metrics-server: x509: certificate signed by unknown authority
Ftclausen.github.io: Setting up K8S with metrics-server

Answer 2

Create a configmap to store the ca certificate which was used to generate kubelet serving certificate.
kubectl -n kube-system create configmap ca --from-file=ca.crt=/etc/kubernetes/pki/ca.crt -o yaml
Then use volumeMounts to use it in metrics server pod
spec:
volumes:
- emptyDir: {}
name: tmp-dir
- configMap:
defaultMode: 420
name: ca
name: ca-dir
containers:
args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-certificate-authority=/ca/ca.crt
- --kubelet-preferred-address-types=Hostname
volumeMounts:
- mountPath: /tmp
name: tmp-dir
- mountPath: /ca
name: ca-dir
You can follow the same approach and use --tls-cert-file and --tls-private-key-file for using your own certificate instead of self signed certificate.

Answer 3

For my friends on EKS make sure you have the username set (and not set to just the session name like I did):
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123546789012:role/MyTeamRole
username: {{SessionName}}
groups:
- system:masters
robert ❱ kubectl top node
error: You must be logged in to the server (Unauthorized)
robert ❱ 1 ❱ kubectl edit configmap -n kube-system aws-auth
configmap/aws-auth edited
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123546789012:role/MyTeamRole
username: literally_anything:{{SessionName}}
groups:
- system:masters
robert ❱ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-10-0-3-103.us-west-2.compute.internal 341m 17% 1738Mi 52%
...
robert ❱ kubectl logs -n kube-system -l app.kubernetes.io/instance=metrics-server
E0407 22:34:45.879156 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:49.399854 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:50.691133 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:51.827629 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:07.288163 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:08.755492 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:09.801957 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:40:32.405458 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:43:09.791769 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:44:14.244221 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
robert ❱