Is there a way to monitor tls certificates in kubernetes using prometheus? - ssl

Question

I want to monitor my tls certificates in Kubernetes using Prometheus and get a dashboard in grafana. I want to monitor their expiry and would want to get an alert when the certificates are going to be expired in 30 days. I did a lot of research and I finally found https://github.com/enix/x509-exporter. How do I use it? Is there any other efficient way to monitor the expiry of the certificates?

Answer 1

DISCLAIMER: I haven't tried this x509-exporter. Just giving suggestion as per my understanding.
The README file seems bit off. The first thing you need to do is create a github issue, no worries I raised one here.
I am listing down steps as per my understanding and referring the usage section.
Use their official docker image and deploy it as a deployment on k8s.
Check sample k8s yaml files for creating deployment. Also note that the deployment yaml should mount a host directory where all the k8s certificates are stored.
As per documentation, usually the certificates are located at /etc/kubernetes/pki.
The deployment yaml should contain a command where you point the exporter to the directory where certificates are located with other necessary options. Like this
command: ["x509-exporter"]
args: ["-d", "/etc/kubernetes/pki", "-p", "8091", "--debug"]
Note: Here I am running exporter in debug mode on port 8091, remember to expose this port.
In prometheus config, add the x509-exporter endpoint as target to scrape the metrics and plot those by creating graphs in Grafana dashboard.

Answer 2

Another way is to install the x509-exporter using the helm chart : https://hub.helm.sh/charts/enix/x509-exporter
See documentation here https://github.com/enix/helm-charts/tree/master/charts/x509-exporter.
You might also find the following prometheus alert rules useful (based on the x509-exporter metrics):
check-kubernetes-certificate.rules.yml :
groups:
- name: check-kubernetes-certificate-expiration.rules
rules:
- alert: KubernetesCertificateExpiration
expr: floor((x509_cert_not_after - time()) / 86400) < 90
for: 5m
labels:
severity: warning
annotations:
summary: 'Certificate expiration on `{{ $labels.nb_cluster }}`'
description: 'Certificate `{{ $labels.subject_CN }}` will expire in {{ $value }} days on `{{ $labels.nb_cluster }}`'
- alert: KubernetesCertificateExpirationCritical
expr: floor((x509_cert_not_after - time()) / 86400) < 10
for: 5m
labels:
severity: critical
annotations:
summary: 'Certificate expiration on `{{ $labels.nb_cluster }}`'
description: 'Certificate `{{ $labels.subject_CN }}` will expire in {{ $value }} days on `{{ $labels.nb_cluster }}`'
- alert: KubeletCertificateEmbedded
expr: x509_cert_not_after{filename="kubelet.conf", embedded_kind="user"}
for: 5m
labels:
severity: warning
annotations:
summary: '{{ $labels.instance }}: Embedded certificate in {{ $labels.filename }}'
description: '{{ $labels.nb_cluster }} has kubelet {{ $labels.subject_CN }} running with an embedded certificate in {{ $labels.filepath }}'

Answer 3

The official prometheus/blackbox_exporter have the ssl cert expiry info already.
Name: "probe_ssl_earliest_cert_expiry",
Help: "Returns earliest SSL cert expiry date",
So all you need is:
Setup blackbox_exporter and the Probe rules to the domain you want to monitor.
You can check my project kehao95/helm-prometheus-exporter to install blackbox_exporter via helm chart.
config rule to monitor certificate expiring.
You can config your prometheusRule like this: (assuming you're using prometheus-operator)
rules:
- alert: TLS certificate expiring
expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 45
labels:
severity: warning
- alert: TLS certificate expiring
expr: (probe_ssl_earliest_cert_expiry - time())/86400 < 30
labels:
severity: critical