TLS certification expired only for some users - ssl

Question

I have a k8s cluster with an ingress nginx as a reverse proxy. I am using letsencrypt to generate TLS certificate
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: ******
privateKeySecretRef:
name: letsencrypt
solvers:
- http01:
ingress:
class: nginx
Everything worked fine for months. Today,
$ curl -v --verbose https://myurl
returns
* Rebuilt URL to: https://myurl/
* Trying 51.103.58.**...
* TCP_NODELAY set
* Connected to myurl (51.103.58.**) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: certificate has expired
* stopped the pause stream!
* Closing connection 0
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.
For 2 other people on my team, error is the same and I have the same error when I use Postman (expired certificate).
But for another one, we get no error :
* Trying 51.103.58.**...
* TCP_NODELAY set
* Connected to myurl (51.103.58.**) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=myurl
* start date: Jul 24 07:15:13 2021 GMT
* expire date: Oct 22 07:15:11 2021 GMT
* subjectAltName: host "myurl" matched cert's "myurl"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd9be00d600)
> GET / HTTP/2
> Host: myurl
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx/1.19.1
< date: Thu, 30 Sep 2021 16:11:23 GMT
< content-type: application/json; charset=utf-8
< content-length: 56
< vary: Origin, Accept-Encoding
< access-control-allow-credentials: true
< x-xss-protection: 1; mode=block
< x-frame-options: DENY
< strict-transport-security: max-age=15724800; includeSubDomains
< x-download-options: noopen
< x-content-type-options: nosniff
< etag: W/"38-3eQD3G7Y0vTkrLR+ExD2u5BSsMc"
<
* Connection #0 to host myurl left intact
{"started":"2021-09-30T13:30:30.912Z","uptime":9653.048}* Closing connection 0
When I use my web browser to go to the website, everything works fine and the certificate is presented as valid and for now, I get no error in prod or staging environment. (same error on staging)
Has anyone an explanation on this ?

Answer 1

Warning! Please plan OS upgrade path. The below advice should be applied only in emergency situation to quickly fix a critical system.
Your team missed OS update or ca-certificates package update.
Below solution works on old Debian/Ubuntu systems.
First check if you have offending DST Root CA X3 cert present:
# grep X3 /etc/ca-certificates.conf
mozilla/DST_Root_CA_X3.crt
Make sure the client OS have the proper ISRG Root X1 present too:
# grep X1 /etc/ca-certificates.conf
mozilla/ISRG_Root_X1.crt
This is going to disable X3:
# sed -i '/^mozilla\/DST_Root_CA_X3/s/^/!/' /etc/ca-certificates.conf && update-ca-certificates -f
Try curl https://yourdomain now, should pass.
Again, plan an upgrade please.

Answer 2

This is related to the expired DST Root CA X3, which expired Sep 30 14:01:15 2021 GMT.
The DST CA Root X3 certificate is part of the "cacert-bundle".
As of today the "cacert-bundle" can be found here: https://curl.se/docs/caextract.html
as part of the bundle https://curl.se/ca/cacert.pem.
The expired certificate is:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
Signature Algorithm: sha1WithRSAEncryption
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Validity
Not Before: Sep 30 21:12:19 2000 GMT
Not After : Sep 30 14:01:15 2021 GMT
Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Which is used to verify peer in curl calls to websites using Let's Encrypt issued certificates.
Here's a detailed solution to your problem: https://stackoverflow.com/a/69411107/1549092
Let's Encrypt formal address of the issue can be found here: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Answer 3

Even if not K8S related, main explanation is contained in :
Sudden OpenSSL Error messages: error:14090086 using file_get_contents . I complete it with K8S related here.
I fixed the same issue by upgarding my certbot and reissuing certificate with `` --preferred-chain 'ISRG Root X1'
you can do the same with options in the yaml of the cert issuer :
see here : https://cert-manager.io/docs/configuration/acme/#use-an-alternative-certificate-chain
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
preferredChain: "ISRG Root X1"