Unable to apply cert-manager free certificate from letsencrypt in kube - ssl

Question

I'm using kubernetes to host my app. I want automatically generate and renew lets-encrypt certificates using [cert-manager]
My project is open-source and all kubernetes configs are publicly available here. The domain I'm requesting a certificate is pychat.org and is managing my cloudflare. Kubernetes is set up already, the domain is pointing to the load-balancer ip address and returns index.html correctly.
So I'm following the guide:
Installing cert-manager CRD and cert-manager worker itself
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm template cert-manager jetstack/cert-manager --namespace cert-manager --version v1.8.0| kubectl apply -f -
Defining cloudflare-api key in cf-secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-token-secret
type: Opaque
stringData:
api-token: cf-token-copied-from-api
Defining issues and certificate in cert-manager.yaml:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: deathangel908#gmail.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
cloudflare:
email: deathangel908#gmail.com
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: pychat-domain
namespace: pychat
spec:
secretName: pychat-tls
issuerRef:
name: letsencrypt-prod
duration: 2160h # 90d
renewBefore: 720h # 30d before SSL will expire, renew it
dnsNames:
- "pychat.org"
- "*.pychat.org"
3 .If I check the certificate, it seems like generated correctly:
kubectl get secret pychat-tls -n default -o yaml
apiVersion: v1
data:
tls.crt: LS0tLS1CRUdJTiB...
tls.key: LS0tLS1CRUdJTiBSU0E...
metadata:
annotations:
cert-manager.io/alt-names: '*.pychat.org,pychat.org'
cert-manager.io/certificate-name: pychat-domain
cert-manager.io/common-name: pychat.org
cert-manager.io/ip-sans: ""
cert-manager.io/issuer-group: ""
cert-manager.io/issuer-kind: Issuer
cert-manager.io/issuer-name: letsencrypt-prod
cert-manager.io/uri-sans: ""
creationTimestamp: "2022-05-21T22:44:22Z"
name: pychat-tls
namespace: default
resourceVersion: "1800"
uid: f38c228b-b3a6-4649-aaf6-d9727685569c
type: kubernetes.io/tls
echo 'tls.crt...LS0tLS1CRUdJTiB' |base64 -d > lol.cert
openssl x509 -in ./lol.cert -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
04:b4:24:ae:61:c9:24:b6:50:5d:c2:50:0c:28:0f:c1:d5:17
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: May 21 21:46:14 2022 GMT
Not After : Aug 19 21:46:13 2022 GMT
Subject: CN = pychat.org
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:c7:7f:08:....
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
38:41:8D:F9:5A...
X509v3 Authority Key Identifier:
keyid:14:2E:B3..
Authority Information Access:
OCSP - URI:http://r3.o.lencr.org
CA Issuers - URI:http://r3.i.lencr.org/
X509v3 Subject Alternative Name:
DNS:*.pychat.org, DNS:pychat.org
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 46:A5:55:..
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:21:00:...
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 6F:53:76:A...
Timestamp : May 21 22:46:14.722 2022 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:...
Signature Algorithm: sha256WithRSAEncryption
6b:21:da:3a:ea:d8:...
If I check ingress, it shows this:
: kubectl describe ingress ingress
Name: ingress
Labels: <none>
Namespace: pychat
Address: 194.195.247.104
Default backend: frontend-service:80 (10.2.0.15:80)
TLS:
pychat-tls terminates pychat.org
Rules:
Host Path Backends
---- ---- --------
pychat.org
/api backend-service:8888 (10.2.0.16:8888,10.2.0.19:8888)
/ws backend-service:8888 (10.2.0.16:8888,10.2.0.19:8888)
/ frontend-service:80 (10.2.0.15:80)
Annotations: cert-manager.io/issuer: letsencrypt-prod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateCertificate 15m cert-manager-ingress-shim Successfully created Certificate "pychat-tls"
Normal Sync 15m (x2 over 15m) nginx-ingress-controller Scheduled for sync
But If I make a request, it returns self-signed kube default certificate:
: curl -vk https://pychat.org/ 2>&1 | grep -e subject: -e issuer:
* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
The server is hosted on Linode and uses its load balancer if it matters:
helm repo add stable https://charts.helm.sh/stable
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install nginx-ingress ingress-nginx/ingress-nginx
The ingress.yaml configuration looks like this:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/issuer: "letsencrypt-prod"
name: ingress
namespace: pychat
spec:
ingressClassName: nginx
tls:
- hosts:
- pychat.org
secretName: pychat-tls
defaultBackend:
service:
name: frontend-service
port:
number: 80
rules:
- host: pychat.org
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: backend-service
port:
number: 8888
- path: /ws
pathType: Prefix
backend:
service:
name: backend-service
port:
number: 8888
- path: /
pathType: Prefix
backend:
service:
name: frontend-service
port:
number: 80

Answer 1

Found the issue, the guides just don't tell you much about if:
All the resources you create should be in the same namespace as your server. secret cloud-flare-api-token-secret, lets-encrypt-prod issuer, pychat-domain certificate, all of these things should have metadata -> namespace of your server, rather than cert-manager or default one.
Further tip: to debug this issue, I took a look at pods logs:
kubectl get all -n cert-manager
kubectl logs pod/cert-manager-id-from-top -n cert-manager