debugging cert-manager certificate creation failure on AKS - ssl

Question

I'm deploying cert-manager on Azure AKS and trying to have it request a Let's Encrypt certificate. It fails with certificate signed by unknown authority error and I have problem troubleshooting it further.
Not sure whether this is a problem with trusting LE server, a tunnelfront pod, or maybe an internal AKS self-generated CA. So my questions would be:
how to force cert-manager to debug (display more info) regarding the certificate it does not trust?
maybe the problem is occuring regularly and there is a known solution?
what steps should be undertaken to debug the issue further?
I have created an issue on jetstack/cert-manager's Github page, but was not answered, so I came here.
The whole story is as follows:
Certificates are not created. The following errors are reported:
the certificate:
Error from server: conversion webhook for &{map[apiVersion:cert-manager.io/v1alpha2 kind:Certificate metadata:map[creationTimestamp:2020-05-13T17:30:48Z generation:1 name:xxx-tls namespace:test ownerReferences:[map[apiVersion:extensions/v1beta1 blockOwnerDeletion:true controller:true kind:Ingress name:xxx-ingress uid:6d73b182-bbce-4834-aee2-414d2b3aa802]] uid:d40bc037-aef7-4139-868f-bd615a423b38] spec:map[dnsNames:[xxx.test.domain.com] issuerRef:map[group:cert-manager.io kind:ClusterIssuer name:letsencrypt-prod] secretName:xxx-tls] status:map[conditions:[map[lastTransitionTime:2020-05-13T18:55:31Z message:Waiting for CertificateRequest "xxx-tls-1403681706" to complete reason:InProgress status:False type:Ready]]]]} failed: Post https://cert-manager-webhook.cert-manager.svc:443/convert?timeout=30s: x509: certificate signed by unknown authority
cert-manager-webhook container:
cert-manager 2020/05/15 14:22:58 http: TLS handshake error from 10.20.0.19:35350: remote error: tls: bad certificate
Where 10.20.0.19 is the IP of tunnelfront pod.
Debugging with https://cert-manager.io/docs/faq/acme/ sort of "fails" when trying to kubectl describe order... as kubectl describe certificaterequest... returns CSR contents with error (as above), but not the order ID.
Environment details:
Kubernetes version: 1.15.10
Cloud-provider/provisioner : Azure (AKS)
cert-manager version: 0.14.3
Install method: static manifests (see below) + cluster issuer (see below) + regular CRDs (not legacy)
cluster issuer:
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: x
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- dns01:
azuredns:
clientID: x
clientSecretSecretRef:
name: cert-manager-stage
key: CLIENT_SECRET
subscriptionID: x
tenantID: x
resourceGroupName: dns-stage
hostedZoneName: x
the manifest:
imagePullSecrets: []
isOpenshift: false
priorityClassName: ""
rbac:
create: true
podSecurityPolicy:
enabled: false
logLevel: 2
leaderElection:
namespace: "kube-system"
replicaCount: 1
strategy: {}
image:
repository: quay.io/jetstack/cert-manager-controller
pullPolicy: IfNotPresent
tag: v0.14.3
clusterResourceNamespace: ""
serviceAccount:
create: true
name:
annotations: {}
extraArgs: []
extraEnv: []
resources: {}
securityContext:
enabled: false
fsGroup: 1001
runAsUser: 1001
podAnnotations: {}
podLabels: {}
nodeSelector: {}
ingressShim:
defaultIssuerName: letsencrypt-prod
defaultIssuerKind: ClusterIssuer
prometheus:
enabled: true
servicemonitor:
enabled: false
prometheusInstance: default
targetPort: 9402
path: /metrics
interval: 60s
scrapeTimeout: 30s
labels: {}
affinity: {}
tolerations: []
webhook:
enabled: true
replicaCount: 1
strategy: {}
podAnnotations: {}
extraArgs: []
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
image:
repository: quay.io/jetstack/cert-manager-webhook
pullPolicy: IfNotPresent
tag: v0.14.3
injectAPIServerCA: true
securePort: 10250
cainjector:
replicaCount: 1
strategy: {}
podAnnotations: {}
extraArgs: []
resources: {}
nodeSelector: {}
affinity: {}
tolerations: []
image:
repository: quay.io/jetstack/cert-manager-cainjector
pullPolicy: IfNotPresent
tag: v0.14.3

Answer 1

Seems that v0.14.3 had a bug of some sort. The problem does not occur for v0.15.0.