Let's Encrypt Certificate Issuance - ssl

Question

I am trying to get a certificate issued from Let's Encrypt, and it has been 3 and a half hours.
I accidentally originally set my secretName as "echo-tls" before switching it to the correct "pandaist-tls" that I want to use instead.
I currently have this:
kubectl get CertificateRequest -o wide
NAME READY ISSUER STATUS AGE
pandaist-tls-1926992011 False letsencrypt-prod Waiting on certificate issuance from order default/pandaist-tls-1926992011-2163900139: "pending" 3h26m
When I describe the certificate, I get this:
Deployment kubectl describe CertificateRequest pandaist-tls-1926992011
Name: pandaist-tls-1926992011
Namespace: default
Labels: <none>
Annotations: cert-manager.io/certificate-name: pandaist-tls
cert-manager.io/private-key-secret-name: pandaist-tls
API Version: cert-manager.io/v1alpha2
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2020-04-07T15:41:13Z
Generation: 1
Owner References:
API Version: cert-manager.io/v1alpha2
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: pandaist-tls
UID: 25c3ff31-447f-4abf-a23e-ec48f5a591a9
Resource Version: 500795
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/default/certificaterequests/pandaist-tls-1926992011
UID: 8295836d-fb99-4ebf-8803-a344d6edb574
Spec:
Csr: ABUNCHOFVALUESTHATIWILLNOTDESCRIBE
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt-prod
Status:
Conditions:
Last Transition Time: 2020-04-07T15:41:13Z
Message: Waiting on certificate issuance from order default/pandaist-tls-1926992011-2163900139: "pending"
Reason: Pending
Status: False
Type: Ready
Events: <none>
And then I look at my logs for my cert-manager pods - here are small slices of each:
I0407 19:01:35.499469 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="pandaist.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-2fp58" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01"
I0407 19:01:35.499513 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="auth.pandaist.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-xhjsr" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01"
I0407 19:01:35.499534 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="pandaist.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-pd9fh" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01"
I0407 19:01:35.499578 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="auth.pandaist.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-b6zr2" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01"
E0407 19:03:46.571074 1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://pandaist.com/.well-known/acme-challenge/6Wduj2Ejr59OZ9SFy_Rw4jnozE50xspK-a5OIvCwYsc': Get http://pandaist.com/.well-known/acme-challenge/6Wduj2Ejr59OZ9SFy_Rw4jnozE50xspK-a5OIvCwYsc: dial tcp 178.128.132.218:80: connect: connection timed out" "dnsName"="pandaist.com" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01"
E0407 19:03:46.571109 1 sync.go:184] cert-manager/controller/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://auth.pandaist.com/.well-known/acme-challenge/gO91--fK0SGG15aS3ALOHXXYtCSly2Q9pbVO8OJW2aE': Get http://auth.pandaist.com/.well-known/acme-challenge/gO91--fK0SGG15aS3ALOHXXYtCSly2Q9pbVO8OJW2aE: dial tcp 178.128.132.218:80: connect: connection timed out" "dnsName"="auth.pandaist.com" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01"
I0407 19:03:46.571382 1 controller.go:135] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="default/pandaist-tls-1926992011-2163900139-832917849"
I0407 19:03:46.571528 1 controller.go:129] cert-manager/controller/challenges "level"=0 "msg"="syncing item" "key"="default/pandaist-tls-1926992011-2163900139-832917849"
I0407 19:03:46.571193 1 controller.go:135] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="default/pandaist-tls-1926992011-2163900139-2157075729"
I0407 19:03:46.572009 1 controller.go:129] cert-manager/controller/challenges "level"=0 "msg"="syncing item" "key"="default/pandaist-tls-1926992011-2163900139-2157075729"
I0407 19:03:46.572338 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "level"=0 "msg"="found one existing HTTP01 solver pod" "dnsName"="auth.pandaist.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-scqtx" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01"
I0407 19:03:46.572600 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="auth.pandaist.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-xhjsr" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01"
I0407 19:03:46.572860 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="auth.pandaist.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-b6zr2" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-832917849" "resource_namespace"="default" "type"="http-01"
I0407 19:03:46.573128 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "level"=0 "msg"="found one existing HTTP01 solver pod" "dnsName"="pandaist.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-jn65v" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01"
I0407 19:03:46.573433 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="pandaist.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-2fp58" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01"
I0407 19:03:46.573749 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="pandaist.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-pd9fh" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="pandaist-tls-1926992011-2163900139-2157075729" "resource_namespace"="default" "type"="http-01"
And then here, where I still see echo-tls, despite the fact that I changed my ingress to use pandaist-tls:
I0407 15:34:37.115159 1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled" "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0407 15:34:37.118246 1 controller.go:170] cert-manager/inject-controller "level"=1 "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"=""
I0407 15:34:37.118520 1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled" "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0407 15:34:37.119415 1 sources.go:176] cert-manager/inject-controller "level"=0 "msg"="Extracting CA from Secret resource" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-tls"
I0407 15:34:37.120959 1 controller.go:170] cert-manager/inject-controller "level"=1 "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"=""
I0407 15:34:37.121399 1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled" "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0407 15:34:37.124545 1 controller.go:170] cert-manager/inject-controller "level"=1 "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"=""
I0407 15:34:37.125160 1 controller.go:242] cert-manager/controller-runtime/controller "level"=1 "msg"="Successfully Reconciled" "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
E0407 16:19:36.762436 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"echo-tls\" not found" "certificate"={"Namespace":"default","Name":"echo-tls"} "secret"={"Namespace":"default","Name":"echo-tls"}
E0407 16:19:36.762573 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"echo-tls\" not found" "certificate"={"Namespace":"default","Name":"echo-tls"} "secret"={"Namespace":"default","Name":"echo-tls"}
E0407 16:19:36.762753 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"echo-tls\" not found" "certificate"={"Namespace":"default","Name":"echo-tls"} "secret"={"Namespace":"default","Name":"echo-tls"}
E0407 16:19:36.762766 1 indexers.go:93] cert-manager/secret-for-certificate-mapper "msg"="unable to fetch certificate that owns the secret" "error"="Certificate.cert-manager.io \"echo-tls\" not found" "certificate"={"Namespace":"default","Name":"echo-tls"} "secret"={"Namespace":"default","Name":"echo-tls"}
My ingress:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: pandaist-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- pandaist.com
- auth.pandaist.com
secretName: pandaist-tls
rules:
- host: pandaist.com
http:
paths:
- backend:
serviceName: pandaist-main
servicePort: 80
- host: auth.pandaist.com
http:
paths:
- backend:
serviceName: pandaist-keycloak
servicePort: 80
This ingress was absolutely applied after the echo one.
Is this just normal certificate approval time (3.5 hours) or did the accidental inclusion of echo-tls mess up my certificate issuance? If so, how do I fix it?

Answer 1

Due to a bug in how load balancers work on Digital Ocean:
https://www.digitalocean.com/community/questions/how-do-i-correct-a-connection-timed-out-error-during-http-01-challenge-propagation-with-cert-manager
This will solve the problem:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
annotations:
# See https://github.com/digitalocean/digitalocean-cloud-controller-manager/blob/master/docs/controllers/services/examples/README.md#accessing-pods-over-a-managed-load-balancer-from-inside-the-cluster
service.beta.kubernetes.io/do-loadbalancer-hostname: "kube.mydomain.com"
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https

Answer 2

This might be worthwhile to look at in case #Cecil's solution doesn't work. I was facing similar issue with Connection Timeout
Change LoadBalancer in ingress-nginx service.
Add/Change externalTrafficPolicy: Cluster.
Reason being, pod with the certificate-issuer wound up on a different node than the load balancer did, so it couldn’t talk to itself through the ingress.
Below is complete block taken from https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
#CHANGE/ADD THIS
externalTrafficPolicy: Cluster
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
---