I have been trying to genereate an SSL certificate from Let's Encrypt to my domain but cert-manager creates the secret and adds a random suffix, causing the NGINX ingress controller to not be able to find it:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/issuer: scaleway
kubernetes.io/tls-acme: "true"
spec:
rules:
- host: <DOMAIN>
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-svc
port:
number: 80
tls:
- hosts:
- <DOMAIN>
secretName: tls-cert
Here is the error in the NGINX logs :
Error obtaining X.509 certificate: no object matching key "default/tls-cert" in local store
Error getting SSL certificate "default/storek8s.igesa.it": local SSL certificate default/storek8s.igesa.it was not found. Using default certificate
The secret is always created like this :
NAME TYPE DATA AGE
...
tls-secret-fjrm5 Opaque 1 60m
How do I disable adding the random suffix for the secret generated by Ingress ? Is there any workaround to solve this ?
At the moment I have too little information to give an exact answer. Are you for example using this Helm chart, what values does your values.yaml contain, and how did you setup your issuer?
Also, I know that tls secrets created by cert-manager should be of type kubernetes.io/tls and not Opaque as is the case in your situation. So there is definitly something going wrong there.
I'm trying to setup TLS for a service that's available outside a Kubernetes cluster (AWS EKS). With cert-manager, I've successfully issued a certificate and configured ingress, but I'm still getting error NET::ERR_CERT_AUTHORITY_INVALID. Here's what I have:
namespace tests and hello-kubernetes in it (both deployment and service have name hello-kubernetes-first, serivce is ClusterIP with port 80 and targetPort 8080, deployment is based on paulbouwer/hello-kubernetes:1.8, see details in my previous question)
DNS and ingress configured to show the service:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-kubernetes-ingress
namespace: tests
spec:
ingressClassName: nginx
rules:
- host: test3.projectname.org
http:
paths:
- path: "/"
pathType: Prefix
backend:
service:
name: hello-kubernetes-first
port:
number: 80
Without configuring TLS, I can access test3.projectname.org via http and see the service (well, it tries to redirect me to https, I see NET::ERR_CERT_AUTHORITY_INVALID, I go to insecure anyway and see the hello-kubernetes page).
note: I have nginx-ingress ingress controller; it was installed before me via the following chart:
apiVersion: v2
name: nginx
description: A Helm chart for Kubernetes
type: application
version: 4.0.6
appVersion: "1.0.4"
dependencies:
- name: ingress-nginx
version: 4.0.6
repository: https://kubernetes.github.io/ingress-nginx
and the values overwrites applied with the chart differ from the original ones mostly in extraArgs: default-ssl-certificate: "nginx-ingress/dragon-family-com" is uncommneted
cert-manager installed via kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml
ClusterIssuer created with the following config:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-backoffice
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
# use https://acme-v02.api.letsencrypt.org/directory after everything is fixed and works
privateKeySecretRef: # this secret is created in the namespace of cert-manager
name: letsencrypt-backoffice-private-key
# email: <will be used for urgent alerts about expiration etc>
solvers:
# TODO: add for each domain/second-level domain/*.projectname.org
- selector:
dnsZones:
- test.projectname.org
- test2.projectname.org
- test3.projectname.org
http01:
ingress:
class: nginx
certificate in the tests namespace. It's config is
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: letsencrypt-certificate-31
namespace: tests
spec:
secretName: tls-secret-31
issuerRef:
kind: ClusterIssuer
name: letsencrypt-backoffice
commonName: test3.projectname.org
dnsNames:
- test3.projectname.org
Now, certificate is ready (kubectl get certificates -n tests tells that) and to apply it, I add this to ingress's spec:
tls:
- hosts:
- test3.projectname.org
secretName: tls-secret-31
However, when I try to open test3.projectname.org via https, it still shows me the NET::ERR_CERT_AUTHORITY_INVALID error. What am I doing wrong? How to debug this? I've checked up openssl s_client -connect test3.projectname.org:443 -prexit* and it shows the following chain:
0 s:CN = test3.projectname.org
i:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
1 s:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
2 s:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3
and tells, among other output
Verification error: unable to get local issuer certificate
Unfortunately, I haven't found anything useful to try further, so any help is appreciated.
Your ClusterIssuer refers to LetsEncrypt staging issuer. Remove that setting / the default should use their production setup. As pointed out in comments: https://acme-v02.api.letsencrypt.org/directory
Deleting the previously generated secrets or switching to new secrets should ensure your certificates would be re-generated, using the right issuer.
The staging issuer could be useful testing LetsEncrypt integration, it shouldn't be used otherwise.
Following the suggestion from SYN, I've fixed this by
switching ACME server in ClusterIssuer config from https://acme-staging-v02.api.letsencrypt.org/directory to https://acme-v02.api.letsencrypt.org/directory. The idea of the staging server seems to be: allow to debug certificate issuing (so that kubectl get certificate [-n <namespace>] shows that READY = true) without providing actual trusted certificates; after certificate issuing is ok, one has to switch to the main server to get production certificates.
Updating certificates, tls secrets and ingress configs. Well, I'm not sure if there's a way to actually update certificates; instead, I've created new ones, which created new secrets, and then updated ingress configs (just secrets' names)
The reason that your certificates didn't work, it not because you used staging server, but because you didn't specify the tls object within the Ingress rules.
Certbot's staging exists only for the purpose of testing, and for not "ban" you while you testing things out if you request more than 5 certificates/hour.
When you verify that everything works as expected, you can use the normal non-staging server.
This is how it should be done:
Cluster Issuer object:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: user#example.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
Ingress Object:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-kubernetes-ingress
namespace: tests
labels:
app: hello-kubernetes-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- test3.projectname.org
secretName: hello-tls
rules:
- host: test3.projectname.org
http:
paths:
- pathType: ImplementationSpecific
path: "/"
backend:
service:
name: hello-kubernetes-ingress
port:
number: 80
The certificate and the key, are stored in a secret called "hello-tls" which you didn't also specify in your initial example, hence the failure you was receiving.
I had a project hosted on kubernetes and using nginx ingress controller and cert manager to handle the SSL certificates. It was all working fine until I change the hosting of the kubernetes cluster to another data center, and I happen shut down the old cluster without backing up the private keys.
Now my sites are showing NET::ERR_CERT_AUTHORITY_INVALID.
Any suggestions on how to either revoke the old cert so that I can regenerate a new certificate in the new cluster, or if there is a way that I can override the old cert?
Thanks for helping out!
Ingress controller:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "prod-letsencrypt"
# nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
spec:
tls:
- hosts:
- xxx.example.com
secretName: example-tls
rules:
- host: xxx.example.com
http:
paths:
- backend:
serviceName: example-svc
servicePort: 80
Issuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: prod-letsencrypt
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: xxx#example.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: prod-letsencrypt
# Enable the HTTP-01 challenge provider
solvers:
- dns01:
route53:
region: us-east-2
role: arn:aws:iam::xxxx:role/dns-challenge-role
describing the certificate in the new cluster and only see the cert is being requested.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 60m cert-manager Issuing certificate as Secret does not exist
Normal Generated 60m cert-manager Stored new private key in temporary Secret resource "example-tls-f4xt5"
Normal Requested 60m cert-manager Created new CertificateRequest resource "example-tls-xmxqm"
I'm setting up a Kubernetes cluster in AWS using Kops. I've got an nginx ingress controller, and I'm trying to use letsencrypt to setup tls. Right now I can't get my ingress up and running because my certificate challenges get this error:
Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://critsit.io/.well-known/acme-challenge/[challengeId]': Get http://critsit.io/.well-known/acme-challenge/[challengeId]: EOF
I've got a LoadBalancer service that's taking public traffic, and the certificate issuer automatically creates 2 other services which don't have public IPs.
What am I doing wrong here? Is there some networking issue preventing the pods from finishing the acme flow? Or maybe something else?
Note: I have setup an A record in Route53 to direct traffic to the LoadBalancer.
> kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cm-acme-http-solver-m2q2g NodePort 100.69.86.241 <none> 8089:31574/TCP 3m34s
cm-acme-http-solver-zs2sd NodePort 100.67.15.149 <none> 8089:30710/TCP 3m34s
default-http-backend NodePort 100.65.125.130 <none> 80:32485/TCP 19h
kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 19h
landing ClusterIP 100.68.115.188 <none> 3001/TCP 93m
nginx-ingress LoadBalancer 100.67.204.166 [myELB].us-east-1.elb.amazonaws.com 443:30596/TCP,80:30877/TCP 19h
Here's my ingress setup:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: critsit-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/acme-challenge-type: "http01"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- critsit.io
- app.critsit.io
secretName: letsencrypt-prod
rules:
- host: critsit.io
http:
paths:
- path: /
backend:
serviceName: landing
servicePort: 3001
And my certificate issuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: michael.vegeto#gmail.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
selector: {}
Update: I've noticed that my load balancer has all of the instances marked as OutOfOrder because they're failing health checks. I wonder if that's related to the issue.
Second update: I abandoned this route altogether, and rebuilt my networking/ingress system using Istio
The error message you are getting can mean a wide variety of issues. However, there are few things you can check/do in order to make it work:
Delete the Ingress, the certificates and the cert-manager fully. After that add them all back to make sure it installs clean. Sometimes stale certs or bad/multi Ingress pathing might be the issue. For example you can use Helm:
helm install my-nginx-ingress stable/nginx-ingress
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.0 --set installCRDs=true
Make sure your traffic allows HTTP or has HTTPS with a trusted cert.
Check if hairpin mode of your loadbalancer and make sure it is working.
Add: nginx.ingress.kubernetes.io/ssl-redirect: "false" annotation to the Ingress rule. Wait a moment and see if valid cert will be created.
You can manually manually issue certificates in your Kubernetes cluster. To do so, please follow this guide.
The problem can solve itself in time. Currently if the self check fails, it
updates the status information with the reason (like: self check failed) and than
tries again later (to allow for propagation). This is an expected behavior.
This is an ongoing issue that is being tracked here and here.
I'm trying to use the certificates obtained through digicert to enable https on my nginx-ingress. We've obtained a wildcard certificate and I have the following files.
domain_name_2019-2021.csr
domain_name_2019-2021.key
domain_name_2019-2021.pem
DigiCertCA2_2019-2021.pem
star_domain_name_2019_2021.pem
TrustedRoot.pem
I've created the tls secrets by running the following commands
kubectl create secret tls tls-secret --key ${KEY_FILE} --cert ${CERT_FILE}
And used these secrets in my ingress configuration like so
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-service
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
tls:
- hosts:
- {{ .Values.host }}
secretName: tls-secret
rules:
- host: {{ .Values.host }}
http:
paths:
- path: /
backend:
serviceName: service_name
servicePort: 443
However when browse to subdomain.domain_name.com I get an invalid certificate with an error of This certificate has not been verified by a third party. And the certificate its using says Kubernetes Ingress Controller Fake Certificate
you can follow this, to install Jetstack cert-manager, once you make this installed, please follow this stackoverflow post.
It will solve your query.
The current certificates created by you are not necessary for this, here the certificate will be automatically created by jetstack once it would be able to get the acme challenge verified, for that verification sake you need to map the DNS or hostname to the Load balancer IP of nginx.
This should solve your purpose to get http to https conversion