I'm trying to uses ssl in several Ingress on k8s. A first look up lead me to cert-manager but I can't make it work and I suspect that the cause is because my Cloud provider (ovh) is not supported.
I'm using kubernetes 1.17 and cert-manager 0.13.0
The first error I encountered was related to the web hook secret and no solutions worked for me.
Because of this I deployed cert-manager without web-hook but I still couldn't get a ClusterIssuer up and running. When I apply the following :
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: leonard.panichi#gmail.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- http01:
ingress:
class: nginx
a clusterissuer is created but there is no Status when I run describe on it.
So, after new searches (and more than a day of struggling), I found that ovh might not be compatible with cert-manager
which let me think that I'm loosing my time with this strategy.
Hence I'm looking for a new strategy.
How can I use certbot, there is a docker image certbot/certbot, to create and renew a few ssl certificates in kubernetes secrets in order to use them in my Ingress ? Is there any other way that do not requiere GKE, AWS, something simple, portable, production ready, etc...
Sincerely,
me
Related
I'm setting up a Kubernetes cluster in AWS using Kops. I've got an nginx ingress controller, and I'm trying to use letsencrypt to setup tls. Right now I can't get my ingress up and running because my certificate challenges get this error:
Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://critsit.io/.well-known/acme-challenge/[challengeId]': Get http://critsit.io/.well-known/acme-challenge/[challengeId]: EOF
I've got a LoadBalancer service that's taking public traffic, and the certificate issuer automatically creates 2 other services which don't have public IPs.
What am I doing wrong here? Is there some networking issue preventing the pods from finishing the acme flow? Or maybe something else?
Note: I have setup an A record in Route53 to direct traffic to the LoadBalancer.
> kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cm-acme-http-solver-m2q2g NodePort 100.69.86.241 <none> 8089:31574/TCP 3m34s
cm-acme-http-solver-zs2sd NodePort 100.67.15.149 <none> 8089:30710/TCP 3m34s
default-http-backend NodePort 100.65.125.130 <none> 80:32485/TCP 19h
kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 19h
landing ClusterIP 100.68.115.188 <none> 3001/TCP 93m
nginx-ingress LoadBalancer 100.67.204.166 [myELB].us-east-1.elb.amazonaws.com 443:30596/TCP,80:30877/TCP 19h
Here's my ingress setup:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: critsit-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/acme-challenge-type: "http01"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- critsit.io
- app.critsit.io
secretName: letsencrypt-prod
rules:
- host: critsit.io
http:
paths:
- path: /
backend:
serviceName: landing
servicePort: 3001
And my certificate issuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: michael.vegeto#gmail.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
selector: {}
Update: I've noticed that my load balancer has all of the instances marked as OutOfOrder because they're failing health checks. I wonder if that's related to the issue.
Second update: I abandoned this route altogether, and rebuilt my networking/ingress system using Istio
The error message you are getting can mean a wide variety of issues. However, there are few things you can check/do in order to make it work:
Delete the Ingress, the certificates and the cert-manager fully. After that add them all back to make sure it installs clean. Sometimes stale certs or bad/multi Ingress pathing might be the issue. For example you can use Helm:
helm install my-nginx-ingress stable/nginx-ingress
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.0 --set installCRDs=true
Make sure your traffic allows HTTP or has HTTPS with a trusted cert.
Check if hairpin mode of your loadbalancer and make sure it is working.
Add: nginx.ingress.kubernetes.io/ssl-redirect: "false" annotation to the Ingress rule. Wait a moment and see if valid cert will be created.
You can manually manually issue certificates in your Kubernetes cluster. To do so, please follow this guide.
The problem can solve itself in time. Currently if the self check fails, it
updates the status information with the reason (like: self check failed) and than
tries again later (to allow for propagation). This is an expected behavior.
This is an ongoing issue that is being tracked here and here.
I've setup cert mananger on microk8s following these instructions, I had it working 6 months ago but have since had to start again from scratch. Now when I setup my Cluster Issuer I'm getting the error below.
Everything else seems fine and in a good state. I'm struggling to know where to start debugging this.
Error initializing issuer: Get "https://acme-v02.api.letsencrypt.org/directory": remote error: tls: handshake failure
Cluster Issuer yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: <myemail>
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: prod-issuer-account-key
solvers:
- http01:
ingress:
class: nginx
UPDATE
Some extra info
All pods for cert manager are running, here are the logs
cert-manager pod logs
cert-manager-cainjector logs only shows some warnings about deprecated apis
cert-mananger-webhook logs
Describe ClusterIssuer
I've tried to get a cert for an ingress resource but it errors saying the cluster issuer isn't in a ready state
After uninstalling and reinstalling everything including Microk8s I tried again no luck. Then I tried using the latest helm chart v1.0.2 which had a newer cert-manager version, seemed to work straight away.
Another note, mainly to myself. This issue was also caused by having search domains setup in netplan, once removed everything started working.
Hello I m struggling to get the Cert-Manager work with let'sencrypt on my Azure AKS to secure and asp.net core web app.
I have a ClusterIssuer like that:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencryptstaging-issuer
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: letsencryptstaging#prodibi.com
privateKeySecretRef:
name: letsencryptstaging-secret
solvers:
- http01:
ingress:
class: nginx
and I request a certificate like that:
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: aks-prodibiv2-com-staging
spec:
secretName: aks-prodibiv2-com-staging-secret
duration: 2160h
renewBefore: 480h
organization:
- prodibiv2
dnsNames:
- aks.prodibiv2.com
issuerRef:
name: letsencryptstaging-issuer
kind: ClusterIssuer
I also have added the annotations to the ingress controller I would like to use
certmanager.k8s.io/acme-challenge-type: http01
certmanager.k8s.io/cluster-issuer: letsencryptstaging-issuer
In the following screenshot we can see that the certificate request is "Waiting to complete"
We can see also that we have two ingress controller and the one for the challenge seems to not have an IP, and the domain is pointing to the ingress-prodibiweb
if I try to put the domain in front of the .well-known path I get a 404 not found error.
So my guess is that the cert-manager is not configured properly to use the ingress-prodibiweb (that point to asp.net core webapp) or something like that. Any idea on what can I try to get it working?
Your ingress ip is private. No way for Let's encrypt to access.
You need to make sure you are using Nginx Ingress which you are using
- http01:
ingress:
class: nginx
Make sure your domain is point to right IP. (Ingress IP and also a Load Balancer IP)
Now is 20.50.42.93
https://mxtoolbox.com/SuperTool.aspx?action=a%3aaks.prodibiv2.com&run=toolpage
And dns01 also a solution to request SSL. You can give it a try if you have enough permissions.
I am using cert-manager-v0.10.0 installed from its helm chart
I am using kong like ingress controller to manage the ingress operations.
So I have created a ClusterIssuer resource in order it could be contacted from an Ingress resource via kong-ingress controller.
The ClusterIssuer is this:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: username#mydomain.org
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: kong
The ingress resource that I am using is this.
You can see here, that I am pointing it to the ClusterIssuer created previously and
also I am pointing it to kong as an ingress controller, according to the kubernetes.io/ingress.class: "kong" annotation included there:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
# add an annotation indicating the issuer to use.
certmanager.k8s.io/cluster-issuer: letsencrypt-prod # letsencrypt-staging
kubernetes.io/ingress.class: "kong"
plugins.konghq.com: swaggerapi-customer-production-basic-auth, swaggerapi-customer-production-acl
name: production-customer-ingress-app
# namespace: default
spec:
rules:
- host: appprod.mydomain.org
http:
paths:
- backend:
serviceName: customer-production-app
servicePort: 80
path: /comcustomerpr
tls: # < placing a host in the TLS config will indicate a cert should be created
- hosts:
- appprod.mydomain.org
secretName: letsencrypt-prod # < cert-manager will store the created certificate in this secret.
So, when I create the Ingress resource above, the secretName referenced above in my ingress is created and also a certificate resource with the same name ... that is letsencrypt-prod.
It will be the certificate resource which receive the LetsEncrypt validation successful process ...
I got TLS encryption and everything is OK here.
But now, I want to know how will be the renewal process. Because I am pretty sure at the moment this renewal certificate process it does not to happen automatically ...
I was reading something here https://docs.cert-manager.io/en/latest/reference/certificates.html?highlight=renewal#certificate-duration-and-renewal-window and this documentation says that is necessary attach to the certificate resource created (kind:Certificate) the spec.duration and spec.renewBefore attributes of this way
spec:
secretName: example-tls
duration: 24h
renewBefore: 12h
If my certificate issued by LetsEncrypt has a 90 days as a default duration, how can I specify these spec.duration and spec.renewBefore attributes?
I want to get into in this concern, because my main question is I am not creating the certificate, it is created when the Ingress resource (above referenced) is executed.
How can I address the renewal process here with this approach that I am doing?
UPDATE
Do I need to create a specific kind:Certificate resource, referencing the secret that I got from LetsEncrypt?
I mean, something like this?
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: letsencrypt-prod
spec:
secretName: letsencrypt-prod
dnsNames:
- mydomain.com
acme:
config:
- http01:
ingressClass: kong
domains:
- mydomain.com
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
I have the doubt here, because currently I am not getting the certificate renewal action
since you have used the letsencrypt-prod issuer, and haven't done anything special/non-standard, the certificate renewal process will be completely automatic for you.
By default the letsencrypt certificates are valid fro 90-days, and renewed automatically every 30-days. If you don't have some strict requirements to use purchased certificates, or use some other specific Certificate Authority, this is a great option to use.
If you still have doubts then you can do the following to see for yourself. First decode the current certificates secret data and inspect the certificate contents with the openssl command. You'll be able to see the certificate expiry date, and make a note of that. Now if you subtract 59-days from that expiry date that should give you roughly the date that cert-manager will attempt to renew the certificate on. I add an extra day just to be safe we aren't too early. Then on that date repeat this process again; decoding the certificate secret, inspecting the certificate with the openssl command, and checking the certificate expiry date. You'll notice the expiry date for the certificate is different than before, hence it's was automatically renewed as we expected.
Hope this helps.
We have setup Kubernetes with nginx-ingress combined with cert-manager to automatically obtain and use SSL certificates for ingress domains using LetsEncrypt using this guide: https://medium.com/#maninder.bindra/auto-provisioning-of-letsencrypt-tls-certificates-for-kubernetes-services-deployed-to-an-aks-52fd437b06b0. The result is that each Ingress defines its own SSL certificate that is automatically provisioned by cert-manager.
This all works well but for one problem, the source IP address of the traffic is lost to applications in Pods.
There is an annotation that is advised to use to apply to the nginx-ingress controller service service.beta.kubernetes.io/aws-load-balancer-backend-protocol: '*'. This has the effect of preserving source IP addresses. However, doing it breaks SSL:
An error occurred during a connection to {my.domain.com}. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG
My head is starting to spin. Does anyone know of any approaches to this (it seems to me that this would be a common requirement)?
Ingress configuration:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-http-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
rules:
- host: my.host.com
http:
paths:
- path: /
backend:
serviceName: my-http-service
servicePort: 80
tls:
- hosts:
- "my.host.com"
secretName: malcolmqa-tls
As #dom_watson mentioned in the comments, adding parameter controller.service.externalTrafficPolicy=Local to Helm install configuration solved the issue due to the fact that Local value preserves the client source IP, thus the network traffic will reach target Pod in Kubernetes cluster. Find more information in the official Kubernetes guidelines.
helm upgrade my-nginx stable/nginx-ingress --set rbac.create=true --set controller.service.externalTrafficPolicy=Local