Right after enabling cert-manager in Ingress Controller by TTFB (time to first byte) increased by 200+ms in most of the regions. Without SSL, it was <200ms to 80% of the regions. After enabling SSL only 30% have TTFB <200ms
without SSL
with SSL
My Ingress definition:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
kubernetes.io/tls-acme: "true"
spec:
rules:
- host: gce.wpspeedmatters.com
http:
paths:
- path: /
backend:
serviceName: wordpress
servicePort: 80
tls:
- secretName: tls-prod-cert
hosts:
- gce.wpspeedmatters.com
Switched to TLS 1.3 and I was able to above shave off extra 50-150ms!
I wrote a detailed blog post too: https://wpspeedmatters.com/tls-1-3-in-wordpress/
With TLS 1.3:
Related
I am playing a little bit with path-based routing via Kubernetes ingresses (we use Nginx ingress). We have one service up and running behind authentication via external service like the one in the first snippet. This service is already used by other services in our ecosystem, however, as the development progresses, the service now has management API and we need to turn off the authentication for specific paths. According to this answer, this is not possible for basic auth and, according to what I have tried (second snippet), it is not possible for authentication via external service either. Is there any other way to allow specific paths without authentication without completely rearchitecting ingresses and domains/paths?
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: path-based-routing-1
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: nginx-external
nginx.ingress.kubernetes.io/auth-url: http://authorizer.dev.svc.cluster.local
spec:
tls:
- hosts:
- 'routing.example.com'
secretName: routing.example.com
rules:
- host: 'routing.example.com'
http:
paths:
- path: /
backend:
serviceName: path-based-routing-1
servicePort: 80
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: path-based-routing-2
annotations:
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/auth-type: ""
kubernetes.io/ingress.class: nginx-external
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- 'routing.example.com'
secretName: routing.example.com
rules:
- host: 'routing.example.com'
http:
paths:
- path: /management-api
backend:
serviceName: path-based-routing-2
servicePort: 80
Assume I have have two ingresses ingress-a and ingress-b for the same host but with different paths:
ingress-a:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: app-a
namespace: namespace-a
spec:
rules:
- host: myhost.com
http:
paths:
- backend:
serviceName: app-a
servicePort: 8080
path: /path-a
tls:
- hosts:
- myhost.com
secretName: tls-a
ingress-b:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name: app-b
namespace: namespace-b
spec:
rules:
- host: myhost.com
http:
paths:
- backend:
serviceName: app-b
servicePort: 8080
path: /path-b
tls:
- hosts:
- myhost.com
secretName: tls-b
Now I need to update the certificate. Assume I create the new secret in tls-new but only update ingress-a to point to that. Which of the two ingresses would win?
I guess I should simply overwrite the existing secret but I am trying to understand how the rules for ingresses would work in the above scenario where two different tls secrets are being referenced for the same host.
NGINX and NGINX Plus Ingress controller for Kubernetes has support for mergeable Ingress Types.
A Master is declared using nginx.org/mergeable-ingress-type: master. A Master will process all configurations at the host level, which includes the TLS configuration, and any annotations which will be applied for the complete host. There can only be one ingress resource on a unique host that contains the master value. Paths cannot be part of the ingress resource.
A Minion is declared using nginx.org/mergeable-ingress-type: minion. A Minion will be used to append different locations to an ingress resource with the Master value. TLS configurations are not allowed. Multiple minions can be applied per master as long as they do not have conflicting paths. If a conflicting path is present then the path defined on the oldest minion will be used.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress-master
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "master"
spec:
tls:
- hosts:
- cafe.example.com
secretName: cafe-secret
rules:
- host: cafe.example.com
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress-coffee-minion
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/mergeable-ingress-type: "minion"
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
The minion can not have TLS, only the master can have TLS and you change TLS in master.
I have installed cert-manager 0.12.0 for SSL certificate.
My Issuer file is
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: my#email.com
privateKeySecretRef:
name: letsencrypt-prod
http01: {}
My certificate file
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: tls-secret
spec:
secretName: tls-secret-prod
dnsNames:
- mydomain.com
acme:
config:
- http01:
ingressClass: nginx
domains:
- mydomain.com
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
Ingress configuration is
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cms
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/tls-acme: "true"
spec:
tls:
- hosts:
- mydomain.com
secretName: tls-secret-prod
rules:
- host: mydomain.com
http:
paths:
- backend:
serviceName: apostrophe
servicePort: 80
path: /
But still, SSL certificated is not valid. And Common name is “Kubernetes Ingress Controller Fake Certificate”.
The following result to show orders and challenges
kubectl get orders, challenges -o wide
NAME STATE DOMAIN REASON AGE
challenge.certmanager.k8s.io/tls-secret-155743219-0 pending mydomain.com pods "cm-acme-http-solver-gk2zx" is forbidden: minimum cpu usage per Container is 100m, but request is 10m. 26m
I have updated the resources limit the range and reinstalled cert-manager with helm. I am still getting this error.
I am not sure what goes wrong or show how to fix this.
Please let me know if you need anything. Thanks in advance!
The problem lays in cpu limits defined for specific pod.
You have to change minimum CPU limit in deployment configuration file. As you can see pod (cm-acme-http-solver) is requesting 100m CPU usage while minimum CPU usage defined for specific pod is *10m**. So change CPU limits in deployment configuration file from 100m to 10m or less or you can also increase CPU requests.
Take a look here: cert-manager-kubernetes, pod-min-cpu-request.
Useful article: resources-limits-kubernetes.
I had my original infrastructure built around this tutorial. https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes.
Now I am trying to migrate to managing my own cert and terminating SSL at the load balancer.
With my YAML updates, the load balancer in DigitalOcean shows that all nodes are unhealthy and I the URL response with "503 Service Unavailable
No server is available to handle this request." However, the endpoint shows a secure HTTPS connection. What am I doing wrong?
My new non-functional YAML definitions below.
LoadBalancer
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
annotations:
service.beta.kubernetes.io/do-loadbalancer-redirect-http-to-https: "true"
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
service.beta.kubernetes.io/do-loadbalancer-certificate-id: "**************"
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: http
Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
spec:
tls:
- hosts:
- ******.com
- api.*******.com
rules:
- host: **********.com
http:
paths:
- backend:
serviceName: frontend-angular
servicePort: 80
- host: api.********.com
http:
paths:
- backend:
serviceName: backend-server
servicePort: 80
I reached out to DigitalOcean support (which is incredible). My issue was I didn't create the ingress-nginx pod. These are the two steps, as listed in the tutorial, that I missed.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml.
My actual YAML definitions were correct.
The best way is just using Digital Ocean Marketplace (https://marketplace.digitalocean.com/apps/nginx-ingress-controller).
Installing it manually will cause a lot of issues due do outdated yml files.
I am using the standard L7 load balancing ingress on Google Container Engine. I have installed it through the following ingress definition:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: l7-ingress-{{environment}}
spec:
tls:
- secretName: web-secret
backend:
serviceName: web
servicePort: 80
Now, my question is, how do I ensure that the TLS certificate is updated once the secret web-secret changes? AFAICT, it currently stays the same even though the underlying secret changes.
Apparently, the L7 ingress doesn't currently monitor the TLS secret for changes. But a PR to solve this has been merged, so it should only be a matter of time.
My experience to workaround this is to delete and create the ingress, but making sure you specify the IP in the YAML you pass to kubectl create -f:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dev-ing
spec:
tls:
- secretName: tls-sekret
rules:
- host: tryout.example.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
status:
loadBalancer:
ingress:
- ip: 130.211.n.n
I couldn't find any documentation stating that this is the way to ensure you will get the same IP, but for me it worked. Use with caution on production systems where you can not afford to loose the IP!
I just tried a simple apply on an edited secret, and yes it worked. The web console and gcloud compute ssl-certificates list reported the change right away, and the load balancer started serving it up in about 10 minutes. It would be nice to have this officially documented! Especially because there are other corners of k8s where changes to secrets aren't automatically picked up, like deployments, so we don't take it for granted.
The Google L7 loadbalancer does exchange the underlying certificate if updated. You have to apply the correct annotations:
Secret
apiVersion: v1
kind: Secret
data:
tls.crt: xxx
tls.key: xxx
metadata:
name: tls-secret
type: kubernetes.io/tls
Ingress
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: dev-ing
annotations:
# Do not forget this annotation
kubernetes.io/ingress.class: "gce"
spec:
tls:
- hosts:
- tryout.example.com
secretName: tls-secret
backend:
serviceName: nginx
servicePort: 80
The exchange happens somehow in the background, be aware of the time it takes (5-15 minutes).