We have setup Kubernetes with nginx-ingress combined with cert-manager to automatically obtain and use SSL certificates for ingress domains using LetsEncrypt using this guide: https://medium.com/#maninder.bindra/auto-provisioning-of-letsencrypt-tls-certificates-for-kubernetes-services-deployed-to-an-aks-52fd437b06b0. The result is that each Ingress defines its own SSL certificate that is automatically provisioned by cert-manager.
This all works well but for one problem, the source IP address of the traffic is lost to applications in Pods.
There is an annotation that is advised to use to apply to the nginx-ingress controller service service.beta.kubernetes.io/aws-load-balancer-backend-protocol: '*'. This has the effect of preserving source IP addresses. However, doing it breaks SSL:
An error occurred during a connection to {my.domain.com}. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG
My head is starting to spin. Does anyone know of any approaches to this (it seems to me that this would be a common requirement)?
Ingress configuration:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-http-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
rules:
- host: my.host.com
http:
paths:
- path: /
backend:
serviceName: my-http-service
servicePort: 80
tls:
- hosts:
- "my.host.com"
secretName: malcolmqa-tls
As #dom_watson mentioned in the comments, adding parameter controller.service.externalTrafficPolicy=Local to Helm install configuration solved the issue due to the fact that Local value preserves the client source IP, thus the network traffic will reach target Pod in Kubernetes cluster. Find more information in the official Kubernetes guidelines.
helm upgrade my-nginx stable/nginx-ingress --set rbac.create=true --set controller.service.externalTrafficPolicy=Local
Related
I am trying to deploy a React website together with an Express API on GKE. There should be multiple subdomains for both the website and the API, i.e.
domain.com, a.domain.com, b.domain.com, ... -> React app
api.domain.com, a.api.domain.com, b.api.domain.com -> Express API
The reasoning behind this is that the application is using cookie-based authentication, so the API and app need to be on the same subdomain (e.g. a.domain.com and api.a.domain.com).
Cert-manager and nginx ingress were deployed in the Kubernetes cluster with commands:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml
Everything works fine when I explicitly put all subdomains in the ingress and give each a separate TLS entry in the ingress. The certificates are successfully issued. But when using wildcards the certificates never get ready and when I try to open api.domain.com or domain.com it returns a "page not found" error.
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
cert-manager.io/issuer: letsencrypt-production
spec:
rules:
- host: "*.domain.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: react
port:
number: 80
- host: wordpress.domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: wordpress
port:
number: 80
- host: "*.api.domain.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api
port:
number: 9000
tls:
- hosts:
- "*.domain.com"
- domain.com
secretName: certificate-wildcard-domain
- hosts:
- "wordpress.domain.com"
secretName: certificate-wordpress-domain
- hosts:
- "*.api.domain.com"
- api.domain.com
secretName: certificate-api-domain
This is what is showing in the logs of the cert-manager pod:
cert-manager/orders "msg"="Failed to determine the list of Challenge resources needed for the Order" "error"="no configured challenge solvers can be used for this challenge" "resource_kind"="Order" "resource_name"="certificate-api-domain-9wvd9-2527200272" "resource_namespace"="default" "resource_version"="v1"
I do not understand the problem, why certificates are not issued correctly for wildcard entries and the website is not reachable, while e.g. for wordpress.domain.com the certificates are issued successfully and the website can be reached.
Issuer/ClusterIssuer solvers(Try Upgrade from v0.10 to v0.11)
Update the apiVersion on all your backed up resources from certmanager.k8s.io/v1alpha1 to cert-manager.io/v1alpha2
Support for the deprecated spec.http01 or spec.dns01 fields in Issuer and ClusterIssuer have been removed. Any Issuer or ClusterIssuer objects must be converted to use the equivalent spec.solvers[].http01 or spec.solvers[].dns01 syntax. You can read more about the Issuer resource in the configuration documentation.
Any issuers that haven't been converted will result in the cert-manager pod being unable to find any solvers at the expected location. This will result in errors like the following: no configured challenge solvers can be used for this challenge.
Let’s Encrypt issues wildcard certificates via ACMEv2 using the DNS-01 challenge. See ACME v2 Production Environment & Wildcard certificates for more technical information. Cert-manager.io: Docs: Configuration: ACME: DNS-01 contains details on the different options available on the Issuer resource's DNS01 challenge solver configuration.
Refer to No configured Challenge Solvers for ACME Prod only #2494 for more information, which may help to resolve your issue.
I'm setting up a Kubernetes cluster in AWS using Kops. I've got an nginx ingress controller, and I'm trying to use letsencrypt to setup tls. Right now I can't get my ingress up and running because my certificate challenges get this error:
Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://critsit.io/.well-known/acme-challenge/[challengeId]': Get http://critsit.io/.well-known/acme-challenge/[challengeId]: EOF
I've got a LoadBalancer service that's taking public traffic, and the certificate issuer automatically creates 2 other services which don't have public IPs.
What am I doing wrong here? Is there some networking issue preventing the pods from finishing the acme flow? Or maybe something else?
Note: I have setup an A record in Route53 to direct traffic to the LoadBalancer.
> kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cm-acme-http-solver-m2q2g NodePort 100.69.86.241 <none> 8089:31574/TCP 3m34s
cm-acme-http-solver-zs2sd NodePort 100.67.15.149 <none> 8089:30710/TCP 3m34s
default-http-backend NodePort 100.65.125.130 <none> 80:32485/TCP 19h
kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 19h
landing ClusterIP 100.68.115.188 <none> 3001/TCP 93m
nginx-ingress LoadBalancer 100.67.204.166 [myELB].us-east-1.elb.amazonaws.com 443:30596/TCP,80:30877/TCP 19h
Here's my ingress setup:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: critsit-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/acme-challenge-type: "http01"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- critsit.io
- app.critsit.io
secretName: letsencrypt-prod
rules:
- host: critsit.io
http:
paths:
- path: /
backend:
serviceName: landing
servicePort: 3001
And my certificate issuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: michael.vegeto#gmail.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
selector: {}
Update: I've noticed that my load balancer has all of the instances marked as OutOfOrder because they're failing health checks. I wonder if that's related to the issue.
Second update: I abandoned this route altogether, and rebuilt my networking/ingress system using Istio
The error message you are getting can mean a wide variety of issues. However, there are few things you can check/do in order to make it work:
Delete the Ingress, the certificates and the cert-manager fully. After that add them all back to make sure it installs clean. Sometimes stale certs or bad/multi Ingress pathing might be the issue. For example you can use Helm:
helm install my-nginx-ingress stable/nginx-ingress
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.0 --set installCRDs=true
Make sure your traffic allows HTTP or has HTTPS with a trusted cert.
Check if hairpin mode of your loadbalancer and make sure it is working.
Add: nginx.ingress.kubernetes.io/ssl-redirect: "false" annotation to the Ingress rule. Wait a moment and see if valid cert will be created.
You can manually manually issue certificates in your Kubernetes cluster. To do so, please follow this guide.
The problem can solve itself in time. Currently if the self check fails, it
updates the status information with the reason (like: self check failed) and than
tries again later (to allow for propagation). This is an expected behavior.
This is an ongoing issue that is being tracked here and here.
I need advice on how to configure nginx ingress controller and keycloak on eks kubernetes with TLS.
My first attempt was to get keycloak working without TLS. This works fine. But when I add TLS to the ingress definition, keycloak gives a Invalid parameter: redirect_uri error.
This is the working sample:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: keycloak-tls-test
namespace: keycloak-tls-test
spec:
rules:
- host: keycloak.test.myhops.com
http:
paths:
- backend:
serviceName: keycloak
servicePort: 80
When I add the tls part, I get the aforementioned error. See yaml file below.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: keycloak-tls-test
namespace: keycloak-tls-test
spec:
tls:
- hosts:
- keycloak.test.myhops.com
secretName: test-myhops-tls
rules:
- host: keycloak.test.myhops.com
http:
paths:
- backend:
serviceName: keycloak
servicePort: 80
Any suggestions on how to solve this? One online suggestion was to add https://keycloak.test.myhops.com/* to the Valid Redirect URL for the security-admin-console client in the master realm. This solved the invalid redirect_uri error, but then I only got blank pages. I could not find hints in the keycloak logging.
I figured out what caused the problem. It seems that when keycloak is used behind a reverse proxy that terminates the TLS, then keycloak needs proxy-address-forwarding to be set. For the docker container this means that the env var PROXY_ADDRESS_FORWARDING=true does the trick. So Ingress was correct and the keycloak deployment needed an extra environment variable.
Please not that ingress-nginx already set the correct headers for this to work:
X-Forwarded-For
X-Forwarded-Proto
Docker documentation
[keycloak documentation2
I'm trying to use the certificates obtained through digicert to enable https on my nginx-ingress. We've obtained a wildcard certificate and I have the following files.
domain_name_2019-2021.csr
domain_name_2019-2021.key
domain_name_2019-2021.pem
DigiCertCA2_2019-2021.pem
star_domain_name_2019_2021.pem
TrustedRoot.pem
I've created the tls secrets by running the following commands
kubectl create secret tls tls-secret --key ${KEY_FILE} --cert ${CERT_FILE}
And used these secrets in my ingress configuration like so
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-service
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
tls:
- hosts:
- {{ .Values.host }}
secretName: tls-secret
rules:
- host: {{ .Values.host }}
http:
paths:
- path: /
backend:
serviceName: service_name
servicePort: 443
However when browse to subdomain.domain_name.com I get an invalid certificate with an error of This certificate has not been verified by a third party. And the certificate its using says Kubernetes Ingress Controller Fake Certificate
you can follow this, to install Jetstack cert-manager, once you make this installed, please follow this stackoverflow post.
It will solve your query.
The current certificates created by you are not necessary for this, here the certificate will be automatically created by jetstack once it would be able to get the acme challenge verified, for that verification sake you need to map the DNS or hostname to the Load balancer IP of nginx.
This should solve your purpose to get http to https conversion
I have a K8s cluster (v1.12.8-gke.10) in GKE and have a nginx ingress with hosts rules. I am trying to enable TLS using cert-manager for ingress routes. I am using a selfsign cluster issuer. But, when I access the site over HTTPS, I am still getting the default K8s certificate. (The certificate is only valid for the following names: kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: test
name: test
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
certmanager.k8s.io/cluster-issuer: selfsign
spec:
tls:
- secretName: test
hosts:
- test.example.com
rules:
- host: test.example.com
http:
paths:
- path: /
backend:
serviceName: test
servicePort: 80
I have checked the following and is working fine:
A cluster issuer named "selfsign"
A valid self-signed certificate backed by a secret "test"
A healthy and running nginx ingress deployment
A healthy and running ingress service of type load-balancer
I think it's issue of clusterissuer
Just have a look at my cluster issuer and check
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: it-support#something.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: prod
# Enable the HTTP-01 challenge provider
http01: {}
Check for the right url to get production-grade certificates:
server: https://acme-v02.api.letsencrypt.org/directory
If your server url is something like this :
server: https://acme-staging-v02.api.letsencrypt.org/directory
which means you are applying for the staging certificate which may occur the error.
I've followed the tutorial from Digital Ocean and was able to enable TLS using cert-manager for ingress routes using Helm, Tiller, Letsencrypt and Nginx Ingress controller in GKE.
Instead of host test-example.com, I used my own domain name and spun up dummy backend services (echo1 and echo2) for testing purposes.
After followed the tutorial and to verify that HTTPS is working correctly, try to curl the host:
$ curl test.example.com
you should see a 308 http response (Permanent Redirect). This indicates that HTTP requests are being redirected to use HTTPS.
On the other hand, try running curl on:
$ curl https://test.example.com
Should show you the site response.
You can run the previous commands with the verbose -v flag to check into the certificate handshake and to verify the certificate information.