keycloak stops working when enabling tls behind nginx ingress controller - ssl

I need advice on how to configure nginx ingress controller and keycloak on eks kubernetes with TLS.
My first attempt was to get keycloak working without TLS. This works fine. But when I add TLS to the ingress definition, keycloak gives a Invalid parameter: redirect_uri error.
This is the working sample:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: keycloak-tls-test
namespace: keycloak-tls-test
spec:
rules:
- host: keycloak.test.myhops.com
http:
paths:
- backend:
serviceName: keycloak
servicePort: 80
When I add the tls part, I get the aforementioned error. See yaml file below.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: keycloak-tls-test
namespace: keycloak-tls-test
spec:
tls:
- hosts:
- keycloak.test.myhops.com
secretName: test-myhops-tls
rules:
- host: keycloak.test.myhops.com
http:
paths:
- backend:
serviceName: keycloak
servicePort: 80
Any suggestions on how to solve this? One online suggestion was to add https://keycloak.test.myhops.com/* to the Valid Redirect URL for the security-admin-console client in the master realm. This solved the invalid redirect_uri error, but then I only got blank pages. I could not find hints in the keycloak logging.

I figured out what caused the problem. It seems that when keycloak is used behind a reverse proxy that terminates the TLS, then keycloak needs proxy-address-forwarding to be set. For the docker container this means that the env var PROXY_ADDRESS_FORWARDING=true does the trick. So Ingress was correct and the keycloak deployment needed an extra environment variable.
Please not that ingress-nginx already set the correct headers for this to work:
X-Forwarded-For
X-Forwarded-Proto
Docker documentation
[keycloak documentation2

Related

Reverse proxy a site with SNI support using kubernetes nginx-ingress

I am setting a reverse proxy using kubernetes nginx-ingress, but I don't know how to add nginx parameters to the configuration, specifically: proxy_ssl_server_name. How do I set ingress parameters in yaml configurations?
I already tried using the server-snippet annotation, but it seems like it's not adding the parameter to the nginx.conf file in the cluster pods.
Here is the current code for the reverse proxy:
kind: Service
apiVersion: v1
metadata:
name: formstack
namespace: serves
spec:
type: ExternalName
externalName: fluidsignal.formstack.com
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: formstack
namespace: serves
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/upstream-vhost: "fluidsignal.formstack.com"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
tls:
- hosts:
- fluidattacks.com
secretName: fluidattacks-cert
rules:
- host: fluidattacks.com
http:
paths:
- backend:
serviceName: formstack
servicePort: 443
path: /forms(.*)
After setting up the proxy, I get a 502 Bad Gateway error from Nginx. After looking at the pods logs, I see I'm getting the following openssl error: SSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:SSL alert number 40, which is why I want to add the parameter I mentioned before.
I just figured out that I was indeed using the right annotation: nginx.ingress.kubernetes.io/server-snippet,
But I needed to add an extra parameter: proxy_ssl_name
Adding the following code fixed the problem:
nginx.ingress.kubernetes.io/server-snippet: |
proxy_ssl_name fluidsignal.formstack.com;
proxy_ssl_server_name on;
Everything seems to be working fine now :D

Ingress TLS routes with cert-manager not applied

I have a K8s cluster (v1.12.8-gke.10) in GKE and have a nginx ingress with hosts rules. I am trying to enable TLS using cert-manager for ingress routes. I am using a selfsign cluster issuer. But, when I access the site over HTTPS, I am still getting the default K8s certificate. (The certificate is only valid for the following names: kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: test
name: test
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/rewrite-target: /
certmanager.k8s.io/cluster-issuer: selfsign
spec:
tls:
- secretName: test
hosts:
- test.example.com
rules:
- host: test.example.com
http:
paths:
- path: /
backend:
serviceName: test
servicePort: 80
I have checked the following and is working fine:
A cluster issuer named "selfsign"
A valid self-signed certificate backed by a secret "test"
A healthy and running nginx ingress deployment
A healthy and running ingress service of type load-balancer
I think it's issue of clusterissuer
Just have a look at my cluster issuer and check
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: it-support#something.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: prod
# Enable the HTTP-01 challenge provider
http01: {}
Check for the right url to get production-grade certificates:
server: https://acme-v02.api.letsencrypt.org/directory
If your server url is something like this :
server: https://acme-staging-v02.api.letsencrypt.org/directory
which means you are applying for the staging certificate which may occur the error.
I've followed the tutorial from Digital Ocean and was able to enable TLS using cert-manager for ingress routes using Helm, Tiller, Letsencrypt and Nginx Ingress controller in GKE.
Instead of host test-example.com, I used my own domain name and spun up dummy backend services (echo1 and echo2) for testing purposes.
After followed the tutorial and to verify that HTTPS is working correctly, try to curl the host:
$ curl test.example.com
you should see a 308 http response (Permanent Redirect). This indicates that HTTP requests are being redirected to use HTTPS.
On the other hand, try running curl on:
$ curl https://test.example.com
Should show you the site response.
You can run the previous commands with the verbose -v flag to check into the certificate handshake and to verify the certificate information.

oauth2_proxy not asking for authentication

I am trying to setup oauth2_proxy on kubernetes to secure one of my single page application and I am using github as the authentication provider. I found this link useful and I followed this for doing my setup.
[https://alikhil.github.io/2018/05/oauth2-proxy-for-kubernetes-services/]
I have created an application on github and used my DNS Name in place of the HomePageURL and CallBackURL (https://auth.example.com replaced with https://example.com) because I do not have TLS secrets generated for auth.example.com. Rather I have TLS certificates generated for example.com because this domain belongs to me. I was getting error in nginx-controller that the certificate belongs to example.com and not to auth.example.com as these URLs have been used in defining the example Ingress and oauth proxy ingress and this was the basis for me to do the before mentioned chang.
My Ingresses looks like this
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: oauth2-proxy
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: oauth2-proxy
servicePort: 4180
path: /oauth2
tls:
- hosts:
- example.com
secretName: oauth-proxy-tls
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: oauth-main-ingress
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
ingress.kubernetes.io/auth-url: https://example.com/oauth2/auth
ingress.kubernetes.io/auth-signin: https://example/oauth2/start?rd=https://$host$request_uri$is_args$args
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: example-service
servicePort: 80
path: /
tls:
- hosts:
- example.com
secretName: tls-secret
I want that whenever I click example.com it should display page for github authentication but in my case its directly giving the response which a service should give after successful authentication. I am not being asked to provide credentials. Also, I am getting error in my ingress controller logs as 7 controller.go:753] Error obtaining Endpoints for Service "default/oauth2-proxy": no object matching key "default/oauth2-proxy" in local store
Also, I tried replacing using nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.oauth-proxy.svc.cluster.local:4180/oauth2/auth as mentioned in the link but it did not work for me. Can some one explain why is oauth2_proxy not asking for authentication and ingress is serving the requests directly without asking for authentication?
The annotation declared in the oauth-main-ingress yaml is incorrect.
As per the kubernetes/nginx-ingress documentation the annotation for external auth-url should be
nginx.ingress.kubernetes.io/auth-url: http://<oauth-service-url>
instead of
ingress.kubernetes.io/auth-url: http://<oauth-service-url>

Kubernetes on AWS: Preserving Client IP with nginx-ingress + cert-manager

We have setup Kubernetes with nginx-ingress combined with cert-manager to automatically obtain and use SSL certificates for ingress domains using LetsEncrypt using this guide: https://medium.com/#maninder.bindra/auto-provisioning-of-letsencrypt-tls-certificates-for-kubernetes-services-deployed-to-an-aks-52fd437b06b0. The result is that each Ingress defines its own SSL certificate that is automatically provisioned by cert-manager.
This all works well but for one problem, the source IP address of the traffic is lost to applications in Pods.
There is an annotation that is advised to use to apply to the nginx-ingress controller service service.beta.kubernetes.io/aws-load-balancer-backend-protocol: '*'. This has the effect of preserving source IP addresses. However, doing it breaks SSL:
An error occurred during a connection to {my.domain.com}. SSL received a record that exceeded the maximum permissible length. Error code: SSL_ERROR_RX_RECORD_TOO_LONG
My head is starting to spin. Does anyone know of any approaches to this (it seems to me that this would be a common requirement)?
Ingress configuration:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-http-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
spec:
rules:
- host: my.host.com
http:
paths:
- path: /
backend:
serviceName: my-http-service
servicePort: 80
tls:
- hosts:
- "my.host.com"
secretName: malcolmqa-tls
As #dom_watson mentioned in the comments, adding parameter controller.service.externalTrafficPolicy=Local to Helm install configuration solved the issue due to the fact that Local value preserves the client source IP, thus the network traffic will reach target Pod in Kubernetes cluster. Find more information in the official Kubernetes guidelines.
helm upgrade my-nginx stable/nginx-ingress --set rbac.create=true --set controller.service.externalTrafficPolicy=Local

SignalR connection via K8S Ingress

I'm trying to expose a SignalR hub hosted in a Kubernetes (Azure) pod. Basically, the authentication and the handshake steps work fine, but when I trigger some action, all clients connected via the k8s Ingress doesn't receive the message. Has anybody experienced this issue or just have shared SignalR hubs through Kubernetes - Ingress?
ingress.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: endpoints
annotations:
kubernetes.io/ingress.class: addon-http-application-routing
ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.org/websocket-services: "myservice"
spec:
rules:
- host: api.[MY-DOMAIN].com
http:
paths:
- backend:
serviceName: myservice
servicePort: 80
path: /myservice
Try:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/session-cookie-hash: sha1
nginx.ingress.kubernetes.io/session-cookie-name: REALTIMESERVERID
I wrote a sample project a while back, if you want a working example: DenisBiondic/RealTimeMicroservices
As a side note, consider using Azure SignalR Service, it should remove many headaches (also in the example above).
Not familiar with SignalR but there could be a couple of things.
The nginx Ingress might be stripping some http headers that SignalR needs. Are you familiar with the http headers that the SignalR is supposed to receive?
After authenticating, is it possible that SignalR hub is trying to speak TLS? I see that you have this running on port 80 with not TLS. You would have to configure something like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: endpoints
annotations:
kubernetes.io/ingress.class: addon-http-application-routing
ingress.kubernetes.io/ssl-redirect: "false" <== you may need to remove
nginx.ingress.kubernetes.io/ssl-redirect: "false" <== you may need to remove
nginx.org/websocket-services: "myservice"
spec:
rules:
- host: api.[MY-DOMAIN].com
http:
paths:
- backend:
serviceName: myservice
servicePort: 80
path: /myservice
tls:
- secretName: <your-tls-certs>
Hope it helps!