Why Traefik 2.2 & Let's Encrypt does not support new annotations? - traefik

I have setup traefik 2.2 in my self managed kubernetes cluster with Let's Encrypt support.
So far everything works. But the ingress Route configuration in my eyes is still clumsy. It only works if I define two IntgresRoutes - one for HTTP with a redirect middleware to https and one for the https. So my objects look like this:
# Middleware for Redirect http -> https
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: https-redirect
spec:
redirectScheme:
scheme: https
# IngressRoute http for a simple whoami service
---
kind: IngressRoute
apiVersion: traefik.containo.us/v1alpha1
metadata:
name: whoami-notls
namespace: default
spec:
entryPoints:
- web
routes:
- match: Host(`mydomain.foo.com`)
kind: Rule
services:
- name: whoami
port: 8080
# redirect http to https
middlewares:
- name: https-redirect
# IngresRoute https
---
kind: IngressRoute
apiVersion: traefik.containo.us/v1alpha1
metadata:
name: whoami-tls
namespace: default
spec:
entryPoints:
- websecure
routes:
- match: Host(`mydomain.foo.com`)
kind: Rule
services:
- name: whoami
port: 8080
tls:
certResolver: default
Is there not a more easy way to simply tell traefik that my service - which is listening on port 8080 - should be redirected to HTTPS in any case. Why do I need two separate ingresRoutes in my setup?
In the announcements for traefik 2.2. there was something like this:
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
name: foo
namespace: bar
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web, websecure
traefik.ingress.kubernetes.io/router.middlewares: redirect-http#kuberntes-crd
spec:
rules:
- host: foo.com
http:
paths:
- path: ""
backend:
serviceName: service1
servicePort: 80
It looks very simple. But this did not work for me - traefik is not recognizing this Ingress configuration.

With the help of the Traefik.io team in this discussion, I now solved the problem:
To use traefik annotations in Ingress make sure that in your deployment object you have added the ‘kubernetesingress’ provider:
...
spec:
containers:
- args:
- --api
....
- --providers.kubernetescrd=true
- --providers.kubernetesingress=true
....
For a global redirect form HTTP to HTTPS you can also configure this in your traefik deplyoment object:
# permanent redirecting of all requests on http (80) to https (443)
- --entrypoints.web.http.redirections.entryPoint.to=websecure
- --entrypoints.websecure.http.tls.certResolver=default
Now you can configure your ingress in an easy way:
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
name: myingress
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web, websecure
spec:
rules:
- host: example.foo.com
http:
paths:
- path: /
backend:
serviceName: whoami
servicePort: 80
See also my latest Blog post.

Related

GCP Health Checks with SSL enabled

I kind of new on Kubernetes stuff and I'm trying to improve one current system we have here.
The Application is developed using Spring Boot and until now it was using HTTP (Port 8080) without any encryption. The system requirement is to enable e2e-encryption for all Data In-Transit. So here is the problem.
Currently, we have GCE Ingress with TLS enabled using Let's Encrypt to provide the Certificates on Cluster entrance. This is working fine. Our Ingress has some Path Rules to redirect the traffic to the correct microservice and those microservices are not using TLS on the communication.
I managed to create a Self-Signed certificate and embedded it inside the WAR and this is working on the Local machine just fine (using certificate validation disabled). When I deploy this on GKE, the GCP Health Check and Kubernetes Probes are not working at all (I can't see any communication attempt on the Application logs).
When I try to configure the Backend and Health Check on GCP changing both to HTTPS, they don't show any error, but after some time they quietly switch back to HTTP.
Here are my YAML files:
admin-service.yaml
---
apiVersion: v1
kind: Service
metadata:
name: admin-service
namespace: default
spec:
type: NodePort
selector:
app: admin
ports:
- port: 443
targetPort: 8443
name: https
protocol: TCP
admin-deployment.yaml
---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
name: "admin"
namespace: "default"
labels:
app: "admin"
spec:
replicas: 1
selector:
matchLabels:
app: "admin"
template:
metadata:
labels:
app: "admin"
spec:
containers:
- name: "backend-admin"
image: "gcr.io/my-project/backend-admin:X.Y.Z-SNAPSHOT"
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8443
scheme: HTTPS
initialDelaySeconds: 8
periodSeconds: 30
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8443
scheme: HTTPS
initialDelaySeconds: 8
periodSeconds: 30
env:
- name: "FIREBASE_PROJECT_ID"
valueFrom:
configMapKeyRef:
key: "FIREBASE_PROJECT_ID"
name: "service-config"
---
apiVersion: "autoscaling/v2beta1"
kind: "HorizontalPodAutoscaler"
metadata:
name: "admin-etu-vk1a"
namespace: "default"
labels:
app: "admin"
spec:
scaleTargetRef:
kind: "Deployment"
name: "admin"
apiVersion: "apps/v1"
minReplicas: 1
maxReplicas: 3
metrics:
- type: "Resource"
resource:
name: "cpu"
targetAverageUtilization: 80
ingress.yaml
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: my-ingress-addr
kubernetes.io/ingress.class: "gce"
kubernetes.io/tls-acme: "true"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
acme.cert-manager.io/http01-edit-in-place: "true"
kubernetes.io/ingress.allow-http: "false"
spec:
tls:
- hosts:
- my-domain.com
secretName: mydomain-com-tls
rules:
- host: my-domain.com
http:
paths:
- path: /admin/v1/*
backend:
serviceName: admin-service
servicePort: 443
status:
loadBalancer:
ingress:
- ip: XXX.YYY.WWW.ZZZ
Reading this document from GCP I understood that Loadbalancer it's compatible with Self-signed certificates.
I would appreciate any insight or new directions you guys can provide.
Thanks in advance.
EDIT 1: I've added here the ingress YAML file which may help to a better understanding of the issue.
EDIT 2: I've updated the deployment YAML with the solution I found for liveness and readiness probes (scheme).
EDIT 3: I've found the solution for GCP Health Checks using annotation on Services declaration. I will put all the details on the response to my own question.
Here is what I found on how to fix the issue.
After reading a lot of documentation related to Kubernetes and GCP I found a document on GCP explaining to use annotations on Service declaration. Take a look at lines 7-8.
---
apiVersion: v1
kind: Service
metadata:
name: admin-service
namespace: default
annotations:
cloud.google.com/app-protocols: '{"https":"HTTPS"}'
spec:
type: NodePort
selector:
app: iteam-admin
ports:
- port: 443
targetPort: 8443
name: https
protocol: TCP
This will hint GCP to create the backend-service and health-check using HTTPS and everything will work as expected.
Reference: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-xlb#https_tls_between_load_balancer_and_your_application

How can I make one route public in kubernetes ingress with auth-url set?

I am playing a little bit with path-based routing via Kubernetes ingresses (we use Nginx ingress). We have one service up and running behind authentication via external service like the one in the first snippet. This service is already used by other services in our ecosystem, however, as the development progresses, the service now has management API and we need to turn off the authentication for specific paths. According to this answer, this is not possible for basic auth and, according to what I have tried (second snippet), it is not possible for authentication via external service either. Is there any other way to allow specific paths without authentication without completely rearchitecting ingresses and domains/paths?
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: path-based-routing-1
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: nginx-external
nginx.ingress.kubernetes.io/auth-url: http://authorizer.dev.svc.cluster.local
spec:
tls:
- hosts:
- 'routing.example.com'
secretName: routing.example.com
rules:
- host: 'routing.example.com'
http:
paths:
- path: /
backend:
serviceName: path-based-routing-1
servicePort: 80
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: path-based-routing-2
annotations:
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/auth-type: ""
kubernetes.io/ingress.class: nginx-external
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- 'routing.example.com'
secretName: routing.example.com
rules:
- host: 'routing.example.com'
http:
paths:
- path: /management-api
backend:
serviceName: path-based-routing-2
servicePort: 80

Paths are not working in Kubernetes NGINX Ingress Controller

I have a Spring Boot application responsible for getting citizens and a NGINX Ingress controller configured with ssl-passthrough to expose my cluster to outside.
When I do:
https://myhostname.com/citizens
It perfectly works. This is the configuration I am using:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: myhostname.com
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 443
But I would like to have something like:
https://myhostname.com/myservice/citizens
It would be something similar to math Simple fanout approach that is described in the kubernetes docs:
https://kubernetes.io/docs/concepts/services-networking/ingress/#simple-fanout
Therefore, in my Spring application I have configured a contextPath so when locally I do:
https://localhost:8080/myservice/citizens
I get the desired result
For NGINX I have set up the following:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: myhostname.com
http:
paths:
- path: /myservice
backend:
serviceName: myservice
servicePort: 443
What am I doing wrong?
Solution:
I've been researching a little and I've realized that the approach that I was trying to achieve is not possible with ssl-passthrough as the proxy is blind and it doesn't know the path to where route the traffic.
Therefore, the solution would be to use subdomains and not paths, because thanks to the SNI protocol, the NGINX controller will know to which hostname the client is trying to connect.
The final solution would be something similar to this
If your backend is expecting the whole path /myservice/citizens you can just use it to point it to the service without the rewrite-target directive, since it only dictates how URIs are going to be treated in the backend:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: myhostname.com
http:
paths:
- path: /myservice/citizens
backend:
serviceName: myservice
servicePort: 443
Be aware that this assumes that your service is able to redirect requests to 8080, which is your backend listening port, so it has to match it.

TLS setup on K8S Ingress with Traefik

I have a setup that is not too much different than the user guide for use with k8s. For some reason I can only access http://app.minikube and not https://app.minikube.
Can someone look at my setup and see what I am obviously missing?
apiVersion: v1
kind: Service
metadata:
name: myapp
labels:
app: myapp
spec:
ports:
- name: http
port: 80
targetPort: 7777
selector:
app: myapp
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myingress
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: app.minikube
http:
paths:
- path: /
backend:
serviceName: myapp
servicePort: http
tls:
- secretName: mytls
FYI, according to the Traefik user guide, the hosts definition in tls is unneeded, which is why I left it out.
The field hosts in the TLS configuration is ignored. Instead, the domains provided by the certificate are used for this purpose. It is recommended to not use wildcard certificates as they will match globally)
You're missing the hosts section:
tls:
- hosts:
- my-host.example.com
secretName: my-secret

Running Nexus in Kubernetes using ingress with path other than /

I have trouble running Nexus 3 in Kubernetes via ingress, when I specify a path other than "/". Nexus does not load fully when i visit the web portal as https://www.myportal.com/mypath. I have a true certificate. This is my ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/rewrite-target: /
ingress.kubernetes.io/add-base-url: "true"
name: myingress
spec:
rules:
- host: mynexus.com
http:
paths:
- path: /mypath
backend:
serviceName: mynexus-sonatype-nexus
servicePort: 9988
tls:
- hosts:
- mynexus.com
secretName: mynexus-cert-secret