Installing 2 custom TLS Certs with Traefik / K8s - ssl

I have a K8s cluster working well, with a domain that has a custom SSL Cert, and all other subdomain use a Let's encrypt wildcard.
Now, I want to add another domain that will use custom SSL Cert, but it doesn't seem to work.
First, I create a configMap with .crt and .key files
kubectl create configmap traefik-sge-certificate --from-file=certificate/sge-prod.crt --from-file=certificate/sge-prod.key --dry-run -o yaml | kubectl apply -f -
Then I add my certs in traefik.toml
traefik.toml:
----
# traefik.toml
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "/etc/ene-certificate/api.crt"
keyFile = "/etc/ene-certificate/api.key"
[[entryPoints.https.tls.certificates]]
certFile = "/etc/sge-certificate/sge-prod.crt"
keyFile = "/etc/sge-certificate/sge-prod.key"
[acme] # Automatically add Let's Encrypt Certificate.
storage= "/etc/certificate/acme.json"
email = "julien#company.fr"
entryPoint = "https"
onHostRule = true
caServer = "https://acme-v02.api.letsencrypt.org/directory"
[acme.dnsChallenge]
provider = "route53"
delayBeforeCheck = 0
[[acme.domains]]
main = "*.company.fr"
And now deployment:
kind: Deployment
apiVersion: apps/v1
metadata:
name: traefik-deployment
labels:
app: traefik
spec:
replicas: 1
selector:
matchLabels:
app: traefik
template:
metadata:
labels:
app: traefik
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
volumes:
- name: traefik-certificate
persistentVolumeClaim:
claimName: traefik-certificate
- name: config
configMap:
name: traefik-config
- name: traefik-ene-certificate
configMap:
name: traefik-ene-certificate
- name: traefik-sge-certificate
configMap:
name: traefik-sge-certificate
containers:
- name: traefik
image: "traefik:1.7"
envFrom:
- secretRef:
name: traefik-env
volumeMounts:
- mountPath: "/etc/traefik/config"
name: config
- mountPath: "/etc/certificate"
name: traefik-certificate
- mountPath: "/etc/ene-certificate/api.crt"
name: traefik-ene-certificate
subPath: api.crt
- mountPath: "/etc/ene-certificate/api.key"
name: traefik-ene-certificate
subPath: api.key
- mountPath: "/etc/sge-certificate/sge-prod.crt"
name: traefik-sge-certificate
subPath: sge-prod.crt
- mountPath: "/etc/sge-certificate/sge-prod.key"
name: traefik-sge-certificate
subPath: sge-prod.key
args:
- --configfile=/etc/traefik/config/traefik.toml
- --api
- --kubernetes
But when I do that, there is 2 problems:
My new app: https://sge.company.fr is using wildcard certificates, instead of custom ones ( the purpose of this operation)
The app that was using the custom certificates api.crt (previously working well with custom SSL cert ) are not using wildcard certs, this is a regression.
I checked the traefik logs, and it doesn't seem to have any issue.
What I also don't understand, is where is linked the custom SSL Cert with domain.
Can anybody tell me why this is happening, and how should I fix it ?
PD: I understand I should use secrets instead of configMaps, but first thing first !

What I also don't understand, is where is linked the custom SSL Cert with domain.
The Cert is issued for a specific domain. So, here is the link.
If you issue your cert for domain1.com and then you try to use it on domain2.com, Traefik will not be able to install it.
My error was to use another domain that the one attached to my certificates.
So, the config is OK, and now I can access the domain with my custom certs
Hope it helps others

Related

Kubernetes tls for tcp subdomain

I am trying to create TCP entrypoint with Traefik v2. I have following in traefik.toml
[global]
checkNewVersion = true
[entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.websecure]
address = ":443"
[entryPoints.tcp5050]
address = ":5050"
and following IngressRouteTCP
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: mosquitto-external
namespace: dev
spec:
entryPoints:
- tcp5050
routes:
- match: HostSNI(`*`)
kind: Rule
services:
- name: mosquitto
port: 5050
tls:
Above one works perfectly. But can't reuse entryPoints tcp5050 for any other host. I tried to make IngressRouteTCP specific to HostSNI like this
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: mosquitto-external
namespace: dev
spec:
entryPoints:
- tcp5050
routes:
- match: HostSNI(`mosquitto.dev.domain.com`)
kind: Rule
services:
- name: mosquitto
port: 5050
tls:
passthrough: true
But it this doesn't work at all.
What I am missing here ?

Traefik V2 get wildcard certificate

I have basic setup running after following this tutorial
But struggling to get wildcard certificate for domain from Let's Encrypt.
traefik configuration :
traefik.toml: |
## static configuration
[global]
checkNewVersion = true
[entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.websecure]
address = ":443"
[providers]
[providers.kubernetesCRD]
[providers.file]
directory = "/etc/traefik/providers/"
watch = true
[log]
level = "INFO"
[accessLog]
[api]
insecure = true
dashboard = true
debug = true
[metrics]
[metrics.prometheus]
buckets = [0.1,0.3,1.2,5.0]
addEntryPointsLabels = true
addServicesLabels = true
entryPoint = "web"
[ping]
entryPoint = "web"
[certificatesResolvers]
[certificatesResolvers.default]
[certificatesResolvers.default.acme]
email = "admin#domain.com"
caServer = "https://acme-v02.api.letsencrypt.org/directory"
storage = "acme.json"
[certificatesResolvers.default.acme.dnsChallenge]
provider = "route53"
delayBeforeCheck = 0
resolvers = ["1.1.1.1:53", "8.8.8.8:53"]
dynamic.toml: |
## dynamic configuration
(Empty)
and route config :
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: traefik-admin
namespace: kube-system
spec:
entryPoints:
- web
routes:
- match: Host(`traefik.domain.ca`)
kind: Rule
services:
- name: traefik
port: 8080
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: whoami-notls
namespace: kube-system
spec:
entryPoints:
- web
routes:
- match: Host(`traefik.domain.ca`) && PathPrefix(`/notls`)
kind: Rule
services:
- name: whoami
port: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: whoami-tls
namespace: kube-system
spec:
entryPoints:
- websecure
routes:
- match: Host(`traefik.domain.ca`) && PathPrefix(`/tls`)
kind: Rule
services:
- name: whoami
port: 80
tls:
certResolver: default
I am able to get certificate for traefik.domain.ca but need to get wildcard certificate for entire domain (*.domain.ca). I am not able find any direct reference config for this.
What I am missing here ?
Updating the tls block worked.
tls:
certResolver: default
domains:
- main: dev.domain.ca
sans:
- "dev.domain.ca"
- "*.dev.domain.ca"

Can't get real user's IP from X-Forwarded-For

I'm running Traefik 1.7.3 on a single node Kubernetes cluster and I'm trying to get the real user IP from the X-Forwarded-For header but what I get instead is X-Forwarded-For: 10.244.0.1 which is an IP in my k8s cluster.
Here's my Traefik deployment and service:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: traefik-conf
data:
traefik.toml: |
# traefik.toml
debug = true
logLevel = "DEBUG"
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
compress = true
[entryPoints.http.forwardedHeaders]
trustedIPs = [ "0.0.0.0/0" ]
entryPoint = "https"
[entryPoints.https]
address = ":443"
compress = true
[entryPoints.https.forwardedHeaders]
trustedIPs = [ "0.0.0.0/0" ]
[entryPoints.https.tls]
[acme]
email = "xxxx"
storage = "/acme/acme.json"
entryPoint = "https"
onHostRule = true
#caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
acmeLogging = true
[[acme.domains]]
main = "xxxx"
[acme.dnsChallenge]
provider = "route53"
delayBeforeCheck = 0
[persistence]
enabled = true
existingClaim = "pvc0"
annotations = {}
accessMode = "ReadWriteOnce"
size = "1Gi"
[kubernetes]
namespaces = ["default"]
[accessLog]
filePath = "/acme/access.log"
[accessLog.fields]
defaultMode = "keep"
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: traefik-ingress-controller
namespace: default
labels:
k8s-app: traefik-ingress-lb
spec:
replicas: 1
selector:
matchLabels:
k8s-app: traefik-ingress-lb
template:
metadata:
labels:
k8s-app: traefik-ingress-lb
name: traefik-ingress-lb
spec:
serviceAccountName: traefik-ingress-controller
terminationGracePeriodSeconds: 60
containers:
- image: traefik
name: traefik-ingress-lb
env:
- name: AWS_ACCESS_KEY_ID
value: xxxx
- name: AWS_SECRET_ACCESS_KEY
value: xxxx
- name: AWS_REGION
value: us-west-2
- name: AWS_HOSTED_ZONE_ID
value: xxxx
ports:
- name: http
containerPort: 80
- name: admin
containerPort: 8080
args:
- --api
- --kubernetes
- --configfile=/config/traefik.toml
volumeMounts:
- mountPath: /config
name: config
- mountPath: /acme
name: acme
volumes:
- name: config
configMap:
name: traefik-conf
- name: acme
persistentVolumeClaim:
claimName: "pvc0"
---
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
namespace: default
spec:
externalIPs:
- x.x.x.x
externalTrafficPolicy: Local
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
name: web
- protocol: TCP
port: 443
name: https
- protocol: TCP
port: 8080
name: admin
type: NodePort
And here's my ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: headers-test
namespace: default
annotations:
ingress.kubernetes.io/proxy-body-size: 500m
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: xxxx
http:
paths:
- path: /
backend:
serviceName: headers-test
servicePort: 8080
I'd read that I only needed to add [entryPoints.http.forwardedHeaders] and a list of trustedIPs but that doesn't seem to work. Am I missing something?
If you use NodePort for the Traefik Ingress Service, you will have to set service.spec.externalTrafficPolicy to "Local". Otherwise you will have a SNAT when your connection enters the K8s-cluster. This SNAT is necessary to forward the incoming connection to your pod if it is not running on the same node.
But be aware that having set service.spec.externalTrafficPolicy to "Local", only the node on which the Traefik pod is executed will accept requests on 80, 443, 8080. There is no forwarding to the pod from the other nodes anymore. This can result in odd delays when connecting to your service. To avoid that your Traefik would need to run in a HA setup (DaemonSet). Just keep in mind that you need a K/V-Store for a distributed Traefik setup to make Letsencrypt work well.
If the service.spec.externalTrafficPolicy setting does not yet resolve your problem you might also need to configure the kubernetes overlay network to not do any SNAT.
service.spec.externalTrafficPolicy is nicely explained here:
https://kubernetes.io/docs/tutorials/services/source-ip/

Can't get kubernetes to pass my tls certificate to browsers

I've been struggling for a while trying to get HTTPS access to my Elasticsearch cluster in Kubernetes.
I think the problem is that Kubernetes doesn't like the TLS certificate I'm trying to use, which is why it's not passing it all the way through to the browser.
Everything else seems to work, since when I accept the Kubernetes Ingress Controller Fake Certificate, the requests go through as expected.
In my attempt to do this I've set up:
The cluster itself
An nginx-ingress controller
An ingress resource
Here's the related yaml:
Cluster:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-08-03T03:20:47Z
labels:
run: my-es
name: my-es
namespace: default
resourceVersion: "3159488"
selfLink: /api/v1/namespaces/default/services/my-es
uid: 373047e0-96cc-11e8-932b-42010a800043
spec:
clusterIP: 10.63.241.39
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 9200
selector:
run: my-es
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
The ingress resource
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS
nginx.ingress.kubernetes.io/cors-origins: http://localhost:3425 https://mydomain.ca
https://myOtherDomain.ca
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
creationTimestamp: 2018-08-12T08:44:29Z
generation: 16
name: es-ingress
namespace: default
resourceVersion: "3159625"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/es-ingress
uid: ece0071d-9e0b-11e8-8a45-42001a8000fc
spec:
rules:
- http:
paths:
- backend:
serviceName: my-es
servicePort: 8080
path: /
tls:
- hosts:
- mydomain.ca
secretName: my-tls-secret
status:
loadBalancer:
ingress:
- ip: 130.211.179.225
The nginx-ingress controller:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-08-12T00:41:32Z
labels:
app: nginx-ingress
chart: nginx-ingress-0.23.0
component: controller
heritage: Tiller
release: nginx-ingress
name: nginx-ingress-controller
namespace: default
resourceVersion: "2781955"
selfLink: /api/v1/namespaces/default/services/nginx-ingress-controller
uid: 755ee4b8-9dc8-11e8-85a4-4201a08000fc
spec:
clusterIP: 10.63.250.256
externalTrafficPolicy: Cluster
ports:
- name: http
nodePort: 32084
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 31182
port: 443
protocol: TCP
targetPort: https
selector:
app: nginx-ingress
component: controller
release: nginx-ingress
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 35.212.6.131
I feel like I'm missing something basic, because it doesn't seem like it should be this hard to expose something this simple...
To get my certificate, I just requested one for mydomain.ca from godaddy.
Do I need to somehow get a certificate using my ingress resource's cluster IP as the common name?
It doesn't seem possible to verify ownership of an IP.
I've seen people mention ways for Kubernetes to automatically create certificates for ingress resources, but those seem to be self signed.
Here are some logs from the nginx-controller:
This one is talking about a PEM with the tls-secret, but it's only a warning.
{
insertId: "1kvvhm7g1q7e0ej"
labels: {
compute.googleapis.com/resource_name: "fluentd-gcp-v2.0.17-5b82n"
container.googleapis.com/namespace_name: "default"
container.googleapis.com/pod_name: "nginx-ingress-controller-58f57fc597-zl25s"
container.googleapis.com/stream: "stderr"
}
logName: "projects/project-7d320/logs/nginx-ingress-controller"
receiveTimestamp: "2018-08-14T02:58:42.135388365Z"
resource: {
labels: {
cluster_name: "my-elasticsearch-cluster"
container_name: "nginx-ingress-controller"
instance_id: "2341889542400230234"
namespace_id: "default"
pod_id: "nginx-ingress-controller-58f57fc597-zl25s"
project_id: "project-7d320"
zone: "us-central1-a"
}
type: "container"
}
severity: "WARNING"
textPayload: "error obtaining PEM from secret default/my-tls-cert: error retrieving secret default/my-tls-cert: secret default/my-tls-cert was not found"
timestamp: "2018-08-14T02:58:37Z"
}
I have a few occurences of this handshake error, which may be a result of the last warning...
{
insertId: "148t6rfg1xmz978"
labels: {
compute.googleapis.com/resource_name: "fluentd-gcp-v2.0.17-5b82n"
container.googleapis.com/namespace_name: "default"
container.googleapis.com/pod_name: "nginx-ingress-controller-58f57fc597-zl25s"
container.googleapis.com/stream: "stderr"
}
logName: "projects/project-7d320/logs/nginx-ingress-controller"
receiveTimestamp: "2018-08-14T15:55:52.438035706Z"
resource: {
labels: {
cluster_name: "my-elasticsearch-cluster"
container_name: "nginx-ingress-controller"
instance_id: "2341889542400230234"
namespace_id: "default"
pod_id: "nginx-ingress-controller-58f57fc597-zl25s"
project_id: "project-7d320"
zone: "us-central1-a"
}
type: "container"
}
severity: "ERROR"
textPayload: "2018/08/14 15:55:50 [crit] 1548#1548: *860 SSL_do_handshake() failed (SSL: error:1417D18C:SSL routines:tls_process_client_hello:version too low) while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:442"
timestamp: "2018-08-14T15:55:50Z"
}
The above logs make it seem like my tls secret isnt working, but when I run kubectl describe ingress, it says my secret terminates.
aaronmw#project-7d320:~$ kubectl describe ing
Name: es-ingress
Namespace: default
Address: 130.221.179.212
Default backend: default-http-backend:80 (10.61.3.7:8080)
TLS:
my-tls-secret terminates mydomain.ca
Rules:
Host Path Backends
---- ---- --------
*
/ my-es:8080 (<none>)
Annotations:
Events: <none>
I figured it out!
What I ended up doing was adding a default ssl certificate to my nginx-ingress controller on creation using the following command
helm install --name nginx-ingress --set controller.extraArgs.default-ssl-certificate=default/search-tls-secret stable/nginx-ingress
Once I had that, it was passing the cert as expected, but I still had the wrong cert as the CN didn't match my load balancer IP.
So what I did was:
Make my load balancer IP static
Add an A record to my domain, to map a subdomain to that IP
Re-key my cert to match that new subdomain
And I'm in business!
Thanks to #Crou, who's comment reminded me to look at the logs and got me on the right track.

Trouble at configuring http(s) for an nginx-ingress

Im currently trying to create an ingress, following the ssl-termination approach, which allows me to connect to a service both via http and https.
I managed to create a working ingress for http, partly for https, but not both together..
heres my config
Ingress Controller: Deployment & Service
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
spec:
replicas: 1
revisionHistoryLimit: 3
template:
metadata:
labels:
k8s-app: nginx-ingress-lb
spec:
containers:
- args:
- /nginx-ingress-controller
- "--default-backend-service=$(POD_NAMESPACE)/default-http-backend"
env:
<!-- default-config ommitted -->
image: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.17"
imagePullPolicy: Always
livenessProbe:
<!-- omitted -->
name: nginx-ingress-controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
volumeMounts:
- mountPath: /etc/nginx-ssl/tls
name: tls-vol
terminationGracePeriodSeconds: 60
volumes:
- name: tls-vol
secret:
secretName: tls-test-project-secret
---
apiVersion: v1
kind: Service
metadata:
name: nginx-ingress
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: http
nodePort: 31115
- name: https
port: 443
targetPort: https
nodePort: 31116
selector:
k8s-app: nginx-ingress-lb
Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/secure-backends: "false"
# modified this to false for http & https-scenario
ingress.kubernetes.io/ssl-redirect: "true"
# modified this to false for http & https-scenario
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
ingress.kubernetes.io/add-base-url: "true"
spec:
tls:
- hosts:
- author.k8s-test
secretName: tls-test-project-secret
rules:
- host: author.k8s-test
http:
paths:
- path: /
backend:
serviceName: cms-author
servicePort: 8080
Backend - Service
apiVersion: v1
kind: Service
metadata:
name: cms-author
spec:
selector:
run: cms-author
ports:
- name: http
protocol: TCP
port: 8080
targetPort: 8080
Backend-Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: cms-author
spec:
selector:
matchLabels:
run: cms-author
replicas: 1
template:
metadata:
labels:
run: cms-author
spec:
containers:
- name: cms-author
image: <someDockerRegistryUrl>/magnolia:kube-dev
imagePullPolicy: Always
ports:
- containerPort: 8080
I have several issues, when follwing the https only scenario, i can reach the application via the ingress https nodePort, but cant login, as the follwing request goes via http instead of https.. If i put manually https before the url in browser, it is working again and any further request goes via https., but I dont know why :(
The final setting (supporting http and https) is completely not working, as if I try to access the app via http-nodePort of Ingress, it always redirects to ssl, but in this scenario, I configured to ssl-redirect to false, but still not working.
I have read many posts on github, dealing with that, but none of them worked for me
I've changed the nginx-controller images from gce_containers to quay.io, also not working
I've tried some older versions, also not working.
Deploy the nginx ingress controller from the official kubernetes charts repo https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress by setting the helm arguments controller.service.targetPorts.https and controller.service.nodePorts.https. Once they are set, the appropriate NodePort (443) will be configured by helm.
Helm uses the YAML files in https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress/templates.
Along with the nginx ingress controller, you'll need an ingress resource too. Refer https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/complete-example for examples.