I'm working on an application that deploys kubernetes resources dynamically, and I'd like to be able to provision a shared SSL certificate for all of them. At any given time, all of the services have the path *.*.*.example.com.
I've heard that cert-manager will provision/re-provision certs automatically, but I don't necessarily need auto-provisioning if its too much overhead. The solution also needs to be able to handle these nested url subdomains.
Any thoughts on the easiest way to do this?
Have a look at nginx-ingress, which is a Kubernetes Ingress Controller that essentially makes it possible to run Nginx reverse proxy/web server/load balancer on Kubernetes.
nginx-ingress is built around the Ingress resource. It will watch Ingress objects and manage nginx configuration in config maps. You can define powerful traffic routing rules, caching, url rewriting, and a lot more via the Kubernetes Ingress resource rules and nginx specific annotations.
Here's an example of an Ingress with some routing. There's a lot more you can do with this, and it does support wildcard domain routing.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
cert-manager.io/cluster-issuer: letsencrypt-prod
name: my-ingress
spec:
rules:
- host: app1.domain.com
http:
paths:
- backend:
serviceName: app1-service
servicePort: http
path: /(.*)
- host: app2.sub.domain.com
http:
paths:
- backend:
serviceName: app2-service
servicePort: http
path: /(.*)
tls:
- hosts:
- app1.domain.com
secretName: app1.domain.com-tls-secret
- hosts:
- app2.sub.domain.com
secretName: app2.sub.domain.com-tls-secret
The annotations section is really important. Above indicates that nginx-ingress should manage this Ingress definition. This annotations section allows to specify additional nginx configuration, in the above example it specifies a url rewrite target that can be used to rewrite urls in the rule section.
See this community post for installing nginx-ingress on GKE.
You'll notice the annotations also have a cert manager specific annotation which, if installed will instruct cert manager to issue certificates based on the hosts and secrets defined under the tls section.
Using cert-manager in combination with nginx-ingress, which isn't that complicated, you can set up automatic certificate creation/renewals.
It's hard to know the exact nature of your setup with deploying dynamic applications. But some possible ways to achieve the configuration are:
Have each app define it's own Ingress with it's own routing rules and TLS configuration, which gets installed/updated each time your the application is deployed
Have an Ingress per domain/subdomain. You could then specify a wild card subdomain and tls section with routing rules for that subdomain
Or possibly you could have one uber Ingress which handles all domains and routing.
The more fine grained the more control, but a lot more moving parts. I don't see this as a problem. For the last two options, it really depends on the nature of your dynamic application deployments.
Related
I'm trying to setup a route do a basic 301 redirect with the added benefit of supporting both HTTP and HTTPS requests. Expected results would be that requests to http://subdom.domain.org or https://subdom.domain.org would receive a 301 and be forwarded to https://othersub.domain.org/route. Actual results is that https://subdom.domain.org 301's as expected, but http://subdom.domain.org 404's. With the config, you can see I've tried doing both an elevate to HTTPS with the hopes that the rule might be caught there, but with the way the middleware is configured, I would expect it would work in either scenario.
Here's my current config:
http:
routers:
subdom.domain.org:
entryPoints:
- web
- web-secure
middlewares:
- https-redirect # I've tried with this on and off
- sudbdom-redirect
service: dummy
rule: Host(`subdom.domain.org`)
tls:
certResolver: letsEncryptResolver
middlewares:
https-redirect:
redirectScheme:
scheme: https
subdom-redirect:
redirectRegex:
regex: ".*"
replacement: "https://othersub.domain.org/route"
permanent: true
services:
dummy:
loadBalancer:
servers:
- url: localhost
I was originally having trouble matching specific regex patterns for the redirect, but in realizing I didn't really need to scope the pattern at all given that I'm applying it per route, a wildcard match seems to work quite well there. Any thoughts or suggestions are appreciated. Thanks!
You should try to make 2 different router one for web entry point where you can perform redirection and 2rd one for redirectRegex where you can redirect your application to different url.
Following TLS section from official documentation:
When a TLS section is specified, it instructs Traefik that the current router is dedicated to HTTPS requests only (and that the router should ignore HTTP (non TLS) requests).
Solutions:
1. Define separate routers (for http and https), but it noisy for multiple services.
If you need to define the same route for both HTTP and HTTPS requests, you will need to define two different routers: one with the tls section, one without.
Example from documentation
2. Disable TLS for router
traefik.http.routers.YOUR-SERVICE.tls=false
Example:
services:
mailhog:
image: mailhog/mailhog
networks:
- traefik-public
deploy:
placement:
constraints:
- node.role != manager
labels:
- "traefik.enable=true"
- "traefik.http.routers.mailhog.rule=Host(`mailhog.somehost.ru`)"
- "traefik.http.routers.mailhog.service=mailhog"
- "traefik.http.routers.mailhog.entrypoints=web,websecure"
# Should be false
- "traefik.http.routers.mailhog.tls=false"
- "traefik.http.middlewares.redirectos.redirectscheme.scheme=https"
- "traefik.http.routers.mailhog.middlewares=redirectos"
- "traefik.http.services.mailhog.loadbalancer.server.port=8025"
- "traefik.tags=traefik-public"
- "traefik.docker.network=traefik-public"
I am trying to use the path based routing mechanism provided by Traefik ingress controller in Kubernetes but I have some issues with the url rewriting.
My [UPDATED] configuration is as follow
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/auth-type: "basic"
traefik.ingress.kubernetes.io/auth-tls-insecure: "true"
traefik.ingress.kubernetes.io/frontend-entry-points: "http,https"
traefik.ingress.kubernetes.io/app-root: "/"
traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
traefik.ingress.kubernetes.io/rewrite-target: "/"
name: webapp-ingress
namespace: my-company
spec:
rules:
- host: local-ubuntu
- http:
paths:
- path: /
backend:
serviceName: webapp
servicePort: 80
- path: /db
backend:
serviceName: db-manager
servicePort: 8081
The traffic is routed to the right services but the url is still prefixed with /db when I look at the log for the db-manager (kubernetes) service.
What I would have expected with the PathPrefixStrip is that the traffic will be routed without the /db prefix to the container running the db-manager micro-service which is listening on / (http://db-manager:8081) on the backend side.
Am I missing something ? Is it supported by traefik or only nginx ?
Thank you by advance for your feedback.
[EDIT]
To be more specific I observe the following with the current annotations discussed below
traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
traefik.ingress.kubernetes.io/rewrite-target: "/"
URL: http://local-ubuntu/db [OK] -> 200
Then other resources are loading but are pointing on the wrong base url
Example:
Resource URL is : http://local-ubuntu/public/css/bootstrap.min.css
But this should be : http://local-ubuntu/db/public/css/bootstrap.min.css
(which works when I've tried manually)
I am not sure what I am missing here in the current configuration.
Regarding the static contents not being served, the documentation states the following:
Use a *Strip matcher if your backend listens on the root path (/) but should be routeable on a specific prefix. For instance, PathPrefixStrip: /products would match /products but also /products/shoes and /products/shirts.
Since the path is stripped prior to forwarding, your backend is expected to listen on /.
If your backend is serving assets (e.g., images or Javascript files), chances are it must return properly constructed relative URLs.
Continuing on the example, the backend should return /products/shoes/image.png (and not /images.png which Traefik would likely not be able to associate with the same backend).
The X-Forwarded-Prefix header (available since Traefik 1.3) can be queried to build such URLs dynamically.
Thank you very much for your help in this matter.
First of all I had to fix an issue regarding the formatting of the annotations in the yaml file.
All the instructions with traefik as a prefix need to be double quoted
Example :
traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip [Not
correct]
traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
[correct]
In the first case none of the annotations were reflected in the ingress.
But I still cannot route properly the traffic.
With the current configuration only the resource served on / is returned.
None of the js, css or other resources are loaded.
So I wonder if I need to use the traefik.frontend.redirect.regex instruction.
Try with one of the following:
traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
traefik.ingress.kubernetes.io/rewrite-target: "/
They both achieve similar results, but they are different, and they have slightly different behavior.
I would read more on our documentation for the differences: (https://docs.traefik.io/v1.7/configuration/backends/kubernetes/#general-annotations)
As for your second issue:
Resource URL is : local-ubuntu/public/css/bootstrap.min.css
But this should be : local-ubuntu/db/public/css/bootstrap.min.css (which works when I've tried
You stripped that path from the request...your DB service never sees the DB prefix...How is it supposed to know to add them back in?
You need to set a root URL in your web application to handle the stripped path.
Once you do that, you may not even need to strip the path at all, and just leave it as is. If you cannot set a base URL for your application, you may not be able to use directories for routing, and may have to use subdomains instead.
use only traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip
Bellow what I used to send only subpath to my k8s pods
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: global-ingress
namespace: app
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip
I'm having some issues when using a path to point to a different
Kubernetes service.
I'm pointing to a secondary service using the path /secondary-app and I can see through my logs that I am correctly reaching that service.
My issue is that any included resource on the site, let's say /css/main.css for example, end up not found resulting in a 404.
Here's a slimmed down version of my ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/rewrite-target: /
name: my-app
spec:
rules:
- host: my-app.example.com
http:
paths:
- backend:
path: /
serviceName: my-app
servicePort: http
- backend:
path: /secondary-app
serviceName: secondary-app
servicePort: http
I've tried a few things and haven't yet been able to make it work. Do I maybe need to do some apache rewrites?
Any help would be appreciated.
Edit - Solution
Thanks to some help from #mk_sta I was able to get my secondary service application working by using the nginx.ingress.kubernetes.io/configuration-snippet annotation like so:
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($request_uri = '/?%secondary-app') { rewrite /(.*) secondary-app/$1 break; }
It still needs a bit of tweaking for my specific app but that worked exactly how I needed it to.
I guess nginx.ingress.kubernetes.io/rewrite-target: / annotation in your Ingress configuration doesn't bring any success for multipath rewrite target paths, read more here. However, you can consider to use Nginx Plus Ingress controller, shipped with nginx.org/rewrites: annotation and can be used for pointing URI paths to multiple services as described in this example.
You can also think about using nginx.ingress.kubernetes.io/configuration-snippet annotation for the existing Ingress, which can adjust rewrite rules to Nginx location, something like:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/configuration-snippet: |
rewrite /first-app/(.*) $1 break;
rewrite /secondary-app/(.*) /$1 break;
name: my-app
spec:
rules:
- host: my-app.example.com
http:
paths:
- backend:
path: /first-app
serviceName: my-app
servicePort: http
- backend:
path: /secondary-app
serviceName: secondary-app
servicePort: http
Traefik ingress controller has been supporting traefik.frontend.rule.type: PathPrefixStrip for quite some time, which is useful when a root path of a microservice needs to be available at example.com/path/.
Here is how an example yaml with a manifest looks like:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example
annotations:
traefik.frontend.rule.type: PathPrefixStrip
spec:
rules:
- host: example.com
http:
paths:
- path: /path/
backend:
serviceName: example
servicePort: http
The problem with this approach is that it does not add a trailing slash when a client goes to example.com/path – if I understand correctly, this can only be achieved with an extra ingress rule.
More recent versions of traefik support a wider set of annotations, which suggests that the addition of a trailing slash may be declared inside just one rule.
Here is my attempt to solve this in traefik 1.7:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example
annotations:
traefik.ingress.kubernetes.io/redirect-permanent: "true"
traefik.ingress.kubernetes.io/redirect-regex: https?://example.com/path$
traefik.ingress.kubernetes.io/redirect-replacement: https://example.com/path/
traefik.ingress.kubernetes.io/request-modifier: "ReplacePathRegex: ^/path/(.*) /$1"
spec:
rules:
- host: example.com
http:
paths:
- path: /path
backend:
serviceName: example
servicePort: http
This works, but I'm not sure if the solution is the most elegant and performant. What could be simplified or improved? Is it possible to generalize the regexps to make copy-pasting easier?
Here's the goal, just to recap:
http://example.com/path → 301 to http://example.com/path/
http://example.com/path/ → example microservice, path /
http://example.com/path/abcde → example microservice, path /abcde
The closest tutorial I can find in getting an SSL terminating Ingress and an nginx based controller running on bare metal (Digital Ocean, for example) is this:
https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx
but it leaves so many assumptions unexplained.
My ingress requirements are simply:
default backend at port 80 for all hosts that:
file access to location ^~ /.well-known/acme-challenge/ which allows my LetsEncrypt cert renewals to work
404 on location /.well-known/acme-challenge/
301 on location /
subdomain based routing to different backend services on port 443
each subdomain points to a different SSL key/cert (generated by my LetsEncrypt, and stored in K8S as a secret I suppose??)
What I think need is this:
full documentation on writing Ingress rules
can I configure SSL certs (on port 443) for each backend individually?
is / the "path" that's a catchall for a host?
updating Ingress rules in place
what nginx controller do I use? nginx? nginx-alpha? nginx-ingress docker container -- and where is the documentation for each of these controllers?
is there a base controller image that I can override the nginx.conf template that gets populated by Ingress changes from the API server?
how do you store SSL keys and certs as secrets?
boo my answers apply to https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx
default backend at port 80 for all hosts that:
404 on location /.well-known/acme-challenge/
this is not possible using Ingress rules
301 on location /
This is already supported. If the server contains a SSL certificate it will redirect to httpsautomatically
subdomain based routing to different backend services on port 443
each subdomain points to a different SSL key/cert (generated by my LetsEncrypt, and stored in K8S as a secret I suppose??)
You need to create multiple Ingress rules, one per subdomain. Each rule can use a different secret name (this will create multiple servers, one per subdomain)
What I think need is this:
full documentation on writing Ingress rules
http://kubernetes.io/docs/user-guide/ingress/
(I don't know id there's additional information besides the go code)
can I configure SSL certs (on port 443) for each backend individually?
is / the "path" that's a catchall for a host?
yes
updating Ingress rules in place
what nginx controller do I use? nginx? nginx-alpha? nginx-ingress docker container -- and where is the documentation for each of these controllers?
This depends on what you need, if you want to build you custom Ingress controller you can use nginx-alpha as reference. If nginx-ingress is not clear in the examples please open an issue and mention what could be improved in the examples or it's missing
is there a base controller image that I can override the nginx.conf template that gets populated by Ingress changes from the API server?
No. The reason for this is that the template is tied to the go code that populates the template. That said, you can build a custom image changing the template but this requires you deploy the image to tests the changes
how do you store SSL keys and certs as secrets?
yes, as secrets like this http://kubernetes.io/docs/user-guide/ingress/#tls
For the letsencrypt support please check this comment https://github.com/kubernetes/kubernetes/issues/19899#issuecomment-184059009
Here is a complete example https://gist.github.com/aledbf/d88c7f7d0b8d4d032035b14ab0965e26 added to examples in #766