To preface this I’m working on the GCE, and Kuberenetes. My goal is simply to expose all microservices on my cluster over SSL. Ideally it would work the same as when you expose a deployment via type=‘LoadBalancer’ and get a single external IP. That is my goal but SSL is not available with those basic load balancers.
From my research the best current solution would be to set up an nginx ingress controller, use ingress resources and services to expose my micro services. Below is a diagram I drew up with my understanding of this process.
I’ve got this all to successfully work over HTTP. I deployed the default nginx controller from here: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx . As well as the default backend and service for the default backend. The ingress for my own micro service has rules set as my domain name and path: /.
This was successful but there were two things that were confusing me a bit.
When exposing the service resource for my backend (microservice) one guide I followed used type=‘NodePort’ and the other just put a port to reach the service. Both set the target port to the backend app port. I tried this both ways and they both seemed to work. Guide one is from the link above. Guide 2: http://blog.kubernetes.io/2016/03/Kubernetes-1.2-and-simplifying-advanced-networking-with-Ingress.html. What is the difference here?
Another point of confusion is that my ingress always gets two IPs. My initial thought process was that there should only be one external ip and that would hit my ingress which is then directed by nginx for the routing. Or is the ip directly to the nginx? Anyway the first IP address created seemed to give me the expected results where as visiting the second IP fails.
Despite my confusion things seemed to work fine over HTTP. Over HTTPS not so much. At first when I made a web request over https things would just hang. I opened 443 on my firewall rules which seemed to work however I would hit my default backend rather than my microservice.
Reading led me to this from Kubernetes docs: Currently the Ingress resource only supports http rules.
This may explain why I am hitting the default backend because my rules are only for HTTP. But if so how am I supposed to use this approach for SSL?
Another thing I noticed is that if I write an ingress resource with no rules and give it my desired backend I still get directed to my original default backend. This is even more odd because kubectl describe ing updated and states that my default backend is my desired backend...
Any help or guidance would be much appreciated. Thanks!
So, for #2, you've probably ended up provisioning a Google HTTP(S) LoadBalancer, probably because you're missing the kubernetes.io/ingress.class: "nginx" annotation as described here: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx#running-multiple-ingress-controllers.
GKE has it's own ingress controller that you need to override by sticking that annotation on your nginx deployment. This article has a good explanation about that stuff.
The kubernetes docs have a pretty good description of what NodePort means - basically, the service will allocate a port from a high range on each Node in your cluster, and Nodes will forward traffic from that port to your service. It's one way of setting up load balancers in different environments, but for your approach it's not necessary. You can just omit the type field of your microservice's Service and it will be assigned the default type, which is ClusterIP.
As for SSL, it could be a few different things. I would make sure you've got the Secret set up just as they describe in the nginx controller docs, eg with a tls.cert and tls.key field.
I'd also check the logs of the nginx controller - find out which pod it's running as with kubectl get pods, and then tail it's logs: kubectl logs nginx-pod-<some-random-hash> -f. This will be helpful to find out if you've misconfigured anything, like if a service does not have any endpoints configured. Most of the time I've messed up the ingress stuff, it's been due to some pretty basic misconfiguration of Services/Deployments.
You'll also need to set up a DNS record for your hostname pointed at the LoadBalancer's static IP, or else ping your service with cURL's -H flag as they do in the docs, otherwise you might end up getting routed to the default back end 404.
To respond directly to your questions, since that's the whole point... Disclaimer: I'm a n00b, so take this all with a grain of salt.
With respect to #2, the blog post I link to below suggests the following architecture:
Create a deployment that deploys the nginx controller pods
Create a service with a type LoadBalancer and a static IP that routes traffic to the controller pods
Create an ingress resource that gets used by the nginx controller pods
Create a secret that gets used by the nginx controller pods to terminate SSL
And other stuff too
From what I understand, the http vs https stuff happens with the nginx controller pods. All of my ingress rules are also http, but the nginx ingress controller forces SSL and takes care of all that, terminating SSL at the controller so that everything below it, all the ingress stuff, can be HTTP. I have all http rules, but all of my traffic through the LoadBalancer service is getting forced to use SSL.
Again, I'm a n00b. Take this all with a grain of salt. I'm speaking in layman's terms because I'm a layman trying to figure this all out.
I came across your question while looking for some answers to my own questions. I ran into a lot of the same issues that you ran into (I'm assuming past tense given the amount of time that has passed). I wanted to point you (and/or others with similar issues) to a blog post that I found helpful when learning about the nginx controller. So far (I'm still at an early stage and in the middle of using the post), everything in the post has worked.
You're probably already past this stuff now being that it's been a few months. But maybe this will help someone else even if it doesn't help you:
https://daemonza.github.io/2017/02/13/kubernetes-nginx-ingress-controller/
It helped me understand what resources needed to be created, how to deploy the controller pods, and how to expose the controller pods (create a LoadBalancer service for the controller pods with a static IP), and also force SSL. It helped me jump over several hurdles and get past the "how do all the moving parts fit together".
The Kubernetes technical documentation is helpful for how to use each piece, but doesn't necessarily lay it all out and slap pieces together like this blog post does. Disclaimer: the model in the blog post might not be the best way to do it though (I don't have enough experience to make that call), but it did help me at least get a working example of an nginx ingress controller that forced SSL.
Hope this helps someone eventually.
Andrew
Related
I have setup the default traefik dashboard example although this is not being exposed. Using kubectl port-forward works. The only thing that comes to mind is that I am using flannel as CNI and my previous k8s cluster instance I was using calico. It is weird though as I have other services exposed from the cluster outside of traefik that are working fine.
Not a proper answer but it seems that Flannel CNI was causing reachability issues. It was either a misconfiguration from my end or some fine tuning the tool was needed. I just replaced it with Calico and ingresses work fine!
I have gone through many blog posts and SO questions as well as k3s documentation and am still coming up short getting the real ip address of clients rather than the internal cluster ip address.
I have a standard k3s install using Traefik 1.8. As indicated in several github issues, I have set all my services to use Clusterip and I set externalTrafficPolicy: Local for my Traefik and apache services per this: https://github.com/k3s-io/k3s/issues/1652
The strange thing is, it seems that Traefik is passing along any headers like x-forwarded-for because if I manually add an x-forwarded-for with my ip address into my browser request, the result in the apache logs has my ip as well as the internal cluster ip separated by commas.
Is there something that gets hit before the Traefik instance when traffic comes in to the cluster that should be injecting the ip address?
It appears there are many things that can cause this problem. In my case, it was one of the more common issues. I simply had to patch the k3s traefik manifest to have hostNetwork: true.
kubectl patch deployment traefik --patch '{"spec":{"template":{"spec":{"hostNetwork":true}}}}'
It should be noted that it is not recommended to manually modify this manifest as it is managed by helm. So if the helm process runs again or k3s is reinstalled or updated, it will revert and you will have to run this patch again. You would have to modify the k3s helm chart for traefik or implement your own in place of the k3s one to get this change to stick.
There seems to be an extensive discussion on this topic in the k3s Discussions here: https://github.com/k3s-io/k3s/discussions/2997 .
However, for me, none of the provided answers works, but YMMV.
This question is more to give me some direction on how to go about the problem in general, not a specific solution to the problem.
I have a working kubernetes cluster that's using an nginx ingress as the gate to the outside world. Right now everything is on minikube, but the end goal is to move it eventually to GKE, EKS or AKS (for on premise clients that want our software).
For this I'm going to use helm charts to paremetrize the yaml files and ENV variables needed to setup the resources. I will keep using nginx as ingress to avoid maintining alb ingress or other cloud-specific ingress controllers.
My question is:
I'm not sure how to manage TLS certificates and then how to point the ingress to a public domain for people to use it.
I wanted some guidance on how to go about this in general. Is the TLS certificate something that the user can provide to the helm chart before configuring it? Where can I see a small exmaple of this. And finally, is the domain responbility of the helm chart? Or is this something that has to be setup on the DNS provide (Route53) for example. Is there an example you can suggest me to take a look at?
Thanks a lot for the help.
Installing certificates using Helm is perfectly fine just make sure you don't accidentally put certificates into public Git repo. Best practice is to have those certificates only on your local laptop and added to .gitignore. After that you may tell Helm to grab those certificates from their directory.
Regarding the DNS - you may use external-dns to make Kubernetes create DNS records for you. You will need first to integrate external-dns with your DNS provider and then it will watch ingress resources for domain names and automatically create them.
Our services use a K8s service with a reverse proxy to receive a request by multiple domains and redirect to our services, additionally, we manage SSL certificates powered by let's encrypt for every user that configures their domain in our service. Resuming I have multiple .conf files in the nginx for every domain that is configured. Works really great.
But now we need to increase our levels of security and availability and now we ready to configure the ingress in K8s to handle this problem for us because they are built for it.
Everything looks fine until we discover that every time that I need to configure a new domain as a host in the ingress I need to alter the config file and re-apply.
So that's the problem, I want to apply the same concept that I already have running, but in the nginx ingress controller. It's that possible? I have more than 10k domains up and running, I can't configure all in my ingress resource file.
Any thoughts?
In terms of scaling Kubernetes 10k domains should be fine to be configured in an Ingress resource. You might want to check how much storage you have in the etcd nodes to make sure you can store enough data there.
The default etcd storage is 2Gb, but if you keep increasing it's something to keep in mind.
You can also refer to the K8s best practices when it comes to building large clusters.
Another practice that you can use is to use apply and not create when changing the ingress resource, that way the changes are incremental. Furthermore, if you are using K8s 1.18 or later you can take advantage of Server Side Apply.
I have a set of applications that we're currently transitioning into a more "cloud-native" architecture. We're beginning by moving them into containers (Docker on Windows), and as part of this, we're hoping to use a load-balancing proxy to handle traffic to the containers.
Having looked at several options, we're hoping to use Traefik as a load-balancing proxy in this iteration of our architecture. It may or may not be important to note that all traffic through Traefik in this setup will be internal; it will not be serving any external traffic. I am also working in a self-hosted situation; because of contractual concerns, cloud providers such as AWS and Azure are not currently available to me.
My question, now, is how Traefik might best be deployed. I can see at least two options:
First, Traefik could be deployed on separate "load-balancer" hosts. I could use round-robin DNS for these hosts, and pass traffic through them.
Second, Traefik could be deployed on each of my application hosts. In this setup, Traefik would be more of a "side-car", and applications on each host would use the local Traefik instance as a proxy to the other hosts' services.
An issue I see is that in neither of these setups is true high availability achieved. In either case, a Traefik instance crashing would result in unavailability for at least some services. In the first case, round-robin DNS and short TTL might mitigate this?
Is lack of high-availability avoidable? Is there an alternative way to architect this solution? Does Traefik itself offer guidance on how this solution should be structured?