Considering a Nginx reverse-proxy handling TLS LetsEncrypt certificates "in front" of a backend service, what is the good deployment architecture of this setup on Kubernetes ?
My first thought was do make a container with both Nginx and my server in a container as a Stateful Set.
All those stateful sets have access to a volume mounted on /etc/nginx/certificates.
All those containers are running a cron and are allowed to renew those certificates.
However, I do not think it's the best approach. This type of architecture is made to be splited, not running completely independant services everwhere.
Maybe I should run an independent proxy service which handle certificates and does the redirection to the backend server deployment (ingress + job for certificate renewal) ?
If you are using a managed service (such as GCP HTTPS Load Balancer), how do you issue a publicly trusted certificate and renew your it?
You want kube-lego.
kube-lego automatically requests certificates for Kubernetes Ingress resources from Let's Encrypt
It works with GKE+LoadBalancer and with nginx-ingress as well. Usage is trivial; automatic certificate requests (including renewals); uses LetsEncrypt.
The README says -perhaps tongue in the cheek- that you need a non production use case. I have been using it for production and I have found it to be reliable enough.
(Full disclosure: I'm loosely associated with the authors but not paid to advertise the product)
Related
I have nodejs app using kafkajs package for connecting to AWS MSK.
We are moving to Strimzi Kafka because we already have a kubernetes cluster and we don't need the MSK anymore.
Until now we were connected with SSL but didn't have to specify any CA path or something. We used this way of connection both on our nodejs apps and kafka-ui and it worked with no issues.
We are trying to the same with Strimzi Kafka, but we get SSL handshake failed.
For my understanding is AWS MSK is using amazon certificates that are known while the Strimzi Kafka is generating self signed certificates which is ok by us.
How can I still using this way like we used with AWS MSK? With just use ssl: true in kafkajs (It works)
Thanks.
The easiest way to use a certificate signed by some public CA is using the listener certificate which lets you provide your own server certificate for given listener. I'm not sure how the Amazon CA works, but this blog post shows how to do it for example using Cert-Manager and Let's Encrypt.
Keep in mind that to use the public CAs, you usually need to use some proper domain names and not just internal Kubernetes services. This might for example increase costs or latency if your applications run in the same Kubernetes cluster because the traffic might need to go through a load balancer or ingress.
I have a kubernetes cluster in a corporate environment, where all HTTPS traffic is man-in-the-middled and the certificates are replaced with the company owns. Right now, all the applications running on the cluster get the Company's certificates injected by rebuilding the Docker image or by mounting them from a secret and adding them to the local store. This is painful and makes it harder to just use public helm charts and docker images without modifying them.
For example, I'm running jenkins on the cluster, which tries to install plugins from https://updates.jenkins-ci.org/. This would normally fail in my case with a SSL exception, unless I add the certficates to the Jenkins keystore.
I was wondering if there's a way to set this up at the cluster level,
So that there's some component that deals with this and the applications can then access the internet normally, without being aware of the whole certificate situation?
My thoughts were:
A cluster proxy pod, that all the applications then use.
Some ambassador container on each pod, that the apps connect to
I would imagine I'm not the only one in this situation but couldn't find a generic solution for this.
You could have a look at istio. It's a service mesh that uses sidecar proxies to (beside other things) take over responsibility for encrypting traffic between applications.
The proxies use the concept of mutual TLS (mTLS), where all connections inside the mesh are encrypted out-of-the-box. The applications them-self don't have to bother with certificates and can send messages in plain text.
Istio also provides a mechanism to migrate to mTLS, so you can include your applications into the mesh one by one, switch to mTLS and disable your own certification overhead.
You can set everything up with your own PKI so you're still using your companies certificates. Also you get a bunch of other features like enhanced observability, canary deployments, on the fly token based authentication/authorization and more.
I have a containerized Flash application (a simple webservice exposed in the internet) with SSL enabled by gunicorn through:
CMD ["gunicorn", "--certfile", "/var/tmp/fullchain.pem", "--keyfile", "/var/tmp/key.pem", "__init__:create_app()", "-b", ":8080"]
I have a bot that renews Let's Encrypt certificates in this path every 3 months.
Now I am creating a Kubernetes cluster to put this application an orchestrate the replicas.
In a related question I've seen some ingress controllers provide this certificate creation/renew functionality so I would not need to map to .pem files anymore. There is also cert-manager that does that.
Now I don't know if I need gunicorn or what is the easyest and recommended way to configuring that to run the application. I am also in the process of chosing an ingress controller for my cluster.
Now I don't know if I need gunicorn.
Gunicorn is like java Tomcat, and it can also improve performance for python web server, so using Gunicorn is also recommend without SSL.
If you have other service in same cluster want to talk to your Flask server, and you want to protect that connection, you should config Gunicorn with SSL. If not, I think using an ingress controller with certificate manager is convenient.
I am also in the process of chosing an ingress controller for my cluster.
Well, I think cert-manager offical doc can help you, it deploy cert-manager with Nginx ingress controller.
Theoretically you don't need to resign from your current setup: Flask app exposed on HTTPS. For instance the NGINX ingress controller can pass (encrypted) TLS packets directly to an upstream server (in your case Gunicorn) using SSL Passthrough feature.
But definitely it would be better to do it in a recommended Kubernetes way, with TLS enabled for Ingress (where cert-manager add-on can help you in obtaining certificates from sources like Let's Encrypt)
We are researching the best solution to allow customers to use their domain names with our hosting services. The hosting services are based on Google App Engine standard. The requirements are:
Customers can point their domain name to our server via CNAME or A record
Our server should be able to generate SSL certs for them automatically using Let's Encrypt
Our server should be able to handle custom SSL certs uploaded by customers
Should be robust and reliable when adding new customers (new confs, SSL certs etc.) into our servers
Should be scalable, and can handle a large number of custom domains and traffic (e.g. from 0 to 10000)
Minimum operation costs (the less time needed for maintaining the infrastructure, the better)
It seems Google Kubernetes Engine (formerly known as Google Container Engine) would be the direction to go. Is there a specific, proven way to set it up? Any suggestions/experiences sharing would be appreciated.
I would recommend going through this link to get started with setting up a GKE cluster.
For your purpose of SSL on GKE I would recommend creating an Ingress as specified in this link which automatically creates a Loadbalancer Resource in GCP if you use the default GLBC ingress controller. The resulting LB's configuration (Ports, Host Path rules, Certificates, Backend Services, etc. ) are defined by the configuration of the Ingress Object itself. You can point the domain the domain name to the IP of the Loadbalancer.
If you want to configure your Ingress(and consequently the resulting LB) to use certs created by 'Let's Encrypt', you would be modifying the configuration presented in the YAML of the ingress.
For actually integrating Let's Encrypt for Kubernetes, it is actually possible by using a service called cert-manager to automate the process of obtaining TLS/SSL certificates and store them inside secrets.
This link shows how to use cert-manager with GKE.
If you want to use self managed SSL certificates please see this link for more information. The GKE is scalable by the GKE's cluster autoscaler which automatically resizes clusters based on the demands of the workloads you want to run.
I am trying to lunch new servers automatically when needed but I am having some difficulty getting the certificate before making the server live. What I want to do is run a setup script which gets all the packages, websites and certificates ready and after that add the server to production. However, Letsencrypt wants me to verify that the server requesting the certificate is actually the website which replies to requests. How can I get the Letsencrypt certificate before adding the server to production? I don't want requests to the real website to be routed to the new server until it is fully setup and has the certificates.
One solution I thought of is to save the certificates on an AWS S3 bucket and synchronize them whenever a renewal is needed. Then when I setup a new server I just get the latest certificate from my AWS S3 bucket and I don't have to worry about getting the certificate from the CA until after the server is added to production.But this solution doesn't seem "clean" and would require me to have an S3 bucket just for my Letsencrypt certificate which also adds another weakness where a certificate could be stolen from.
Is there a more simple solution which I haven't thought of yet?
In a load-balanced (LB) scenario, you should consider having exactly one entity responsible for performing LE certificate acquisition. Things get complicated with multiple entities doing this asynchronously - you'd need to be able to guarantee that the ACME challenges get routed to the relevant server(s), and your LB doesn't have that information (without additional complexity).
So I'd suggest either:
Terminating HTTPS at your load-balancer. Then none of your servers need to care about HTTPS or certificates.
Having one "special" server that's responsible for interacting with LE, and then distributing the cert to the other servers. The details of how you do that is implementation-dependent, because it depends on how you're managing server/service configuration.