I have Ec2 instances sitting behind an application load balancer that is forwarding HTTPS traffic. Each instance is using the same TLS cert, which I also uploaded to ACM so the load balancer can use it. If I get new certificates, can I piece by piece replace the certs, or do I need to do something else to prevent the traffic from breaking? It isn't clear to me where the SSL handshake is happening if I use a load balancer like this.
The load balancers do not validate the TLS certs. See this blog post for more on this: https://kevin.burke.dev/kevin/amazons-albs-insecure-internal-traffic/
So, essentially, you can use any cert, even a self-signed one on the instances and the LBs will be fine with that. It does not have to match the TLS cert that the LB presents to the world.
Related
I am using an Ingress using Google-managed SSL certs mostly similar to what is described here:
https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs#setting_up_a_google-managed_certificate
However my backend service is a grpc service that is using HTTP2. According to the same documentation if I am using HTTP2 my backend needs to be "configured with SSL".
This sounds like I need a separate set of certificates for my backend service to configure it with SSL.
Is there a way to use the same Google managed certs here as well?
What are my other options here? I am using, Google managed certs for the Ingress not to manage any certs on my own, if I then use self signed certificates for my service, that kind of defeats the purpose.
i don't think it's required to create SSL for the backend services if you are terminating the HTTPS at LB level. You can attach your certs to at LB level and the backed-end will be HTTPS > HTTP.
You might need to create SSL/TLS new cert in case there is diff version ssl-protocols: TLSv1.2 TLSv1.3, Cipher set in your ingress controller configmap which you are using Nginx ingress controller, Kong etc.
If you are looking for End to End HTTPS traffic definitely you need to create a cert for the backend service.
You can also create/manage the Managed certificate or Custom cert with Cert manager the K8s secret and mount to deployment which will be used further by the service, in that case, no need to manage or create the certs. Ingress will passthrough the HTTPS request to service directly.
In this case, it will be an end-to-end HTTPS setup.
Update :
Note: To ensure the load balancer can make a correct HTTP2 request to
your backend, your backend must be configured with SSL. For more
information on what types of certificates are accepted, see Encryption
from the load balancer to the backends ." end to end tls seems to be a
requirement for HTTP2
This is my site https://findmeip.com it's running on HTTP2 and terminating the SSL/TLS at the Nginx level only.
Definitely, it's good to go with the suggested practice so you can use the ESP option from the Google, setting GKE ingress + ESP + grpc stack.
https://cloud.google.com/endpoints/docs/openapi/specify-proxy-startup-options?hl=tr
If not want to use ESP check above suggested :
You can Mount Managed certificate to
deployment which will be used further by the service, in that case, no
need to manage or create the certs. In other words, cert-manager will create/manage/re-new SSL/TLS on behalf of you in K8s secret which will used by service.
Google Managed Certificates can only be used for the frontend portion of the load balancer (aka client to LB). If you need encryption from the LB to the backends you will have use self-signed certificates or some other way to store said certificates on GKE as secrets and configuring the Ingress to connect to the backend using these secrets.
Like this https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-xlb#setting_up_https_tls_between_client_and_load_balancer
Is it possible to directly expose a server behind a L4 load balancer, with a public certificate?
This server is inside a Kubernetes pod. There is a TCP loadbalancer service in front of it which creates the external L4 LB.
My problem is that the TLS traffic does not reach the container inside the pod. So if you succeeded with a similar configuration, I would be interested into knowing.
Update
I did not mention that the traffic is GRPC.
Here is what I did: I have a domain and a corresponding official certificate. I want to secure the grpc connection.
I tried two approches:
with google ESP container, I put the cert as an nginx secret, pass it to the container, set an ssl-port. Behind the ESP in the same pod, I have my grpc server
In this case I get a message like this on the client side:
D0610 14:38:46.246248584 32401 security_handshaker.cc:176] Security
handshake failed:
{"created":"#1591792726.246234613","description":"Handshake
failed","file":"../deps/grpc/src/core/lib/security/transport/security_handshaker.cc","file_line":291,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
I see some TLS exchanges with wireshark but no log in esp.
without ESP, I put the cert in my GRPC server. There the GRPC server fails with something like this:
error:1408F10B:SSL routines:ssl3_get_record:wrong version number
In the google ESP documentation I see that I have to prove the domain belongs to me and upload the cert (but where)?
Update 2
As of today, I see no evidence that it is feasible.
IMO, the main issue is that the L4 has the IP corresponding to the domain name of the certificate. Hence the pods don't have the correct IP to prove that they can use the cert so their request towards roots are denied (I have no proof of that as I can't get debug info from nginx in the ESP. I have seen a request with the pure GRPC server solution though).
The issue was in TLS exchange.
By installing the cert in the ESP, it works fine with a web browser and the certificate is indicated valid, whereas with a GRPC client, the TLS handshake fails. Adding additional trace info helped.
By checking my certificate (not self signed but attached to my domain), I found that there is an intermediate certificate provided with it. I installed it along with the domain certificate (in the same crt file) and then it worked.
I don't know exactly why it is behaving like this but maybe it's due to the root_cert file in grpc client lib being too old.
By the way for a domain cert, there is no specific requirement regarding CN and subjectAltName for the certificate. It works without it. So it must only apply to self signed certs as I have seen elsewhere.
I had another issue that disturbed my task: be careful not to name the service port of the L4 load balancer with 'http2' inside. I had some side effect, making another deployment fail due to this. In fact when you do https, don't put http2 in the name.
Anyway it is now working and answers the request for the bounty. Thanks to all who tried to help :)
We have 10 different kubernetes pods which runs inside a private VPN, this pods are HTTP serving endpoints(not HTTPS). But this services would interact with HTTPS serving endpoints. Logically to make call to HTTP-S serving endpoints from a HTTP serving pod , the SSL server certificate trust is required. Hence we decided to store the SSL certificates inside each HTTP Service pods to make call to HTTPS serving pods.
I am wondering is there are any alternative approaches for managing SSL certificates across different pods in Kubernetes cluster? How about kubeadm for K8s certificate management ... any suggestions ?
This is more of a general SSL certificate question rather than specific to Kubernetes.
If the containers/pods providing the HTTPS endpoint already have their SSL correctly configured and the SSL certificate you are using was purchased/generated from a known, trusted CA (like letsencrypt or any one of the known, trusted certificate companies out there) then there is no reason your other container apps that are making connections to your HTTPS endpoint serving pods would need anything special stored in them.
The only exception to this is if you have your own private CA and you've generated certificates on that internally and are installing them in your HTTPS serving containers. (Or if you are generating self-signed certs). Your pods/containers connecting to the https endpoints would then need to know about the CA certificate. Here is a stackoverflow question/answer that deals with this scenario:
How do I add a CA root certificate inside a docker image?
Lastly, there are better patterns to manage SSL in containers and container schedulers like Kubernetes. It all depends on your design/architecture.
Some general ideas:
Terminate SSL at a load balancer before traffic hits your pods. The load balancer then handles the traffic from itself to the pods as HTTP, and your clients terminate SSL at the Load Balancer. (This doesn't really tackle your specific use case though)
Use something like Hashicorp Vault as an internal CA, and use automation around this product and Kubernetes to manage certificates automatically.
Use something like cert-manager by jetstack to manage SSL in your kubernetes environment automatically. It can connect to a multitude of 'providers' such as letsencrypt for free SSL. https://github.com/jetstack/cert-manager
Hope that helps.
I'm trying to have https for a website which is hosted in AWS EC2. I have followed the steps mention in the following link.
https://www.digitalocean.com/community/tutorials/how-to-create-a-ssl-certificate-on-apache-for-ubuntu-14-04
But still its showing the privacy thing to all user who are visiting the website. How can make the certificate as trusted or how long it will take Amazon to make it a trusted one.
Please help me to solve this. I'm stuck with this for last 2 days. Answers will be appreciated and Thank you.
You can use AWS Certificate Manager to issue free SSL certificate signed by AWS Certificate Authority. However for this to work, you need to use a Load Balancer and attach the certificate to the Load Balancer which will forward the traffic to the EC2 instance.
Depending on your requirements you may wish to use SSL termination on an Elastic Load Balancer (ELB) instead.
This involves creating a free AWS certificate and an ELB. Attach both your instance the certificate to the ELB with HTTPS forwarded to port 80 on your instance.
Then just point your DNS name to the ELB. If you're using Route53 then you can just use an A-record alias.
Edit: If you want to automatically direct HTTP to HTTPS you'll need to check the X-Forwarded-Proto header in Apache's .htaccess file. More information here.
The certificate which you are using is a "Self Signed Certificate (https://en.wikipedia.org/wiki/Self-signed_certificate)".
In order to get rid of insecure certificate or privacy issues on HTTPS, you need to get your CSR signed from a trusted CA like Comodo, Godaddy etc.
Ref -
https://in.godaddy.com/help/apache-generate-csr-certificate-signing-request-5269
https://help.comodo.com/topic-437-1-843-10843-.html
OR
In case you want free verified SSL certificates, "letsencrypt" is the way to go.
https://letsencrypt.org/
You don't need to pay anyone for a certificate. Just use LetsEncrypt and their CertBot ACME client. The CertBot automates the task of issuing and renewing certificates.
LetsEncrypt is the leading free SSL certificate authority (CA) and their certs are as good as any paid cert.
I would like to set up a TLS authentication on the two-server setup with a load balancer that spreads traffic between them. I do not have access to the load balancer, so I need to set up both servers separately. I need a certificate that would be signed by my own CA that I can pass on to the client which I want to authenticate. I know that it is possible to create a CA that is common for both servers, I am not sure how though. Is it as simple as creating a CA on each server with the same configuration?
I would appreciate some guidance.
TLS uses TCP connections, so the load balancer will only balance the establishment of TCP connections. After that the client is tied to any one of the two servers.
The certificate the servers present to their clients should match the URL or IP of the load balancer, so the clients can check if the certificate matches what they intended to connect to.
If you expose which backend server the client has reach (eg. because the load balancer redirects the connection, instead of tunneling it), your servers will probably need a wildcard certificate.