Google Cloud TCP external load balancer and TLS not self signed - ssl

Is it possible to directly expose a server behind a L4 load balancer, with a public certificate?
This server is inside a Kubernetes pod. There is a TCP loadbalancer service in front of it which creates the external L4 LB.
My problem is that the TLS traffic does not reach the container inside the pod. So if you succeeded with a similar configuration, I would be interested into knowing.
Update
I did not mention that the traffic is GRPC.
Here is what I did: I have a domain and a corresponding official certificate. I want to secure the grpc connection.
I tried two approches:
with google ESP container, I put the cert as an nginx secret, pass it to the container, set an ssl-port. Behind the ESP in the same pod, I have my grpc server
In this case I get a message like this on the client side:
D0610 14:38:46.246248584 32401 security_handshaker.cc:176] Security
handshake failed:
{"created":"#1591792726.246234613","description":"Handshake
failed","file":"../deps/grpc/src/core/lib/security/transport/security_handshaker.cc","file_line":291,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
I see some TLS exchanges with wireshark but no log in esp.
without ESP, I put the cert in my GRPC server. There the GRPC server fails with something like this:
error:1408F10B:SSL routines:ssl3_get_record:wrong version number
In the google ESP documentation I see that I have to prove the domain belongs to me and upload the cert (but where)?
Update 2
As of today, I see no evidence that it is feasible.
IMO, the main issue is that the L4 has the IP corresponding to the domain name of the certificate. Hence the pods don't have the correct IP to prove that they can use the cert so their request towards roots are denied (I have no proof of that as I can't get debug info from nginx in the ESP. I have seen a request with the pure GRPC server solution though).

The issue was in TLS exchange.
By installing the cert in the ESP, it works fine with a web browser and the certificate is indicated valid, whereas with a GRPC client, the TLS handshake fails. Adding additional trace info helped.
By checking my certificate (not self signed but attached to my domain), I found that there is an intermediate certificate provided with it. I installed it along with the domain certificate (in the same crt file) and then it worked.
I don't know exactly why it is behaving like this but maybe it's due to the root_cert file in grpc client lib being too old.
By the way for a domain cert, there is no specific requirement regarding CN and subjectAltName for the certificate. It works without it. So it must only apply to self signed certs as I have seen elsewhere.
I had another issue that disturbed my task: be careful not to name the service port of the L4 load balancer with 'http2' inside. I had some side effect, making another deployment fail due to this. In fact when you do https, don't put http2 in the name.
Anyway it is now working and answers the request for the bounty. Thanks to all who tried to help :)

Related

Using and then removing self-signed certificate localhost

Problem Background:
As part of the Computer Networking course assignment, I have been given task of implementing a Proxy Server ( using python socket and ssl module ) that handles https communications between the browser and the origin server (The real server that my browser wants to talk to).
What I have done so far:
I have implemented the above requirement using ssl sockets and also generated self-signed 'cert.pem' 'key.pem' files.
What I need to do:
Now I just need to tell my browser (chrome 89 on kubuntu 20.04) to accept this self-signed certificate and then test the working of my proxy server.
Reading from this stackoverflow question, I can see that I have to:
(1) become my own CA (2) then sign my SSL certificate as a CA. (3) Then import the CA certificate (not the SSL certificate, which goes onto my server) into Chrome.
My confusion/question:
So if I do this, when eventually I am done with this assignment, how do I reverse all these steps to get my browser in the previous state before I had made all these changes. Also, how to reverse the "become your own CA" and also delete the SSL certificates signed by my CA.
Basically, I want my system to return to the previous state it was before I would have made all these changes.
UPDATE:
I have done the previously outlined steps but now I get an error.
Here is a snippet of my code:
serv_socket = socket(AF_INET, SOCK_STREAM)
serv_socket.bind(('', serv_port))
serv_socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context = context.load_cert_chain('cert.pem', 'key.pem')
context.set_ciphers('EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH')
serv_socket.listen(10)
socket_to_browser, addr = serv_socket.accept()
conn_socket_to_browser = context.wrap_socket(socket_to_browser, server_side=True)
At the last line conn_socket_to_browser = context.wrap_socket(socket_to_browser, server_side=True) an exception is thrown: [SSL: HTTPS_PROXY_REQUEST] https proxy request (_ssl.c:1123)
What am I doing wrong ?
As glamorous as "becoming your own CA" sounds, with openssl it basically comes down to creating a self-signed certificate, and then creating a directory where some CA-specific configuration will be stored (I don't fully remember the specifics, but I think it was just some files related to CNs and serial numbers) so basically reversing the "become your own CA" step is something as mundane as deleting this directory along with the private key and self-signed certificate you were using for the CA. That's it, the CA is no more.
And for chrome returning to the previous state, you would just go the the CA list where you added the CA certificate, select it and delete it. Chrome will stop accepting certificates signed by your CA.
Regarding your new problem... In my opinion, you have developed some kind of reverse proxy (meaning that you expect normal HTTPS requests that you then redirect to the real server) but you have configured Chrome to use it as a forward proxy. In this case, Chrome does not send it a normal HTTPS request, it sends a special non-encrypted CONNECT command and only after receiving the non-encrypted response, it negotiates the TLS connection. That's why openssl says "https proxy request" because it has detected a "https proxy request" (a CONNECT command) instead of the normal TLS negotiation.
You can take a look at How can a Python proxy server (using SSL socket) pretend to be an HTTPS server and specify my own keys to get decrypted data?
It's python, but I think that you'll get the idea

TLS termination behind AWS ELB on HAProxy (tcp mode)

I am using AWS ELB as entry point (in Proxy mode) to load balance among 2 HAProxy-es behind it from where traffic goes further to MQTT broker.
Those 2 HAProxies are responsible for client TLS termination (2 way TLS).
Certificates kind of work. I've tested on local setup between 2 servers. I've been able to publish with 2-way TLS, properly terminate it, and publish message to mqtt. Problem arise when moving everything to AWS.
I am using self signed root CA, intermediate CA, server certificate and client certificates. Using Elliptic Curve...
Problem might be due to a servers CN. I think it has to be the same as hostname connecting to with tools like mosquitto_pub.
Error that I get is TLS error, whith debug -> ssl handshake failure. Somehow I am not able to produce more verbose errors. Using openssl with s_client and debug for max debug output. Which produce me ssl handshake failure.
I would really appreciate any hint/suggestion.
Thanks in advance.
Tomaz
I solved this by using subjectAltName feature. I edited openssl.cnf and add new section [alt_names] and reference it later on in configuration. Under alt_names I added 1 DNS key, and 2 IPs. Found with man x509v3_config.
Best,
Tomaz

SSL: where is the certificate hosted? when does the verification occurs?

I am quite confused here:
I use DNSMadeeasy to manage my DNS. I have two apps.
One is Heroku hosted, and has https on https://example.com - Heroku has many great tutorials to setup the certificate, it hasn't been a problem.
The other one is a wordpress, hosted in 1and1 (though it shouldn't matter here), and is reachable at http://subdomain.example.com and we want it to be available at https://subdomain.example.com
1and1 does sell SSL certificate, but their automated setup works only when one uses their services for DNS also, as they say. Their support says it should be DNSMadeEasy which should be hosting our SSL certificate. I have the feeling it is not true, because for https://example.com, DNSMadeEasy was never involved.
Questions:
When does certificate querying occurs? Before, After, or in parallel of DNS resolution?
Who is hosting a certificate? The DNS provider? The server (accessible like a sitemap.xml at the root for instance)? A third party?
To enlarge the case, in general if I have a personal server with a fix IP, how can I communicate through https with a valid certificate?
In my case, how can I get my way out of it to make https://subdomain.example.com work?
You are right for not believing the 1and1 suggestion.
To answer your questions:
When does certificate querying occurs? Before, After, or in parallel
of DNS resolution?
A client resolves domain name to an IP address first. So DNS resolution happens first.
Who is hosting a certificate?
The server (in simplistic terms) hosts the certificate.
When a client wants to connect to your site (via HTTPS) it will first establish a secure connection with that IP address on port 443 (this is why usually (without SNI) you can only have one SSL certificate per IP address). As part of this process (which is called handshake) a client can also specify a server name (so-called server name extension) - this is a domain name of your site. This is useful if you have an SSL certificate that is valid for multiple domains.
A good/detailed explanation how it works can be found here
http://www.moserware.com/2009/06/first-few-milliseconds-of-https.html
if I have a personal server with a fix IP, how can I communicate
through https with a valid certificate?
Your server will need to be able to respond on port 443 and have/host an SSL certificate for a domain that resolves to that IP address.
In my case, how can I get my way out of it to make
https://subdomain.example.com work?
You need to purchase a certificate for subdomain.example.com and install it on the wordpress server.
Usually in hosted solution like yours you have 2 options:
Buy the SSL certificate via the provider (1and1 in your case) - a simpler option, they will configure everything for you.
Buy the SSL certificate yourself. Here you will most likely need to login to your 1and1/Wordpress management interface and generate a CSR (essentially a certificate request). Then you purchase the SSL certificate using this CSR and then you can install it via the same management interface.
The process will look similar to this:
http://wpengine.com/support/add-ssl-site/

ssl_error_rx_record_too_long - client and server SSL certs for the same IP

I'm implementing mutual SSL between service A and service B. Service A uses both 1-way and 2-way SSL. 1-way for the communication between a user and website A, and 2-way SSL to forward requests from that user to the service B in a secure way.
1-way SSL in service A is specified in Tomcat server.xml. 2-way SSL is implemented using JSEE secure socket communication on the client side (service A), and Tomcat config (service B). Atm. when I try to access service A I get ssl_error_rx_record_too_long error.
According to this answer ssl_error_rx_record_too_long and Apache SSL one of the reasons may be the fact that I'm using more than one SSL certificate for the same IP. Is this really the case that you can't use the same IP for several certificates? Even if one certificate is a server certificate (for 1-way SSL) and another is a client certificate (for 2-way SSL)?
This may not be the cause of my problem, but I just want to make sure if it's actually possible to have several certificates for the same FQDN. Thanks for help!
ssl_error_rx_record_too_long generally has nothing to do with certificate configuration, but the fact that what's talking on that port isn't actually using SSL/TLS.
The answers (and even the update to the question) in the question you linked to also point to this problem (e.g. missing SSLEngine on). You probably forgot something like SSLEnabled="true" in your connector configuration.
As I was saying in an answer to your other question, being able to configure two server certificates on the same IP address isn't really a problem for your case.
it's actually possible to have several certificates for the same FQDN
It is possible to configure multiple certificates on the same IP address and port using the Server Name Indication TLS extension, but both servers and clients would need to support it. In particular, this is not supported by the JSSE in Java 7 on the server side (only on the client side), but there are workarounds if you're willing to put a reverse proxy in front of your Java server.
This wouldn't be possible to do this with the same FQDN, since it's what allows to pick the certificate. This being said, having multiple server certificates for the same FQDN on the same IP address is generally pointless. Supporting multiple certificates is precisely useful when you need to support distinct names.

403.7 IIS 7.5 SSL client certificate authentication issue

I am testing a web service with an external partner using 2 way SSL under IIS 7.5. I am requiring SSL, requiring a client cert, and using one to one mapping to authenticate to a domain account. I have configured everything and it works fine on our network (I am able to provide a client cert, get authenticated and invoke the service from browser and test harness).
From outside of our network (in most cases, see below), I am getting a 403.7 error. I have gone through the machine level certificate store and made sure the certificates and CAs are trusted.
Here's the weird thing. I obtained a Type I cert to test from home (and got 403.7 like our intended partner is). So I setup Fiddler to debug SSL and send my certificate, and this works for some reason. I setup a test harness to pass the exact same certificate, and got 403.7. I test in my browser (IE 9), don't get a prompt for a client cert, and get 403.7.
Any help appreciated.
Bill
Last time I checked, IIS was using re-negotiation (by default) to get the client certificate: there is a first handshake where the server doesn't request a client certificate, followed by another handshake (encrypted this time) where the server requests the certificate (via a TLS CertificateRequest message). This will prevent you from seeing anything from Wireshark, unless you configure it to use the server's private key and decipher the traffic (note that this only works with some cipher suites).
One way to see the client-certificate negotiation is to configure IIS to use initial client certificate negotiation, using netsh and clientcertnegotiation=true (which is about initial negotiation). At least the CertificateRequest and the certificate will be sent in clear during the handshake, so you should be able to see this with Wireshark.
If the client isn't sending a certificate to the server as a response to the CertificateRequest, you'll still see an empty Certificate message from the client.
If you don't export the private key with the certificate to use with Fiddler or whichever other client, there is no chance that it will be able to use the certificate. It may at best try to send the certificate, but the handshake will fail (since the CertificateVerify message needs to be signed by the client's private key).
I guess you may encounter a problem whereby:
not presenting a certificate is accepted by the server (it's effectively optional),
presenting an invalid certificate makes it fail and causes this 403.7 status code (many servers and SSL/TLS stacks would implement this as a fatal error, but TLS specification doesn't say that unsupported_certificate, certificate_revoked, certificate_expired, certificate_unknown should be fatal, so this is at the server's discretion).
Are you using the same physical machine to test both the in-network and external-network connections? If not, are you sure that the external-network client has the private key accessible?
I have not configured Fiddler client authentication before. Does it read the client certificate and key from the standard certificate stores? Does it read directly from a PKCS12?
One other thing that may be helpful is inspecting the TLS handshake in WireShark. Specifically, check out the Server's "Certificate Request" message, as the data here clues the client (IE9) which client certificates it should display in the prompt. Compare this for the internal and external connections.