When I check the definition of "WebhookClientConfig" of API of Kubernetes I found comments like this:
// `caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate.
// If unspecified, system trust roots on the apiserver are used.
// +optional
CABundle []byte `json:"caBundle,omitempty" protobuf:"bytes,2,opt,name=caBundle"`
in WebhookClientConfig
I wonder to know, what's exactly the "system trust roots "?
and I'm afraid the internal signer for CSR API of Kubernetes is not one of them.
It is a good practice to use secure network connections. A Webhook-endpoint in Kubernetes is typically an endpoint in a private network. A custom private CABundle can be used to generate the TLS certificate to achieve a secure connection within the cluster. See e.g. contacting the webhook.
Webhooks can either be called via a URL or a service reference, and can optionally include a custom CA bundle to use to verify the TLS connection.
This CABundle is optional. See also service reference for how to connect.
If the webhook is running within the cluster, then you should use service instead of url. The service namespace and name are required. The port is optional and defaults to 443. The path is optional and defaults to "/".
Here is an example of a mutating webhook configured to call a service on port "1234" at the subpath "/my-path", and to verify the TLS connection against the ServerName my-service-name.my-service-namespace.svc using a custom CA bundle
Related
I was able to achieve ssl offloading with Haproxy. So great product and appreciate that capability!
With that said, I need to doing mutual TLS but am a little confused on how that will work with the ingress controller configmap.
Going through this reference i've created a client cert, intermediate cert and root cert.
To note, I am terminating the ssl cert (which is from letsencrpt) on the load balancer currently.
However, the client cert and org CA are different than the lesencrypt tls/ssl cert that I have assigned as the SSL now; does that matter?
So, the first question I would have is does the ssl-certificate have to be set to the CA that will sign the client and server certs or can I just use the new ones I created in the instruction.
Setting up the configmap.
This is the part i'm confused on.
You can setup server-ca and server-crt but I don't think that applys here because after the ssl offloading there is nothing meant to be checked. However, I do want mTLS via the ssl termination.
So there is an configuration client-ca
Sets the client certificate authority enabling HAProxy to check clients certificate (TLS authentication), thus enabling client mTLS.
NB, ssl-offloading should be enabled for TLS authentication to work.
The client in this case being the actual client I want which is the device/frontend. Not the loadbalancer acting as a client to the backend server.
When I look at how this is setup:
frontend mysite
bind 192.168.56.20:80
bind 192.168.56.20:443 ssl crt /etc/haproxy/certs/ssl.crt verify required ca-file /etc/haproxy/certs/intermediate-ca.crt ca-verify-file /etc/haproxy/certs/root-ca.crt
http-request redirect scheme https unless { ssl_fc }
default_backend apiservers
Is it possible to do the same with the controller configmap as what is listed here below? There's a lot more going on that what I am seeing as flags / configurations that are in this methodology of applying client mTLS. Is there a way to achieve this in kubernetes without configmap?
The ssl parameter enables SSL termination for this listener. The crt parameter identifies the location of the PEM-formatted SSL certificate. This certificate should contain both the public certificate and private key.
You can restrict who can access your application by giving trusted clients a certificate that they must present when connecting. HAProxy will check for this if you add a verify required parameter to the bind line, as shown:
the ssl argument enables HTTPS
the crt argument specifies the server SSL certificate, which you will typically obtain from a certificate provider like Let’s Encrypt
the verify required argument requires clients to send a client certificate
the ca-file argument specifies the intermediate certificate with which we will verify that the client’s certificate has been signed with our organization’s CA
the ca-verify-file argument (introduced in HAProxy 2.2) includes the root CA certificate, allowing HAProxy to send a shorter list of CAs to the client in the SERVER HELLO message that will be used for verification, but keeping upper level CAs, such as the root, out of that list. HAProxy requires the root CA to be set with this argument or else included in the intermediate-ca.crt file (compatibility with older versions of HAProxy).
Also, my reasoning for now wanting to use letsencrypt and rather a private CA is because I can't renew device certificates every 60 - 90 days. That would not be efficient. In this case, and please let me know otherwise, I think it better to use either a real key/cert provider or in development testing utilize the openssl certs like in the HAProxy instruction.
It's odd but you really have to think about what a "client" is with these abstractions because I would never use this for a normal web page login but rather the server to server communication and in that sense this server is a client to this server. Or in my case this device is a client to this loadbalancer.
Is it possible to directly expose a server behind a L4 load balancer, with a public certificate?
This server is inside a Kubernetes pod. There is a TCP loadbalancer service in front of it which creates the external L4 LB.
My problem is that the TLS traffic does not reach the container inside the pod. So if you succeeded with a similar configuration, I would be interested into knowing.
Update
I did not mention that the traffic is GRPC.
Here is what I did: I have a domain and a corresponding official certificate. I want to secure the grpc connection.
I tried two approches:
with google ESP container, I put the cert as an nginx secret, pass it to the container, set an ssl-port. Behind the ESP in the same pod, I have my grpc server
In this case I get a message like this on the client side:
D0610 14:38:46.246248584 32401 security_handshaker.cc:176] Security
handshake failed:
{"created":"#1591792726.246234613","description":"Handshake
failed","file":"../deps/grpc/src/core/lib/security/transport/security_handshaker.cc","file_line":291,"tsi_code":10,"tsi_error":"TSI_PROTOCOL_FAILURE"}
I see some TLS exchanges with wireshark but no log in esp.
without ESP, I put the cert in my GRPC server. There the GRPC server fails with something like this:
error:1408F10B:SSL routines:ssl3_get_record:wrong version number
In the google ESP documentation I see that I have to prove the domain belongs to me and upload the cert (but where)?
Update 2
As of today, I see no evidence that it is feasible.
IMO, the main issue is that the L4 has the IP corresponding to the domain name of the certificate. Hence the pods don't have the correct IP to prove that they can use the cert so their request towards roots are denied (I have no proof of that as I can't get debug info from nginx in the ESP. I have seen a request with the pure GRPC server solution though).
The issue was in TLS exchange.
By installing the cert in the ESP, it works fine with a web browser and the certificate is indicated valid, whereas with a GRPC client, the TLS handshake fails. Adding additional trace info helped.
By checking my certificate (not self signed but attached to my domain), I found that there is an intermediate certificate provided with it. I installed it along with the domain certificate (in the same crt file) and then it worked.
I don't know exactly why it is behaving like this but maybe it's due to the root_cert file in grpc client lib being too old.
By the way for a domain cert, there is no specific requirement regarding CN and subjectAltName for the certificate. It works without it. So it must only apply to self signed certs as I have seen elsewhere.
I had another issue that disturbed my task: be careful not to name the service port of the L4 load balancer with 'http2' inside. I had some side effect, making another deployment fail due to this. In fact when you do https, don't put http2 in the name.
Anyway it is now working and answers the request for the bounty. Thanks to all who tried to help :)
I'm currently trying to setup GlusterFS integration for a Kubernetes cluster. Volume provisioning is done with Heketi.
GlusterFS-cluster has a pool of 3 VMs
1st node has Heketi server and client configured. Heketi API is secured with a self-signed certificate OpenSSL and can be accessed.
e.g. curl https://heketinodeip:8080/hello -k
returns the expected response.
StorageClass definition sets the "resturl" to Heketi API https://heketinodeip:8080
When storageclass was created successfully and I try to create a PVC, this fails:
"x509: certificate signed by unknown authority"
This is expected, as ususally one has to allow this insecure HTTPS-connection or explicitly import the issuer CA (e.g. a file simply containing the pem-String)
But: How is this done for Kubernetes? How do I allow this insecure connection to Heketi from Kubernetes, allowing insecure self-signed cert HTTPS or where/how do I import a CA?
It is not an DNS/IP problem, this was resolved with correct subjectAltName settings.
(seems that everybody is using Heketi, and it seems to be still a standard usecase for GlusterFS integration, but always without SSL, if connected to Kubernetes)
Thank you!
To skip verification of server cert, caller just need specify InsecureSkipVerify: true. Refer this github issue for more information (https://github.com/heketi/heketi/issues/1467)
In this page, they have specified a way to use self signed certificate. Not explained thoroughly but still can be useful (https://github.com/gluster/gluster-kubernetes/blob/master/docs/design/tls-security.md#self-signed-keys).
I have two questions about operation security of fabric.
1.In the official documentation( https://hyperledger-fabric.readthedocs.io/en/latest/operations_service.html?highlight=security#operations-security ), it says:"When TLS is enabled, a valid client certificate must be provided in order to access all resources unless explicitly noted otherwise below." Does this mean that when I access the operation RESTFUL API(e.g.GET /logspec) by curl or POSTMAN, I need to provide client certificate, right? If so, the client certificate must be generated by the corresponding organization's Fabric CA, right?
2.The documentation also says:"When clientAuthRequired is also enabled, the TLS layer will require a valid client certificate regardless of the resource being accessed.". I can't figure out what is the difference between "TLS" and "clientAuthRequired" configuration. Could any expert tell me this?
Thanks very much!
Yes ... it means that when TLS is enabled you will require a client certificate (expect where noted refers to the fact that the /healthz endpoint does not require a client certificate). The client certificate must be issued by a CA configured using the operations.tls.clientRootCAs.files property for the peer and/or the Operations.TLS.RootCAs property for the orderer.
If clientAuthRequired is enabled, then all endpoints will require a client certificate (again for now this means that /healthz would require a client cert).
I am rather confused by the purpose of the trustore file in a tomcat server.
I.e use of the truststoreFile parameter when setting up a TLS enabled connector.
According to the tomcat docs:
truststoreFile
The trust store file to use to validate client certificates. The
default is the value of the javax.net.ssl.trustStore system property.
If neither this attribute nor the default system property is set, no
trust store will be configured.
To me this is telling me that for an inbound connection to my tomcat server, this is where I would store certificates that a client would present. (e.g. a users browser)
Would I really put in 3rd party CA certificates here or other 3rd party server certificates, if my tomcat server wants to make an outbound connection to a TLS enabled server (e.g. an LDAPS server)
As per my understanding the place to do this is the cacerts file.
Is my understanding of the truststore wrong?
Yes, you are right, it is quite confusing.
If you need to make an outbound connection a 3rd party, you need to add the certificates to cacerts.
There is a very popular "signing, validation, etc." service for government-owned companies here in Spain, #firma, that just changed its wildcard server certificate and we had to add it to cacerts.