nginx client sent no required SSL certificate while SSL handshaking - ssl

about to bash my head against the wall after trying to sort this for over a week so really hoping someone can shed some light on where I've gone wrong with nginx!
I have this nginx.conf file which works perfectly (the way I want anyway)
events {
worker_connections 4096;
}
stream {
upstream stream_backend {
server backendapp:80;
}
server {
listen 443 ssl;
listen 8080;
proxy_pass stream_backend;
ssl_certificate ssl.crt;
ssl_certificate_key ssl.key;
ssl_protocols SSLv3 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_session_cache shared:SSL:20m;
ssl_session_timeout 4h;
ssl_handshake_timeout 30s;
#ssl_client_certificate ca.crt;
#ssl_verify_client on;
error_log /var/log/nginx/error.log debug;
}
}
However, I've been asked to enable client certificate, and when I remove the comments from #ssl_client_certificate ca.crt; and #ssl_verify_client on; I get these error logged in nginx *13 client sent no required SSL certificate while SSL handshaking, client: 192.168.65.3, server: 0.0.0.0:443
This is despite me sending client certs as part of the request.
I've scoured the internet and have found various solutions but none have worked for me so far (including here at SO). Any help would be greatly appreciated.
I'm using curl like so
curl -v -k C:\Temp\opscert\user.key -cert C:\Temp\opscert\user.crt https://local.example.com
and
curl -vk --key C:\Temp\opscert\user.key -cert C:\Temp\opscert\user.crt https://local.example.com
also with powershell for good measure
Invoke-WebRequest https://local.example.com -CertificateThumbprint 3b23775c0abfa0e9cb43e87b206dd6992ffc7e07
Additionally, I would have expected browsers to prompt for a certificate when browsing to https://local.example.com/ but none do, I've tried on firefox, IE and Chrome.
The only time chrome did request a certificate once is when I changed ssl_verify_client on; to ssl_verify_client optional; however this is not what we want.
Here are the errors from nginx
2018/11/23 17:21:42 [info] 6#6: *21 client 192.168.65.3:58176 connected to 0.0.0.0:443
2018/11/23 17:21:42 [info] 6#6: *21 client sent no required SSL certificate while SSL handshaking, client: 192.168.65.3, server: 0.0.0.0:443
2018/11/23 17:21:43 [info] 6#6: *22 client 192.168.65.3:58178 connected to 0.0.0.0:443
2018/11/23 17:21:43 [info] 6#6: *22 client sent no required SSL certificate while SSL handshaking, client: 192.168.65.3, server: 0.0.0.0:443
2018/11/23 17:21:48 [info] 6#6: *23 client 192.168.65.3:58194 connected to 0.0.0.0:443
2018/11/23 17:21:48 [info] 6#6: *23 client sent no required SSL certificate while SSL handshaking, client: 192.168.65.3, server: 0.0.0.0:443
2018/11/23 17:22:18 [info] 6#6: *24 client 192.168.65.3:58256 connected to 0.0.0.0:443
2018/11/23 17:22:18 [info] 6#6: *24 client sent no required SSL certificate while SSL handshaking, client: 192.168.65.3, server: 0.0.0.0:443
2018/11/23 17:23:18 [info] 6#6: *25 client 192.168.65.3:58378 connected to 0.0.0.0:443
2018/11/23 17:23:18 [info] 6#6: *25 client sent no required SSL certificate while SSL handshaking, client: 192.168.65.3, server: 0.0.0.0:443
Response from curl
PS C:\Users\abx> curl -v -k C:\Temp\opscert\user.key -cert C:\Temp\opscert\user.crt https://client.example.com
* Rebuilt URL to: C:\Temp\opscert\user.key/
* Port number ended with '\'
* Closing connection -1
curl: (3) Port number ended with '\'
* Rebuilt URL to: C:\Temp\opscert\user.crt/
* Port number ended with '\'
* Closing connection -1
curl: (3) Port number ended with '\'
* Rebuilt URL to: https://client.example.com/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to client.example.com (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=GB; ST=Surrey; L=London; O=H. Example Company Ltd; CN=*.example.com
* start date: Jan 5 00:00:00 2017 GMT
* expire date: Jan 10 12:00:00 2020 GMT
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: client.example.com
> User-Agent: curl/7.59.0
> Accept: */*
>
* TLSv1.2 (IN), TLS alert, Client hello (1):
* Empty reply from server
* Connection #0 to host client.example.com left intact
curl: (52) Empty reply from server
PS C:\Users\abx> curl -vk --key C:\Temp\opscert\user.key -cert C:\Temp\opscert\user.crt https://local.example.com
* Rebuilt URL to: C:\Temp\opscert\user.crt/
* Port number ended with '\'
* Closing connection -1
curl: (3) Port number ended with '\'
* Rebuilt URL to: https://local.example.com/
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to client.example.com (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=GB; ST=Surrey; L=London; O=H. Example Company Ltd; CN=*.example.com
* start date: Jan 5 00:00:00 2017 GMT
* expire date: Jan 10 12:00:00 2020 GMT
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: client.example.com
> User-Agent: curl/7.59.0
> Accept: */*
>
* TLSv1.2 (IN), TLS alert, Client hello (1):
* Empty reply from server
* Connection #0 to host client.example.com left intact
curl: (52) Empty reply from server

for powershell import just the user.crt (not pfx)
for curl, same thing, use user.crt curl -v --key C:\Temp\opscert\user.key --cert C:\Temp\opscert\user.crt https://client.example.com
I'd still prefer if when accessing via a browser I was prompted to for cert working but for now this is perfect as the requests will come from an api.

Related

Am I using 1 Way or 2 way SSL

root#XXXXXX:/var/tmp# curl --tlsv1.2 --tls-max 1.2 -v
https://example.com:8443/health --cacert Internal_Root_CA.cer
Trying 10.50.65.56...
TCP_NODELAY set
Connected to example.com (10.50.65.56) port 8443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: Internal_Root_CA.cer CApath: /etc/ssl/certs
TLSv1.2 (OUT), TLS handshake, Client hello (1):
TLSv1.2 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Client hello (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
ALPN, server did not agree to a protocol
Server certificate:
subject: C=US; ST=ZZZ; L=CCC; O=Company; CN=example.com
start date: Sep 29 22:30:19 2022 GMT
expire date: Sep 27 22:30:49 2024 GMT
subjectAltName: host "example.com" matched cert's "example.com"
issuer: O=Company; CN= Issuing CA
SSL certificate verify ok.
GET /health HTTP/1.1
Host: example.com:8443
User-Agent: curl/7.58.0
Accept: /
< HTTP/1.1 200 < Content-Type: text/plain;charset=UTF-8 < Content-Length: 0 < Date: Wed, 12 Oct 2022 18:33:10 GMT <
Connection #0 to host mdm-dev.gcp.aexp.com left intact
Am I using 1 way or 2 way SSL? THe REST API is developed using Spring boot.
I have to pass in the Root CA for the Call to work.
This is 1-way SSL because a) you don't give a client certificate to use and b) the server does not even request one (no CertificateRequest message from server).

GET request ssl_choose_client_version:unsupported protocol

I have a problem dealing with an upgrade of an application doing GET request to a remote server.
First thing first : a functional example of a GET done by the old version, and as expected it works
curl -k -vvvvv https://mywebsite.com/mywonderfulwebsite/mypage.php
* Trying 192.168.0.70...
* TCP_NODELAY set
* Connected to mywebsite.com (192.168.0.70) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (IN), TLS handshake, Server finished (14):
* TLSv1.0 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.0 (OUT), TLS change cipher, Client hello (1):
* TLSv1.0 (OUT), TLS handshake, Finished (20):
* TLSv1.0 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=MYWEBSITE.COM
* start date: Mar 24 10:20:51 2020 GMT
* expire date: Mar 24 00:00:00 2021 GMT
* issuer: CN=MYWEBSITE.COM
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /mywonderfulwebsite/mypage.php HTTP/1.1
> Host: mywebsite.com
> User-Agent: curl/7.58.0
> Accept: */*
....... and here the content of the page.....
And now from the new version, it doesn't work
curl -vvvvv https://mywebsite.com/mywonderfulwebsite/mypage.php
* Trying 192.168.0.70:443...
* TCP_NODELAY set
* Connected to mywebsite.com (192.168.0.70) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS alert, protocol version (582):
* error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
* Closing connection 0
curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
So I think it was from the TLS version, no problem let's force it :
curl --tlsv1.0 -vvvvv https://mywebsite.com/mywonderfulwebsite/mypage.php
* Trying 192.168.0.70:443...
* TCP_NODELAY set
* Connected to mywebsite.com (192.168.0.70) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS alert, protocol version (582):
* error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
* Closing connection 0
curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
and it's a fail.
I've tried adding the certificates from the remote website, and I have the same answer.
I've looked at a request using openssl client :
# openssl s_client -connect mywebsite.com:443 -tls1
CONNECTED(00000003)
139820362433856:error:141E70BF:SSL routines:tls_construct_client_hello:no protocols available:../ssl/statem/statem_clnt.c:1112:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 7 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
And now I'm playing with versions and requests and I have no clue where I should check.
Do you know how I could troubleshoot my problem ?
Here is the solution : https://askubuntu.com/questions/1233186/ubuntu-20-04-how-to-set-lower-ssl-security-level
Late openssl package is configured to forbid the usage of TLS < 1.2 however, the first curl request shows a communication using TLS 1.0
So in debian Buster openssl package was too new
dpkg -l | grep openssl
ii openssl 1.1.1d-0+deb10u7
I didn't have to downgrade Openssl
Edit /etc/ssl/openssl.cnf
add in the beginning of the file
openssl_conf = default_conf
And this to the end of the file
[ default_conf ]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
MinProtocol = TLSv1
CipherString = DEFAULT:#SECLEVEL=1
Changing the configuration allow the usage of minimal version of TSL starting TSL 1.0 and more, so from now I can request my legacy partner.

AWS CLI: unable to get issuer certificate

I have deployed an AWS S3 compatible solution on which I configured signed certificates for the API endpoint.
Now, if I try to list some buckets, I get:
$ aws s3 ls s3://bucket --endpoint-url https://s3.endpoint --ca-bundle /etc/ssl/certs/ca-certificates.crt
SSL validation failed for https://s3.endpoint/bucket?list-type=2&prefix=&delimiter=%2F&encoding-type=url
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1125)
But if I curl the same S3 endpoint from the same machine, I get SSL certificate verify ok:
curl https://s3.endpoint -v
* Trying 192.168.0.1:443...
* TCP_NODELAY set
* Connected to s3.endpoint (192.168.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=my-C; ST=my-ST; L=my-L; O=my-O; OU=my-OU; CN=s3.endpoint; emailAddress=my-email
* start date: Jun 28 11:03:53 2021 GMT
* expire date: Jun 28 11:03:53 2023 GMT
* subjectAltName: host "s3.endpoint" matched cert's "s3.endpoint"
* issuer: C=issuer-C; O=issuer-O; OU=issuer-OU; CN=issuer-CN
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: s3.endpoint
> User-Agent: curl/7.68.0
> Accept: */*
>
...
Is there something wrong with my aws command ?
$ aws --version
aws-cli/2.2.14 Python/3.8.8 Linux/5.4.0-73-generic exe/x86_64.ubuntu.20 prompt/off

Kubernetes 1.14.2 HA Master NGINX load balancer log.go:172] http: TLS handshake error from 192.168.5.32:43148: remote error: tls: bad certificate

This is driving me crazy, I am NO Kubernetes expert but I am also not a novice.
I have tried unsuccessfully for three days to get past this issue but I can't and I am at the end of my rope.
I can query the cluster from my desktop after I copied the certificates from (kube-apiserver-1:/etc/kubernetes/pki/*) to my desktop.
$ kubectl -n kube-system get nodes
NAME STATUS ROLES AGE VERSION
kube-apiserver-1 Ready master 71m v1.14.2
The Kubernetes cluster appears healthy when I query the kube-system pods:
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-6c85q 1/1 Running 3 65m
coredns-fb8b8dccf-qwxlp 1/1 Running 3 65m
kube-apiserver-kube-apiserver-1 1/1 Running 2 72m
kube-controller-manager-kube-apiserver-1 1/1 Running 2 72m
kube-flannel-ds-amd64-phntk 1/1 Running 2 62m
kube-proxy-swxrz 1/1 Running 2 65m
kube-scheduler-kube-apiserver-1 1/1 Running 1 54m
but when I query the api kubelet:
$ kubectl -n kube-system logs kube-apiserver-kube-apiserver-1
...
I0526 04:33:51.523828 1 log.go:172] http: TLS handshake error from 192.168.5.32:43122: remote error: tls: bad certificate
I0526 04:33:51.537258 1 log.go:172] http: TLS handshake error from 192.168.5.32:43124: remote error: tls: bad certificate
I0526 04:33:51.540617 1 log.go:172] http: TLS handshake error from 192.168.5.32:43126: remote error: tls: bad certificate
I0526 04:33:52.333817 1 log.go:172] http: TLS handshake error from 192.168.5.32:43130: remote error: tls: bad certificate
I0526 04:33:52.334354 1 log.go:172] http: TLS handshake error from 192.168.5.32:43128: remote error: tls: bad certificate
I0526 04:33:52.335570 1 log.go:172] http: TLS handshake error from 192.168.5.32:43132: remote error: tls: bad certificate
I0526 04:33:52.336703 1 log.go:172] http: TLS handshake error from 192.168.5.32:43134: remote error: tls: bad certificate
I0526 04:33:52.338792 1 log.go:172] http: TLS handshake error from 192.168.5.32:43136: remote error: tls: bad certificate
I0526 04:33:52.391557 1 log.go:172] http: TLS handshake error from 192.168.5.32:43138: remote error: tls: bad certificate
I0526 04:33:52.396566 1 log.go:172] http: TLS handshake error from 192.168.5.32:43140: remote error: tls: bad certificate
I0526 04:33:52.519666 1 log.go:172] http: TLS handshake error from 192.168.5.32:43142: remote error: tls: bad certificate
I0526 04:33:52.524702 1 log.go:172] http: TLS handshake error from 192.168.5.32:43144: remote error: tls: bad certificate
I0526 04:33:52.537127 1 log.go:172] http: TLS handshake error from 192.168.5.32:43146: remote error: tls: bad certificate
I0526 04:33:52.550177 1 log.go:172] http: TLS handshake error from 192.168.5.32:43150: remote error: tls: bad certificate
I0526 04:33:52.550613 1 log.go:172] http: TLS handshake error from 192.168.5.32:43148: remote error: tls: bad certificate
On the NGINX load balancer (IP: 192.168.5.32) I have configured the TCP passthrough option as specified in the Kubernetes documentation:
upstream kubernetes-api-cluster {
server 192.168.5.19:6443;
server 192.168.5.29:6443;
}
server {
listen 6443;
ssl_certificate /etc/nginx/ssl/kube-apiserver.pem;
ssl_certificate_key /etc/nginx/ssl/private/kube-apiserver.key;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
proxy_pass kubernetes-api-cluster;
}
I can query the API server directly from the NGINX LB (IP: 192.168.5.32):
$ curl -v https://192.168.5.29:6443
* Rebuilt URL to: https://192.168.5.29:6443/
* Trying 192.168.5.29...
* TCP_NODELAY set
* Connected to 192.168.5.29 (192.168.5.29) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: May 26 03:39:36 2019 GMT
* expire date: May 25 03:39:36 2020 GMT
* subjectAltName: host "192.168.5.29" matched cert's IP address!
* issuer: CN=kubernetes
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55840f1d9900)
> GET / HTTP/2
> Host: 192.168.5.29:6443
> User-Agent: curl/7.58.0
> Accept: */*
I can also query the api using the DNS entry to the api as specified in the documents:
curl -v https://kube-apiserver.mydomain.com:6443
* Rebuilt URL to: https://kube-apiserver.mydomain.com:6443/
* Trying 10.50.1.50...
* TCP_NODELAY set
* Connected to kube-apiserver.mydomain.com (10.50.1.50) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: May 26 03:39:36 2019 GMT
* expire date: May 25 03:39:36 2020 GMT
* subjectAltName: host "kube-apiserver.mydomain.com" matched cert's "kube-apiserver.mydomain.com"
* issuer: CN=kubernetes
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564287cbd900)
> GET / HTTP/2
> Host: kube-apiserver.mydomain.com:6443
> User-Agent: curl/7.58.0
> Accept: */*
I can query the api server using curl as well on the API server:
curl -v https://kube-apiserver.mydomain.com:6443
* Rebuilt URL to: https://kube-apiserver.mydomain.com:6443/
* Trying 10.50.1.50...
* TCP_NODELAY set
* Connected to kube-apiserver.epc-instore.com (10.50.1.50) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: May 26 03:39:36 2019 GMT
* expire date: May 25 03:39:36 2020 GMT
* subjectAltName: host "kube-apiserver.mydomain.com" matched cert's "kube-apiserver.mydomain.com"
* issuer: CN=kubernetes
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5628b9dbc900)
> GET / HTTP/2
> Host: kube-apiserver.mydomain.com:6443
> User-Agent: curl/7.58.0
> Accept: */*
The manifest on the api server contains:
cat /etc/kubernetes/manifest/kube-apiserver.yaml
...
- command:
- kube-apiserver
- --advertise-address=192.168.5.29
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-servers=http://etcd-cluster.mydomain.com:2379
- --insecure-port=0
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
image: k8s.gcr.io/kube-apiserver:v1.14.2
imagePullPolicy: IfNotPresent
...
If you have any idea or hints on how to fix this I am all ears. I am so frustrated with this issue, it really has gotten to me at this point. I will continue to work on it but if anyone has a clue about this issue and can help it will be great.
Thank you.
The actual root cause of the original issue was (citing the author of this post #Daniel Maldonado):
This was my mistake, I had a firewall configuration error and all
tests indicated that it was the load balancer probing the
kube-apiserver when in fact it was not. The issue was completely local
to the api-server itself. If anyone gets to this point please verify
that ALL ports are available to the API server from itself i.e.
loopback.
Your current nginx config isn't setting up a client cert. ssl_certificate is the server cert, if you want it to present a client cert to kubernetes-api-cluster you'll have to configure nginx to forward the incoming client certificate. I've previously done this using proxy_set_header X-SSL-CERT $ssl_client_escaped_cert (documentation)
upstream kubernetes-api-cluster {
server 192.168.5.19:6443;
server 192.168.5.29:6443;
}
server {
listen 6443;
ssl_certificate /etc/nginx/ssl/kube-apiserver.pem;
ssl_certificate_key /etc/nginx/ssl/private/kube-apiserver.key;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
proxy_pass kubernetes-api-cluster;
#forward incoming client certificate
ssl_verify_client optional; #requests the client certificate and verifies it if the certificate is present
proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;
}
This is more of a troubleshooting idea to really target the source of the problem.
If you can do:
kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
from the api server and you get a response then the problem is NOT the load balancer. To further prove this you can copy the appropriate certificates and files to a remote workstation and do the same:
kubectl --kubeconfig [workstation location]/admin.conf get nodes
This second one obviously implies that you have direct access to the load balancer.
If this works too you have confirmation that the certificates are being passed through the TCP load balancer.
However, the error will persist as the load balancer has a check "availability" of a backend server. This check does NOT use a certificate which produces the exception.

IIS 10 and HTTP/2 - require client certificate

Currently I'm testing web-application on IIS 10 using HTTP 1.1 and HTTP/2.
My test application has one endpoint (/api/test) which returns just 'true'.
I have 3 certificates:
Root CA (self-signed)
Server certificate signed by Root CA
Client certificate signed by Root CA
Root CA and Server certificate installed on Windows Server 2016, and IIS website configured for listen https://example.net:8081/ using Server certificate. Also I configure website to require client certificate (it is important for my tests, I need server/client certificates validation).
I test my app via curl, and for http1.1 all works fine.
Command:
curl.exe --http1.1 --get --url https://example.net:8081/api/test --cacert E:\ca.pem --cert E:\client.pem --key E:\client.key --cert-type PEM --verbose
Output:
* TCP_NODELAY set
* Connected to example.net port 8081 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: E:\ca.pem
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate: XXX
* SSL certificate verify ok.
> GET /api/test HTTP/1.1
> Host: example.net:8081
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.2 (IN), TLS handshake, Hello request (0):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Content-Type: application/json; charset=utf-8
< Server: Kestrel
< Date: Tue, 18 Sep 2018 13:45:35 GMT
<
true* Connection #0 to host abrakadabra.cranecs.net left intact
But if I try to send request using http/2, it is failed after server certificate validation.
Command:
curl.exe --http2 --get --url https://example.net:8081/api/test --cacert E:\ca.pem --cert E:\client.pem --key E:\client.key --cert-type PEM --verbose
Output:
* TCP_NODELAY set
* Connected to example.net port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: E:\ca.pem
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate: XXX
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x1f635299100)
> GET /api/test HTTP/2
> Host: example.net:8081
> User-Agent: curl/7.61.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* HTTP/2 stream 0 was not closed cleanly: HTTP_1_1_REQUIRED (err 13)
* stopped the pause stream!
* Connection #0 to host example.net left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: HTTP_1_1_REQUIRED (err 13)
In IIS logs I see next records:
2018-09-18 13:46:00 172.32.0.193 GET /api/test - 8081 - 134.17.25.89 HTTP/1.1 curl/7.61.1 - 200 0 0 421
2018-09-18 13:55:01 172.32.0.193 GET /api/test - 8081 - 134.17.25.89 HTTP/2.0 curl/7.61.1 - 403 7 64 0
So, for http/2 it seems like client certificate absent (403.7 status code).
And finally, if I'll just change 'require client certificate' to 'ignore client certificate' on IIS site settings - http1.1 and http/2 work both.
How can I use client certificate with HTTP/2 on IIS?
After couple hours of research, I find out that the IIS 10 currently doesn't support HTTP/2 with client certificate verification.
https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/http2-on-iis#when-is-http2-not-supported
In a few cases, HTTP/2 can't be used in combination with other features. In these situations, Windows will fall back to HTTP/1.1 and continue the transaction. This may involve negotiating HTTP/1.1 during the handshake, or sending an error code to the client instructing it to retry over an HTTP/1.1 connection.
I was reconfigure my server to use nginx instead of IIS as a proxy for app, and all works fine.
I also stumbled upon this issue and after reading this information, I discovered that you need to enable "Negotiate Client Certificate" ath the binding level.
I set up a DotNet Framework 4.8 website in Server 2022 with IIS 10 and turned on require client certificates. IE worked with this fine. TLS1.3 enabled browsers did not (tried Edge and Chrome). A Wireshark capture showed the connection being reset during the handshake. After playing with editing the website bindings, I found in order for it to work, I could either enable TLS 1.3 or HTTP/2, but not both, when requiring client certificates and using a TLS1.3 enabled client.
Hopefully Microsoft will fix this for us someday