SSLError while trying to create EC2 server with knife - ssl

I'm trying to create and provision a new EC2 instance with knife, but keep running into an SSL error:
$bundle exec knife ec2 server create
ERROR: Excon::Errors::SocketError: hostname "ec2.us-east-1b.amazonaws.com" does not match the server certificate (OpenSSL::SSL::SSLError)
I'm running this from a mac (10.7) using ruby 2.0.0p0:
$ruby -v
ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-darwin11.4.2]
I'm fairly certain I compiled ruby with openssl support correctly. Running require 'openssl' from irb returns true. I'm using OpenSSL 1.0.1e installed via homebrew.
I also tried running knife with ruby 1.9.3-p194. That has the same result, with a slightly less helpful error message: "ERROR: Excon::Errors::SocketError: hostname does not match the server certificate (OpenSSL::SSL::SSLError)". That difference is the result of this pull request, which improved the error message: https://github.com/ruby/ruby/pull/122.
The following output from curl might be relevant:
$curl -v https://ec2.us-east-1b.amazonaws.com
* About to connect() to ec2.us-east-1b.amazonaws.com port 443 (#0)
* Trying 67.215.65.132... connected
* Connected to ec2.us-east-1b.amazonaws.com (67.215.65.132) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
* subject: serialNumber=UoFmxu6ta5ecJiIs4su2w-q-u8rxJ/d3; OU=GT55236522; OU=See www.rapidssl.com/resources/cps (c)12; OU=Domain Control Validated - RapidSSL(R); CN=*.opendns.com
* start date: 2012-08-23 10:11:50 GMT
* expire date: 2014-09-25 12:42:00 GMT
* subjectAltName does not match ec2.us-east-1b.amazonaws.com
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
* SSL peer certificate or SSH remote key was not OK
curl: (51) SSL peer certificate or SSH remote key was not OK
Is there something else I need to configure in order to successfully create an EC2 instance with knife?

In my knife.rb configuration file, I had this line:
knife[:region] = 'us-east-1b'
That worked at some point in the past, but the correct current setting is:
knife[:region] = 'us-east-1'
Removing the 'b' resolves the SSL hostname error:
$curl -v https://ec2.us-east-1.amazonaws.com
* About to connect() to ec2.us-east-1.amazonaws.com port 443 (#0)
* Trying 205.251.242.7... connected
* Connected to ec2.us-east-1.amazonaws.com (205.251.242.7) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=ec2.us-east-1.amazonaws.com
* start date: 2010-10-08 00:00:00 GMT
* expire date: 2013-10-07 23:59:59 GMT
* subjectAltName: ec2.us-east-1.amazonaws.com matched
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)09; CN=VeriSign Class 3 Secure Server CA - G2
* SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: ec2.us-east-1.amazonaws.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Location: http://aws.amazon.com/ec2
< Content-Length: 0
< Date: Sat, 16 Mar 2013 21:15:51 GMT
< Server: AmazonEC2
<
* Connection #0 to host ec2.us-east-1.amazonaws.com left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):

When you install the chef-client on your mac book it will automatically installs the knife and dependent libraries, you dont have to do it manually. You dont have to run with bundle exec, just type knife ec2 server create, you will get the following output
** EC2 COMMANDS **
knife ec2 server list (options)
knife ec2 server delete SERVER [SERVER] (options)
knife ec2 server create (options)
knife ec2 instance data (options)
knife ec2 flavor list (options)
If you are getting this output then your knife is working properly. And also make sure your knife.rb is configured properly, if you have any problem let me know.

Related

GET request ssl_choose_client_version:unsupported protocol

I have a problem dealing with an upgrade of an application doing GET request to a remote server.
First thing first : a functional example of a GET done by the old version, and as expected it works
curl -k -vvvvv https://mywebsite.com/mywonderfulwebsite/mypage.php
* Trying 192.168.0.70...
* TCP_NODELAY set
* Connected to mywebsite.com (192.168.0.70) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (IN), TLS handshake, Server finished (14):
* TLSv1.0 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.0 (OUT), TLS change cipher, Client hello (1):
* TLSv1.0 (OUT), TLS handshake, Finished (20):
* TLSv1.0 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.0 / AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=MYWEBSITE.COM
* start date: Mar 24 10:20:51 2020 GMT
* expire date: Mar 24 00:00:00 2021 GMT
* issuer: CN=MYWEBSITE.COM
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /mywonderfulwebsite/mypage.php HTTP/1.1
> Host: mywebsite.com
> User-Agent: curl/7.58.0
> Accept: */*
....... and here the content of the page.....
And now from the new version, it doesn't work
curl -vvvvv https://mywebsite.com/mywonderfulwebsite/mypage.php
* Trying 192.168.0.70:443...
* TCP_NODELAY set
* Connected to mywebsite.com (192.168.0.70) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS alert, protocol version (582):
* error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
* Closing connection 0
curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
So I think it was from the TLS version, no problem let's force it :
curl --tlsv1.0 -vvvvv https://mywebsite.com/mywonderfulwebsite/mypage.php
* Trying 192.168.0.70:443...
* TCP_NODELAY set
* Connected to mywebsite.com (192.168.0.70) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS alert, protocol version (582):
* error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
* Closing connection 0
curl: (35) error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol
and it's a fail.
I've tried adding the certificates from the remote website, and I have the same answer.
I've looked at a request using openssl client :
# openssl s_client -connect mywebsite.com:443 -tls1
CONNECTED(00000003)
139820362433856:error:141E70BF:SSL routines:tls_construct_client_hello:no protocols available:../ssl/statem/statem_clnt.c:1112:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 7 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
And now I'm playing with versions and requests and I have no clue where I should check.
Do you know how I could troubleshoot my problem ?
Here is the solution : https://askubuntu.com/questions/1233186/ubuntu-20-04-how-to-set-lower-ssl-security-level
Late openssl package is configured to forbid the usage of TLS < 1.2 however, the first curl request shows a communication using TLS 1.0
So in debian Buster openssl package was too new
dpkg -l | grep openssl
ii openssl 1.1.1d-0+deb10u7
I didn't have to downgrade Openssl
Edit /etc/ssl/openssl.cnf
add in the beginning of the file
openssl_conf = default_conf
And this to the end of the file
[ default_conf ]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
MinProtocol = TLSv1
CipherString = DEFAULT:#SECLEVEL=1
Changing the configuration allow the usage of minimal version of TSL starting TSL 1.0 and more, so from now I can request my legacy partner.

Kubernetes 1.14.2 HA Master NGINX load balancer log.go:172] http: TLS handshake error from 192.168.5.32:43148: remote error: tls: bad certificate

This is driving me crazy, I am NO Kubernetes expert but I am also not a novice.
I have tried unsuccessfully for three days to get past this issue but I can't and I am at the end of my rope.
I can query the cluster from my desktop after I copied the certificates from (kube-apiserver-1:/etc/kubernetes/pki/*) to my desktop.
$ kubectl -n kube-system get nodes
NAME STATUS ROLES AGE VERSION
kube-apiserver-1 Ready master 71m v1.14.2
The Kubernetes cluster appears healthy when I query the kube-system pods:
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-6c85q 1/1 Running 3 65m
coredns-fb8b8dccf-qwxlp 1/1 Running 3 65m
kube-apiserver-kube-apiserver-1 1/1 Running 2 72m
kube-controller-manager-kube-apiserver-1 1/1 Running 2 72m
kube-flannel-ds-amd64-phntk 1/1 Running 2 62m
kube-proxy-swxrz 1/1 Running 2 65m
kube-scheduler-kube-apiserver-1 1/1 Running 1 54m
but when I query the api kubelet:
$ kubectl -n kube-system logs kube-apiserver-kube-apiserver-1
...
I0526 04:33:51.523828 1 log.go:172] http: TLS handshake error from 192.168.5.32:43122: remote error: tls: bad certificate
I0526 04:33:51.537258 1 log.go:172] http: TLS handshake error from 192.168.5.32:43124: remote error: tls: bad certificate
I0526 04:33:51.540617 1 log.go:172] http: TLS handshake error from 192.168.5.32:43126: remote error: tls: bad certificate
I0526 04:33:52.333817 1 log.go:172] http: TLS handshake error from 192.168.5.32:43130: remote error: tls: bad certificate
I0526 04:33:52.334354 1 log.go:172] http: TLS handshake error from 192.168.5.32:43128: remote error: tls: bad certificate
I0526 04:33:52.335570 1 log.go:172] http: TLS handshake error from 192.168.5.32:43132: remote error: tls: bad certificate
I0526 04:33:52.336703 1 log.go:172] http: TLS handshake error from 192.168.5.32:43134: remote error: tls: bad certificate
I0526 04:33:52.338792 1 log.go:172] http: TLS handshake error from 192.168.5.32:43136: remote error: tls: bad certificate
I0526 04:33:52.391557 1 log.go:172] http: TLS handshake error from 192.168.5.32:43138: remote error: tls: bad certificate
I0526 04:33:52.396566 1 log.go:172] http: TLS handshake error from 192.168.5.32:43140: remote error: tls: bad certificate
I0526 04:33:52.519666 1 log.go:172] http: TLS handshake error from 192.168.5.32:43142: remote error: tls: bad certificate
I0526 04:33:52.524702 1 log.go:172] http: TLS handshake error from 192.168.5.32:43144: remote error: tls: bad certificate
I0526 04:33:52.537127 1 log.go:172] http: TLS handshake error from 192.168.5.32:43146: remote error: tls: bad certificate
I0526 04:33:52.550177 1 log.go:172] http: TLS handshake error from 192.168.5.32:43150: remote error: tls: bad certificate
I0526 04:33:52.550613 1 log.go:172] http: TLS handshake error from 192.168.5.32:43148: remote error: tls: bad certificate
On the NGINX load balancer (IP: 192.168.5.32) I have configured the TCP passthrough option as specified in the Kubernetes documentation:
upstream kubernetes-api-cluster {
server 192.168.5.19:6443;
server 192.168.5.29:6443;
}
server {
listen 6443;
ssl_certificate /etc/nginx/ssl/kube-apiserver.pem;
ssl_certificate_key /etc/nginx/ssl/private/kube-apiserver.key;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
proxy_pass kubernetes-api-cluster;
}
I can query the API server directly from the NGINX LB (IP: 192.168.5.32):
$ curl -v https://192.168.5.29:6443
* Rebuilt URL to: https://192.168.5.29:6443/
* Trying 192.168.5.29...
* TCP_NODELAY set
* Connected to 192.168.5.29 (192.168.5.29) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: May 26 03:39:36 2019 GMT
* expire date: May 25 03:39:36 2020 GMT
* subjectAltName: host "192.168.5.29" matched cert's IP address!
* issuer: CN=kubernetes
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55840f1d9900)
> GET / HTTP/2
> Host: 192.168.5.29:6443
> User-Agent: curl/7.58.0
> Accept: */*
I can also query the api using the DNS entry to the api as specified in the documents:
curl -v https://kube-apiserver.mydomain.com:6443
* Rebuilt URL to: https://kube-apiserver.mydomain.com:6443/
* Trying 10.50.1.50...
* TCP_NODELAY set
* Connected to kube-apiserver.mydomain.com (10.50.1.50) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: May 26 03:39:36 2019 GMT
* expire date: May 25 03:39:36 2020 GMT
* subjectAltName: host "kube-apiserver.mydomain.com" matched cert's "kube-apiserver.mydomain.com"
* issuer: CN=kubernetes
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564287cbd900)
> GET / HTTP/2
> Host: kube-apiserver.mydomain.com:6443
> User-Agent: curl/7.58.0
> Accept: */*
I can query the api server using curl as well on the API server:
curl -v https://kube-apiserver.mydomain.com:6443
* Rebuilt URL to: https://kube-apiserver.mydomain.com:6443/
* Trying 10.50.1.50...
* TCP_NODELAY set
* Connected to kube-apiserver.epc-instore.com (10.50.1.50) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kube-apiserver
* start date: May 26 03:39:36 2019 GMT
* expire date: May 25 03:39:36 2020 GMT
* subjectAltName: host "kube-apiserver.mydomain.com" matched cert's "kube-apiserver.mydomain.com"
* issuer: CN=kubernetes
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5628b9dbc900)
> GET / HTTP/2
> Host: kube-apiserver.mydomain.com:6443
> User-Agent: curl/7.58.0
> Accept: */*
The manifest on the api server contains:
cat /etc/kubernetes/manifest/kube-apiserver.yaml
...
- command:
- kube-apiserver
- --advertise-address=192.168.5.29
- --allow-privileged=true
- --authorization-mode=Node,RBAC
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-admission-plugins=NodeRestriction
- --enable-bootstrap-token-auth=true
- --etcd-servers=http://etcd-cluster.mydomain.com:2379
- --insecure-port=0
- --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
- --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
- --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
- --requestheader-allowed-names=front-proxy-client
- --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --secure-port=6443
- --service-account-key-file=/etc/kubernetes/pki/sa.pub
- --service-cluster-ip-range=10.96.0.0/12
- --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
image: k8s.gcr.io/kube-apiserver:v1.14.2
imagePullPolicy: IfNotPresent
...
If you have any idea or hints on how to fix this I am all ears. I am so frustrated with this issue, it really has gotten to me at this point. I will continue to work on it but if anyone has a clue about this issue and can help it will be great.
Thank you.
The actual root cause of the original issue was (citing the author of this post #Daniel Maldonado):
This was my mistake, I had a firewall configuration error and all
tests indicated that it was the load balancer probing the
kube-apiserver when in fact it was not. The issue was completely local
to the api-server itself. If anyone gets to this point please verify
that ALL ports are available to the API server from itself i.e.
loopback.
Your current nginx config isn't setting up a client cert. ssl_certificate is the server cert, if you want it to present a client cert to kubernetes-api-cluster you'll have to configure nginx to forward the incoming client certificate. I've previously done this using proxy_set_header X-SSL-CERT $ssl_client_escaped_cert (documentation)
upstream kubernetes-api-cluster {
server 192.168.5.19:6443;
server 192.168.5.29:6443;
}
server {
listen 6443;
ssl_certificate /etc/nginx/ssl/kube-apiserver.pem;
ssl_certificate_key /etc/nginx/ssl/private/kube-apiserver.key;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS;
proxy_pass kubernetes-api-cluster;
#forward incoming client certificate
ssl_verify_client optional; #requests the client certificate and verifies it if the certificate is present
proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;
}
This is more of a troubleshooting idea to really target the source of the problem.
If you can do:
kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes
from the api server and you get a response then the problem is NOT the load balancer. To further prove this you can copy the appropriate certificates and files to a remote workstation and do the same:
kubectl --kubeconfig [workstation location]/admin.conf get nodes
This second one obviously implies that you have direct access to the load balancer.
If this works too you have confirmation that the certificates are being passed through the TCP load balancer.
However, the error will persist as the load balancer has a check "availability" of a backend server. This check does NOT use a certificate which produces the exception.

IIS 10 and HTTP/2 - require client certificate

Currently I'm testing web-application on IIS 10 using HTTP 1.1 and HTTP/2.
My test application has one endpoint (/api/test) which returns just 'true'.
I have 3 certificates:
Root CA (self-signed)
Server certificate signed by Root CA
Client certificate signed by Root CA
Root CA and Server certificate installed on Windows Server 2016, and IIS website configured for listen https://example.net:8081/ using Server certificate. Also I configure website to require client certificate (it is important for my tests, I need server/client certificates validation).
I test my app via curl, and for http1.1 all works fine.
Command:
curl.exe --http1.1 --get --url https://example.net:8081/api/test --cacert E:\ca.pem --cert E:\client.pem --key E:\client.key --cert-type PEM --verbose
Output:
* TCP_NODELAY set
* Connected to example.net port 8081 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: E:\ca.pem
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate: XXX
* SSL certificate verify ok.
> GET /api/test HTTP/1.1
> Host: example.net:8081
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.2 (IN), TLS handshake, Hello request (0):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Content-Type: application/json; charset=utf-8
< Server: Kestrel
< Date: Tue, 18 Sep 2018 13:45:35 GMT
<
true* Connection #0 to host abrakadabra.cranecs.net left intact
But if I try to send request using http/2, it is failed after server certificate validation.
Command:
curl.exe --http2 --get --url https://example.net:8081/api/test --cacert E:\ca.pem --cert E:\client.pem --key E:\client.key --cert-type PEM --verbose
Output:
* TCP_NODELAY set
* Connected to example.net port 8081 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: E:\ca.pem
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate: XXX
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x1f635299100)
> GET /api/test HTTP/2
> Host: example.net:8081
> User-Agent: curl/7.61.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* HTTP/2 stream 0 was not closed cleanly: HTTP_1_1_REQUIRED (err 13)
* stopped the pause stream!
* Connection #0 to host example.net left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: HTTP_1_1_REQUIRED (err 13)
In IIS logs I see next records:
2018-09-18 13:46:00 172.32.0.193 GET /api/test - 8081 - 134.17.25.89 HTTP/1.1 curl/7.61.1 - 200 0 0 421
2018-09-18 13:55:01 172.32.0.193 GET /api/test - 8081 - 134.17.25.89 HTTP/2.0 curl/7.61.1 - 403 7 64 0
So, for http/2 it seems like client certificate absent (403.7 status code).
And finally, if I'll just change 'require client certificate' to 'ignore client certificate' on IIS site settings - http1.1 and http/2 work both.
How can I use client certificate with HTTP/2 on IIS?
After couple hours of research, I find out that the IIS 10 currently doesn't support HTTP/2 with client certificate verification.
https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/http2-on-iis#when-is-http2-not-supported
In a few cases, HTTP/2 can't be used in combination with other features. In these situations, Windows will fall back to HTTP/1.1 and continue the transaction. This may involve negotiating HTTP/1.1 during the handshake, or sending an error code to the client instructing it to retry over an HTTP/1.1 connection.
I was reconfigure my server to use nginx instead of IIS as a proxy for app, and all works fine.
I also stumbled upon this issue and after reading this information, I discovered that you need to enable "Negotiate Client Certificate" ath the binding level.
I set up a DotNet Framework 4.8 website in Server 2022 with IIS 10 and turned on require client certificates. IE worked with this fine. TLS1.3 enabled browsers did not (tried Edge and Chrome). A Wireshark capture showed the connection being reset during the handshake. After playing with editing the website bindings, I found in order for it to work, I could either enable TLS 1.3 or HTTP/2, but not both, when requiring client certificates and using a TLS1.3 enabled client.
Hopefully Microsoft will fix this for us someday

Docker daemon running under SSL contacted using cURL

I have setup docker in non-ssl mode by editing /etc/default/docker and setting DOCKER_OPTS="-H=0.0.0.0:2375".
Now when I start docker and connect to it using curl curl -sv http://10.24.16.17:2375/v1.22/containers/json, I get some response.
When I secure the docker using certificates (ca.pem, server-key.pem, server-cert.pem) and editing /etc/default/docker
DOCKER_OPTS="--tlsverify --tlscacert=/home/ubuntu/certs/ca.pem --tlscert=/home/ubuntu/certs/server-cert.pem --tlskey=/home/ubuntu/certs/server-key.pem -H=0.0.0.0:2376"
I am not able to connect service using curl:
curl -sv https://10.24.16.17:2376/v1.22/containers/json
I get the following output:
* Hostname was NOT found in DNS cache
* Trying 10.24.16.17...
* Connected to 10.24.16.17 (10.24.16.17) port 2376 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS alert, Server hello (2):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
I am able to run docker ps -a command using CLI though, only cURL to hit the API endpoint does not work.
When I try curl -sv --cacert /home/ubuntu/certs/ca.pem https://10.24.16.17:2376/v1.22/containers/json, I get:
* Hostname was NOT found in DNS cache
* Trying 10.24.16.17...
* Connected to 10.24.16.17 (10.24.16.17) port 2376 (#0)
* successfully set certificate verify locations:
* CAfile: /home/ubuntu/certs/ca.pem
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS alert, Server hello (2):
* error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate
* Closing connection 0
I read somewhere that the above error may be due to corrupt ca.pem file, but that may not be the case as the command docker ps -a was able to connect to docker host using the same ca.pem file.

Curl error - Empty reply from server (ssl)

I have problems accessing certain URL using curl on OSX. For other urls I receive response, and for this specific UWL I can receive response from other, ubuntu machine.
I am stucked and do not know what component is broken and do not have any idea, other than upgrading whole system and rould really appreciate any advice that would prevent me do so.
Output of curl:
$ curl -v https://subdomain.example.com/pushdispatcher/dispatcher.asp
* About to connect() to subdomain.example.com port 443 (#0)
* Trying 213.229.251.147... connected
* Connected to subdomain.example.com (213.229.251.147) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /Users/bmihelac/.cert/cacert.pem
CApath: none
* SSLv2, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
* subject: /C=SI/ST=Ljubljana/L=Ljubljana/O=Example d.d./OU=Example/CN=subdomain.example.com
* start date: 2011-07-01 00:00:00 GMT
* expire date: 2012-07-26 23:59:59 GMT
* common name: subdomain.example.com (matched)
* issuer: /C=US/O=Thawte, Inc./CN=Thawte SSL CA
* SSL certificate verify ok.
> GET /pushdispatcher/dispatcher.asp HTTP/1.1
> User-Agent: curl/7.16.4 (i386-apple-darwin9.0) libcurl/7.16.4 OpenSSL/0.9.7l zlib/1.2.3
> Host: subdomain.example.com
> Accept: */*
>
* SSLv3, TLS handshake, Hello request (0):
* SSL read: error:00000000:lib(0):func(0):reason(0), errno 54
* Empty reply from server
* Connection #0 to host subdomain.example.com left intact
curl: (52) SSL read: error:00000000:lib(0):func(0):reason(0), errno 54
* Closing connection #0
It seems that the server is dropping your connection for some reason.
Try to add User-Agent headers to the request ( bot detection? )
Try to connect from other ip ( your ip is in black list? )