Caddy reverse proxy server alerted internal error on attempt to relay TLS client hello - ssl

I tried to let the Caddy reverse proxy server relay the TLS handshake between the client and the back server, but failed at very beginning when the proxy just received the TLS client hello. Caddy v2.6.2 and Curl v7.87.0 were under test below.
/etc/hosts:
127.0.0.1 localhost back-server.local proxy-server.local
Caddyfile-back-server:
{ http_port 2015 }
back-server.local:2016
respond "Hello, world!"
Caddyfile-proxy-server:
{ http_port 2017 }
proxy-server.local:2018 {
reverse_proxy https://back-server.local:2016 {
header_up Host {upstream_hostport}
}
}
Initial tests were OK below.
$ caddy start --config Caddyfile-back-server
$ caddy start --config Caddyfile-proxy-server
$ curl https://back-server.local:2016
Hello, world!
$ curl https://proxy-server.local:2018
Hello, world!
The next test failed to relay the TLS client hello below.
$ curl --connect-to back-server.local:2016:proxy-server.local:2018 https://back-server.local:2016
curl: (35) ... internal error
According to the captured packets, sni: back-server.local was sent from the TLS client hello to proxy-server.local:2018, but the proxy returned TLS server alert: internal error, then closed the connection right away. No communication happened at all between proxy-server.local and back-server.local:2016.
What could be wrong and how to make it work?

Caddy server as reverse proxy doesn't use SNI from TLS client hello for the related TLS passthrough to the back server, it's the job of TLS forward proxy. There's Caddy L4 plugin existing for L4 routing including TLS SNI-based routing.
Below was successful against this problem along with another reverse proxy FRP capable of acting as TLS forward proxy as well.
/etc/hosts
127.0.0.1 localhost back-server.local proxy-server.local
Caddyfile-back-server
{ http_port 2015 }
back-server.local:2016
respond "Hello, world!"
frps.ini
[common]
vhost_https_port = 2018
frpc.ini
[common]
[TLS passthrough]
type = https
local_port = 2016
custom_domains = w4.local
Then the test went through as expected. All the configuration files were put at PWD for simplicity.
$ caddy start --config Caddyfile-back-server
$ frps
$ frpc
$ curl --connect-to back-server.local:2016:proxy-server.local:2018 https://back-server.local:2016
Hello, world!

Related

Haproxy TLS terminating and passthrough based on sni

I have similar path for the requests:
client mydomain.com -> nlb:443 -> haproxy -> cloudfront
client a.mydomain.com -> nlb:443 -> haproxy -> target_group_a
Main idea is do tls passthrough for the main domain name and send it to cloudfront without TLS termination. Requests into a.mydomain.com should pass to target_group_a and it should terminate tls. So my config for this is:
frontend main
bind *:443
mode tcp
option tcplog
log global
tcp-request inspect-delay 5s
acl is_main req_ssl_sni -i "${pDomainName}"
acl is_a req_ssl_sni -m beg "a"
tcp-request content accept if { req_ssl_hello_type 1 }
use_backend main if is_main
use_backend a if is_a
backend main
mode tcp
option ssl-hello-chk
server cloudfront "${pCloudFrontUrl}:443" check resolvers aws
backend a
mode tcp
server local 127.0.0.1:9666 send-proxy
frontend a
bind *:9666 ssl crt server.pem ca-file ca.pem verify required accept-proxy
mode http
default_backend proxy_a
backend proxy_a
mode http
server elb "${pServer}:80" check resolvers aws
Main record pass successfull and I get CloudFront SSL termination and everything is okay, but not for a.mydomain.com.
Also I tried to watch what SNI Haproxy is capture but I got only capture0: - in logs. I did like (right after tcp inspect line)
tcp-request content capture req_ssl_sni len 15
log-format "capture0: %[capture.req.hdr(0)]"
and it's strange because routing works.
I've tried a lot of possibilities.. For now I get SSL peer handshake failed, the server most likely requires a client certificate to connect error, but if I do listen frontend a on another port and in http mode everyting works fine.
Maybe I miss something basic or not, but I'm stuck on it for ages and maybe someone could help me.
For someone who is suffering or will suffer with that situation, just be sure that you are testing with gnu version of curl (or build it with properly libraries) because it doesn't work for me with BSD curl. My curl version and libs
curl 7.66.0 (x86_64-apple-darwin17.7.0) libcurl/7.66.0 SecureTransport zlib/1.2.11
Release-Date: 2019-09-11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets

SSL handshake failed using wget, no error message from log

I am trying to wget a file from a https server using the command wget -d --no-check-certificate https://abcde.com/test.tar.gz
This is the log i obtained
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.17.1 on linux-gnu.
Reading HSTS entries from /root/.wget-hsts
URI encoding = ‘UTF-8’
URI encoding = ‘UTF-8’
--2018-08-20 19:02:44-- https://abcde.com/test.tar.gz
Resolving proxy.png.mycompany.com (proxy.png.mycompany.com)... 172.xx.x.17x
Caching proxy.png.mycompany.com => 172.xx.x.17x
Connecting to proxy.png.mycompany.com (proxy.png.mycompany.com)|172.xx.x.17x|:911... connected.
Created socket 3.
Releasing 0x0000560ce6ef84f0 (new refcount 1).
---request begin---
CONNECT abcde.com:443 HTTP/1.1
User-Agent: Wget/1.17.1 (linux-gnu)
Host: abcde.com:443
---request end---
proxy responded with: [HTTP/1.1 200 Connection established
]
Initiating SSL handshake.
SSL handshake failed.
Closed fd 3
Unable to establish SSL connection.
Saving HSTS entries to /root/.wget-hsts

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

When I try to connect to any server (e.g. google.com) using curl (or libcurl) I get the error message:
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Verbose output:
$ curl www.google.com --verbose
* Rebuilt URL to: www.google.com/
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,localaddress,.localdomain.com'
* Uses proxy env variable http_proxy == 'https://proxy.in.tum.de:8080'
* Trying 131.159.0.2...
* TCP_NODELAY set
* Connected to proxy.in.tum.de (131.159.0.2) port 8080 (#0)
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number'
For some reason curl seems to use TLSv1.3 even if I force it to use TLSv1.2 with the command --tlsv1.2 (it will still print TLSv1.3 (OUT), ..."
I am using the newest version of both Curl and OpenSSL :
$ curl -V
curl 7.61.0-DEV (x86_64-pc-linux-gnu) libcurl/7.61.0-DEV OpenSSL/1.1.1 zlib/1.2.8
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy
I think this is a problem related to my installation of the programms.
Can somebody explain to me what this error message means?
* Uses proxy env variable http_proxy == 'https://proxy.in.tum.de:8080'
^^^^^
The https:// is wrong, it should be http://. The proxy itself should be accessed by HTTP and not HTTPS even though the target URL is HTTPS. The proxy will nevertheless properly handle HTTPS connection and keep the end-to-end encryption. See HTTP CONNECT method for details how this is done.
If anyone is getting this error using Nginx, try adding the following to your server config:
server {
listen 443 ssl;
...
}
The issue stems from Nginx serving an HTTP server to a client expecting HTTPS on whatever port you're listening on. When you specify ssl in the listen directive, you clear this up on the server side.
This is a telltale error that you are serving HTTP from the HTTPS port.
You can easily test with telnet
telnet FQDN 443
GET / HTTP/1.0
[hit return twice]
and if you see regular HTTP document here [not some kind of error], you know that your configuration is incorrect and the responding server is not SSL encrypting the response.
Simple answer
If you are behind a proxy server, please set the proxy for curl. The curl is not able to connect to server so it shows wrong version number.
Set proxy by opening subl ~/.curlrc or use any other text editor. Then add the following line to file:
proxy= proxyserver:proxyport
For e.g. proxy = 10.8.0.1:8080
If you are not behind a proxy, make sure that the curlrc file does not contain the proxy settings.
Also check your /etc/hosts file. Wasted 2 hours on this. If you have an url rerouted to 127.0.0.1 or any other loopback, this will fail the ssl handshake.
In my case the cause of this error was that my web server was not configured to listen to IPv6 on SSL port 443. After enabling it the error disappeared.
Here's how you do it for Apache:
<VirtualHost ip.v4.address:443 ip:v::6:address:443>
...
</VirtualHost>
And for nginx:
listen 443 ssl http2;
listen [::]:443 ssl http2;
Thanks to #bret-weinraub,
I found that something is weird about the server's reply. After a bit of investigation, it turned out that I have a static IP in /etc/hosts file for the target domain and as they have changed their IP address I'm not getting to the correct server.
More simply in one line:
proxy=192.168.2.1:8080;curl -v example.com
eg. $proxy=192.168.2.1:8080;curl -v example.com
xxxxxxxxx-ASUS:~$ proxy=192.168.2.1:8080;curl -v https://google.com|head -c 15 % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Trying 172.217.163.46:443...
* TCP_NODELAY set
* Connected to google.com (172.217.163.46) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
Another possible cause of this problem is if you have not enabled the virtual host's configuration file in Apache (or if you don't have that virtual host at all) and the default virtual host in Apache is only configured for non-SSL connections -- ie there's no default virtual host which can talk SSL. In this case because Apache is listening on port 443 the request for the virtual host that doesn't exist will arrive at the default virtual host -- but that virtual host doesn't speak SSL.
In the case of using MySQL CLI to connect to an external MySQL DB, depending on the version of MySQL, you can pass the --ssl-mode=disabled like:
$ mysql --ssl-mode=disabled -h yourhost.tld -p
Or simply in your client config, for example in /etc/my.cnf.d/client.cnf:
[client]
ssl-mode=DISABLED
This is for dev and sometimes security and these things can be forfeited in certain situations in a closed, private dev environment.

curl with `-k` and without `-k`

When I am opening a url using curl without -k, my request is passing and I am able to see the expected result.
$ curl -vvv https://MYHOSTNAME/wex/archive.info -A SUKU$RANDOM
* Trying 10.38.202.192...
* Connected to MYHOSTNAME (10.38.202.192) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: *.MYCNAME
* Server certificate: ProdIssuedCA1
* Server certificate: InternalRootCA
> GET /wex/archive.info HTTP/1.1
> Host: MYHOSTNAME
> User-Agent: SUKU19816
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.10.2
< Date: Thu, 26 Jan 2017 01:08:40 GMT
< Content-Type: text/html;charset=ISO-8859-1
< Content-Length: 19
< Connection: keep-alive
< Set-Cookie: JSESSIONID=1XXXXXXXX3E58093E816FE62D81; Path=/wex/; HttpOnly
< X-WebProxy-Id: 220ffb81872a
<
status=Running
* Connection #0 to host MYHOSTNAME left intact
But when I am opening same url with -k its failing. To me its not making any sense since in my understanding the purpose of -k is only to skip certificate verification
$ curl -vvv https://MYHOSTNAME/wex/archive.info -A SUKU$RANDOM -k
* Trying 10.38.202.192...
* Connected to MYHOSTNAME (10.38.202.192) port 443 (#0)
* Server aborted the SSL handshake
* Closing connection 0
curl: (35) Server aborted the SSL handshake
Request flow:
SSL termination is happening on HAPROXY machine
HAPROXY will forward request to nginx
For troubleshooting this kind of problem, the --resolve option can be useful:
curl -k -I --resolve www.example.com:80:192.0.2.1 https://www.example.com/
Provide a custom address for a specific host and port pair. Using
this, you can make the curl requests(s) use a specified address and
prevent the otherwise normally resolved address to be used. Consider
it a sort of /etc/hosts alternative provided on the command line. The
port number should be the number used for the specific protocol the
host will be used for. It means you need several entries if you want
to provide address for the same host but different ports.
Especially if the site you’re trying to fetch from uses SNI: In that case you can use the --resolve option to specify the server name that gets used in the TLS client hello.
One troubleshooting step to try: update curl or compile it yourself from the sources and retry. For one thing, some curl versions (e.g., MacOS) supposedly don’t send SNI for -k/--insecure.
If that’s the issue you’ve hit and you can’t replace curl, there’s a workaround you can use that essentially involves creating your own CA and private keys and CSRs, and tweaks to your haproxy.
After setting it up, then in place of specifying -k/--insecure, you use --cacert or --capath:
curl https://example.com/api/endpoint --cacert certs/servers/example.com/chain.pem
curl https://example.com/api/endpoint --capath certs/ca
If the issue you’ve hit is due to SNI, you may also troubleshoot it with a site like https://sni.velox.ch/:
curl --insecure https://sni.velox.ch/
Otherwise, if it’s not SNI, then I recall seeing somewhere that -k/--insecure may not work as expected with some proxy configurations. So if you are going through some kind of proxy from the client side and you could somehow test directly without the proxy, that might be worth exploring.

OpenSSL installed and working, one client can connect but another connection refused

I have an API server (Debian Apache2) with OpenSSL installed and working. I also have a staging and production web server (also Debian Apache2, exactly the same spec - they are VM clones). All servers are on the same subnet. I can browse to the wsdl from my local machine on 443 successfully, and I can wget the wsdl successfully from my staging server on 443, but a wget from my production web server will not connect:
--2015-04-16 10:26:18-- https://www.example.com/index.php/api?wsdl
Resolving https://www.example.com (https://www.example.com)... XX.XX.XX.XX
Connecting to https://www.example.com (https://www.example.com)|XX.XX.XX.XX|:443... failed: Connection refused.
I can connect over https from a PHP nusoap client on staging no problem, but the same code on my production server returns:
wsdl error: HTTP ERROR: cURL ERROR: 7: couldn't connect to host
url: https://www.example.com/index.php/api?wsdl
content_type:
http_code: 0
header_size: 0
request_size: 0
filetime: -1
ssl_verify_result: 0
redirect_count: 0
total_time: 5.272228
namelookup_time: 5.271805
connect_time: 0
pretransfer_time: 0
size_upload: 0
size_download: 0
speed_download: 0
speed_upload: 0
download_content_length: -1
upload_content_length: -1
starttransfer_time: 0
redirect_time: 0
certinfo: Array
primary_ip: XX.XX.XX.XX
primary_port: 443
local_ip:
local_port: 0
redirect_url:
An openssl s_client -connect from both web servers produces the same output.
After my production server returns connection refused, there are no new entries in the API server's error.log, therefore this must be a client issue.
Is there a Debian-specific/internal firewall config I may have inadvertently changed that would prevent the one client from connecting to a secure web server over HTTPS and not another?
"Connection refused" usually indicates a failure to complete the initial TCP connection. Things to check include:
iptables, firewalls, hosts.deny
is apache listening on the interface/ip address the is attempting to connect to?
Does wget or curl work from the local server when you use http://127.0.0.1/ but not http://THE-SERVER'S-PUBLIC-IP-ADDRESS/ ?
What do you see when you run wget with --debug and --verbose ?