Haproxy TLS terminating and passthrough based on sni - ssl

I have similar path for the requests:
client mydomain.com -> nlb:443 -> haproxy -> cloudfront
client a.mydomain.com -> nlb:443 -> haproxy -> target_group_a
Main idea is do tls passthrough for the main domain name and send it to cloudfront without TLS termination. Requests into a.mydomain.com should pass to target_group_a and it should terminate tls. So my config for this is:
frontend main
bind *:443
mode tcp
option tcplog
log global
tcp-request inspect-delay 5s
acl is_main req_ssl_sni -i "${pDomainName}"
acl is_a req_ssl_sni -m beg "a"
tcp-request content accept if { req_ssl_hello_type 1 }
use_backend main if is_main
use_backend a if is_a
backend main
mode tcp
option ssl-hello-chk
server cloudfront "${pCloudFrontUrl}:443" check resolvers aws
backend a
mode tcp
server local 127.0.0.1:9666 send-proxy
frontend a
bind *:9666 ssl crt server.pem ca-file ca.pem verify required accept-proxy
mode http
default_backend proxy_a
backend proxy_a
mode http
server elb "${pServer}:80" check resolvers aws
Main record pass successfull and I get CloudFront SSL termination and everything is okay, but not for a.mydomain.com.
Also I tried to watch what SNI Haproxy is capture but I got only capture0: - in logs. I did like (right after tcp inspect line)
tcp-request content capture req_ssl_sni len 15
log-format "capture0: %[capture.req.hdr(0)]"
and it's strange because routing works.
I've tried a lot of possibilities.. For now I get SSL peer handshake failed, the server most likely requires a client certificate to connect error, but if I do listen frontend a on another port and in http mode everyting works fine.
Maybe I miss something basic or not, but I'm stuck on it for ages and maybe someone could help me.

For someone who is suffering or will suffer with that situation, just be sure that you are testing with gnu version of curl (or build it with properly libraries) because it doesn't work for me with BSD curl. My curl version and libs
curl 7.66.0 (x86_64-apple-darwin17.7.0) libcurl/7.66.0 SecureTransport zlib/1.2.11
Release-Date: 2019-09-11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets

Related

Can't authenticate for NiFi/NiFi Registry instance sitting behind HAProxy server

I have a NiFi and NiFi Registry instance sitting behind a HAProxy server. The NiFi instances are both secured using SSL. I am not able to pass the SSL information from the proxy server to the NiFi server. I also tried SSL Passthrough but that has some other limitations further on the line.
My current HAProxy config looks like this:
frontend https_in
bind *:443 ssl crt /etc/ssl/nifi/nifi-server.pem verify required ca-file /etc/ssl/nifi/nifi-ca.cert
mode http
option httplog
option http-server-close
acl is_registry path_beg /nifi-registry
use_backend nifi-registry if is_registry
default_backend nifi
backend nifi-registry
mode http
balance roundrobin
http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-ProxyScheme https
http-request set-header X-ProxyHost xx.xxx.xxx.xx
http-request set-header X-ProxyPort 443
server registry01 172.xx.xx.xxx:18443 check ssl verify none
When I browse to https://xx.xxx.xxx.xx:443/nifi-registry and select the client certificate I get the NiFi Registry UI but not logged in with my client user. I am not able to pass my SSL information to the NiFi servers. Following the documentation I've set some headers but they don't seem to have effect.
Am I missing something here ?
EDIT
So as suggested in the comments and mentioned above I also tried SSL Passthrough in tlc mode. With this I manage to pass the SSL authentication to the NiFi servers but I get in trouble with "invalid host header" message.
My HAProxy config:
frontend http_in
bind *:80 v4v6
mode http
redirect scheme https if !{ ssl_fc }
frontend nifi_registry_in
bind *:1443 v4v6
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nifi_registry
frontend nifi_in
bind *:2443 v4v6
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nifi
backend nifi_registry
mode tcp
balance roundrobin
server registry01 xxx.xx.xx.xxx:18443 check
backend nifi
mode tcp
balance roundrobin
server nifi01 xxx.xx.xx.xxx:9443 check
When browsing to the [public-haproxy-server-domain]:1443 I get the registry ui after authentication.
When browsing to the [public-haproxy-server-domain]:2443 I get the following error after authentication.
My NiFi config is
nifi.web.https.host=xxx.xx.xx.xxx
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=[public-haproxy-server-domain]:2443
So I've figured it out and I feel really stupid on this one. On the other hand I don't find this well documented. So I will post my answer here for other people in need.
Apparently for the nifi.web.proxy.host setting to work, the nifi.web.proxy.context.path setting should also be set. Simply putting a '/' as a value for this and everything works as expected.
nifi.web.proxy.context.path=/
nifi.web.proxy.host=[public-haproxy-server-domain]:2443

Is there a way to use a forwarding proxy as a backend (including authentication) in HAProxy?

I am quite new to HAProxy and want to achieve the following setup / packet flow:
Client -> HAProxy (as reverse proxy) -> Forwarding Proxy (HAProxy, IIS, Squid...) -> Internet -> server.example.com
I would like to have encrypted connections with TLS/SSL from the Client -> HAProxy and from HAProxy -> server.example.com
This means, that the forwarding proxy needs to support the HTTP CONNECT method, to establish a TCP tunnel and transmits packets without trying to interpret them. Over this TCP port I should be able to send bytes to server.example.com - TLS/SSL encrypted, so HTTPS.
Furthermore it could be, that authentication against the forwarding proxy is needed e.g. HTTP Basic authentication.
The software stack of my test setup is as follows:
Client Firefox 76.0.1
HA-Proxy version 1.6.3 2015/12/25
Squid Cache: Version 3.5.27
I have setup this HAProxy configuration:
global
# Standard settings
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
# Tuning
maxconn 2000
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-options no-sslv3
# Ensure a secure enough DH paramset
tune.ssl.default-dh-param 2048
defaults
log global
mode http
option httplog
option dontlognull
# Redispatch 503 errors on backends, reducing the number of 503 reaching clients
option redispatch
# We want to stall clients for as long as possible, before giving
# up with 503:
timeout connect 5m
# Clients must be acceptably responsive
timeout client 1m
# Server not as much...
timeout server 5m
# HTTPS server
frontend https-in
bind :443 ssl crt-ignore-err all crt /etc/haproxy/ssl/certkey.pem
# Don't serve HTTP directly, but redirect to same URL in https
redirect scheme https code 301 if !{ ssl_fc }
default_backend backend-proxy
backend backend-proxy
# Create the Authorization / Proxy-Authorization header value
# echo -n "user:password" | base64
http-request add-header Proxy-Authorization "Basic dXNlcjpwYXNzd29yZA=="
# We need to use the CONNECT method
http-request set-method CONNECT
# The proxyserver needs to know the full server name
http-request set-path server.example.com:443
server proxy 192.168.1.1:8080
In my test setup I use a Squid server as a forwarding proxy with the following configuration:
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/squid-passwd
acl basic_client proxy_auth REQUIRED
http_access deny !basic_client
http_access allow basic_client
http_access deny all
http_port 8080
coredump_dir /var/spool/squid
Using the Squid forwarding proxy in a regular browser on the same subnet including authentication works fine.
So when the first request comes in from the Client at the HAProxy it gets forwarded via the backend backend-proxy to the Forward Proxy (Squid). The CONNECT succeeds, as I see in the Squid log.
1591166403.966 60146 192.168.1.10 TCP_TUNNEL/200 39 CONNECT server.example.com:443 test HIER_DIRECT/6.7.8.9 -
(IP addresses were replaced with generic values)
The HAProxy log shows, that the correct backend is used:
Jun 3 08:39:57 localhost haproxy[3547]: 192.168.1.20:39398 [03/Jun/2020:08:38:56.855] https-in~ backend-proxy/proxy 154/0/13/209/60375 200 39 - - ---- 0/0/0/0/0 0/0 "GET /someurl HTTP/1.1"
(IP addresses were replaced with generic values)
So far so good. But I am unable to establish a successful communication to server.example.com from my Client. I think I have to use a second/other backend, which will not mangle the requests any more (exchange method and path) but instead use the given TCP port from the forwarding proxy to transmit the request.
How can I save the 'state' of the communication to my backend / proxy server in HAProxy, so the request could be resend to another backend?
How to extract and use the TCP port from the response of the Forwarding Proxy?
Is there a way to check, if the TCP tunnel on the Forwarding Proxy is still opened or do I need to request it using CONNECT every time before I want to use it?
EDIT:
I solved the situation by using stunnel as an intermediary to handle the TCP tunnel creation with CONNECT against the Forwarding Proxy.
If you have an upstream HTTP proxy (like squid) (not a socks proxy) and you want to have haproxy accept connections and open the tunnel thru the upstream proxy (such that the haproxy clients do not support doing so themselves via http/CONNECT method) on behalf of the clients, then this functionality does not exist in haproxy today.
I crated a branch that does this via server keyword proxy-tunnel.
The below example config will behave such that clients of haproxy that connect to port 20025 (haproxy) will result in haproxy establishing a http/CONNECT tunnel via the upstream proxy 172.16.0.99:50443 to 172.16.0.2:2023:
listen SMTP-20025
bind 0.0.0.0:20025
server TEST_SERVERVIA_PROXY 172.16.0.2:2023 proxy-tunnel 172.16.0.99:50443

haproxy reverse ssl termination

How can I achieve reverse SSL termination with ha proxy?
From my backend via HAproxy I need to a https enabled web service. How can I successfully proxy all traffic to that service via HAProxy?
Below results in Unable to communicate securely with peer: requested domain name does not match the server's certificate.
frontend foofront
bind 127.0.0.1:443
mode tcp
default_backend foo
backend fooback
mode tcp
balance leastconn
server foo foo.bar.com:443 check
With HAProxy you usually have two options for handling TLS-related scenarios. TLS Passthrough and TLS Termination.
TLS Passthrough
Looks like you're trying to do this in the example you gave.
In this mode, HAProxy does not touch traffic in any way, but is just forwarding it to the backend. When TLS is involved, that means that the backend has to have a proper certificate for a domain it's accessed from - if your HAProxy is handling traffic for myexample.com, backend servers will need to have appropriate certificates for myexample.com installed.
You can always check which certificate is served by using openssl s_client:
openssl s_client -connect localhost:443
TLS Termination
Alternatively, you can terminate TLS traffic on HAProxy itself. This will allow you to use any backend (both encrypted and unencrypted). In this case, HAProxy itself decrypts traffic for myexample.com and forwards it to backend.
In your case, configuration would look something like:
frontend foofront
bind 127.0.0.1:80
bind 127.0.0.1:443 ssl crt /path/to/cert/for/myexample.com
mode tcp
default_backend foo
backend foo
mode tcp
balance leastconn
server foo foo.bar.com:443 check ssl verify none # or verify all to enforce ssl checking
You can find more info on both approaches here.
Hope this helps.

Ejabberd with haproxy ssl termination

anyone got successful with ejabberd + ssl termination with haproxy 1.7?
frontend ejabberd
bind *:4000 ssl crt /etc/haproxy/certs/my-domain.com.pem
mode tcp
timeout client 3h
option tcplog
option clitcpka
default_backend ejabberd
backend ejabberd
mode tcp
timeout server 60m
option redispatch
option srvtcpka
option independent-streams
balance leastconn
default-server inter 5s rise 3 fall 3 on-marked-down shutdown-sessions
server ejabberd localhost:5222 check slowstart 120s
it works fine without ssl crt. My smack client gives me a No response received within reply timeout. Timeout was 5000ms (~5s). While waiting for establishing TLS.
I also tried with or without ttls on ejabberd conf
I was able to have the exact configuration working, you need to:
Disable tls in ejabberd since HAProxy is doing the ssl offloading and ejabberd is hosted in the same machine:
Comment the certfile line
Comment the starttls line
Correctly configure your client, I have used pidgin, but you should be able to do the same with smack:
Configure correctly address and port (4000 here)
Ensure that you use "old style TLS" and not STARTTLS, the latter won't work with HAProxy.
Otherwise, if you want clients to use STARTTLS configuration, then you have to configure HAProxy to proxy XMPP encrypted traffic (removing "ssl crt .." )

Weird behavior/error trying to use HAProxy to forward requests to ELBs

We have an architecture where have several APIs on multiple hosts. Each API sits behind an AWS ELB. We want to put a proxy in front that will route requests based on URI to the ELB.
What I have so far mostly works, but about 3 out of 10 requests result in the following error (using cURL, but the problem isn't cURL):
curl: (35) Unknown SSL protocol error in connection to test-router.versal.com:-9847
I have a feeling that the ELB is the culprit. The SSL is terminated there.
Here is our HAProxy config:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 10240
user haproxy
group haproxy
daemon
debug
stats socket /var/run/haproxy.sock
log-send-hostname test-router.domain.com
description "HAProxy on test-router.domain.com"
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
option forwardfor
option httpclose
option dontlognull
option tcpka
maxconn 10240
timeout connect 10000ms
timeout client 600000ms
timeout server 600000ms
frontend public
bind *:80
bind *:443
acl elb0 path_beg /elb0
acl elb1 path_beg /elb1
use_backend elb0 if elb0
use_backend elb1 if elb1
bind 0.0.0.0:443 ssl crt /etc/ssl/cert.pem no-sslv3 ciphers AES128+EECDH:AES128+EDH
backend elb0
server server_vcat elb0.domain.com:443 ssl verify none
backend elb1
server server_laapi elb1.domain.com:443 ssl verify none
The SSL from curl is not terminated at the ELB. It is terminated at HAProxy, in your configuration...
bind 0.0.0.0:443 ssl crt /etc/ssl/cert.pem no-sslv3 ciphers AES128+EECDH:AES128+EDH
...and then an entirely different SSL session is established by HAProxy on the connection to the ELB:
server ... ssl verify none
An SSL problem with ELB could not possibly be propagated back to curl through HAProxy in this configuration.
The problem is in your HAProxy configuration, here:
bind *:443
Remove that line. It's redundant (and incorrect).
You are telling HAProxy to bind to port 443 twice: once speaking SSL, and once not speaking SSL.
So, statistically, on approximately 50% of your connection attempts, curl finds HAProxy is not speaking SSL on port 443 -- it's just speaking HTTP, and curl can't (and shouldn't) handle that gracefully.
I believe this (mis)configuration goes undetected by HAProxy, not because of an actual bug, but rather because of the way some things are implemented in the HAProxy internals, related to multi-process deployments and hot reloads, in which case it would be valid to have HAProxy bound to the same socket more than once.