Forward SSL traffic and authentication certificates through HAProxy - ssl

I have an nginx from my client where I can POST successfully with:
curl -v --cacert ca.crt --cert client.crt --key client.key -POST https://nginx:8443/api/ -H 'Content-Type: application/json' -H 'cache-control: no-cache' -d#test.json
Now I installed an haproxy in front of nginx and I'm trying to do a POST the same way, unsuccessful:
curl -v --cacert ca.crt --cert client.crt --key client.key -POST http://haproxy:8443/api/ -H 'Content-Type: application/json' -H 'cache-control: no-cache' -d#test.json
Error:
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx</center>
Here is my haproxy configuration:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode tcp
log global
option tcplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend main *:8443
acl url_static path_beg -i /static /images /javascript /stylesheets
acl url_static path_end -i .jpg .gif .png .css .js
use_backend static if url_static
default_backend app
backend static
balance roundrobin
server static 127.0.0.1:8443
backend app
mode tcp
balance roundrobin
server nginx nginx01:8443
I want to forward SSL traffic through HAProxy and pass the certificates for authentication to nginx.
I know it doesn't make any sense to have two LBs but I can't modify nginx and the api server behind, but the clients will be internal.
As you can see at this point I'm able to reach nginx but haproxy doesn't pass the certificates and keys from the request to nginx backend.
Am I missing something? Is this something that I can achieve?
ps: If I'm setting 'ssl verify none' at backend, I'm getting 'No required SSL certificate was sent'.
If I'm setting 'send-proxy' at backend, I'm getting '400 Bad Request' from nginx.

You will need to add the ssl configuration to haproxy and set some headers which will be forwarded to the nginx.
# your other config from above
backend app
mode tcp
balance roundrobin
server nginx nginx01:8443 ssl ca-file <The ca from nginx backend>

the solution implemented was with SS/TLS pass-through from https://www.haproxy.com/documentation/haproxy/deployment-guides/tls-infrastructure/
Setting both frontend and backend to mode tcp I was able to pass the certificates and nginx validate and made the authentication.

Related

metallb round robin not working when accessed from external HAProxy

I have a sample app running in a kubernetes cluster with 3 replicas. I am exposing the app with type=LoadBalancer using metallb.
The external ip issued is 10.10.10.11
When I run curl 10.10.10.11 I get a different pod responding for each request as you would expect from round robin. This is the behaviour I want.
I have now setup HAProxy with a backend pointing to 10.10.10.11, however each time I access the HAProxy frontend, I get the same node responding to each request. If I keep refreshing I intermittently get different pods, sometimes after 20 refreshes, sometimes after 50+ refreshes. I have tried clearing my browser history, but that has no effect.
I assume it is my HAProxy config which is the cause the problem, perhaps caching? but I have not configured any caching. I am a HAProxy newbie, so I might be missing something.
Here is my HAProxy config.
I have tried both mode tcp and mode http, but both give the same result (the same pod responding to each request)
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /home/simon/haproxy/haproxy_certs
crt-base /home/simon/haproxy/haproxy_certs
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend https
bind *:443 ssl crt /home/simon/haproxy/haproxy_certs
timeout client 60s
mode tcp
#Hello App
acl ACL_hello_app hdr(host) -i hello.xxxxxxxxxdomain2.com
use_backend hello_app if ACL_hello_app
#Nginx App
acl ACL_nginx_app hdr(host) -i nginx.xxxxxxxxxdomain1.com
use_backend nginx_app if ACL_nginx_app
backend hello_app
timeout connect 10s
timeout server 100s
mode tcp
server hello_app 10.10.10.11:80
backend nginx_app
mode tcp
server nginx_app 10.10.10.10:80
UPDATE
Upon further testing the the issue seems to be related to the timeout client, timeout connect, timeout server. I reduce these to 1 second, then I get a different POD every 1 second, however these times are so short, I also get intermittent connection failures.
So, I also have the question. Is HAProxy able to work as a reverse proxy in front of another load balancer, or do I need to use another technology such as Nginx?
I eventually found the answer. I needed to use option http-server-close in my frontend settings.
frontend https
bind *:443 ssl crt /home/simon/haproxy/haproxy_certs
http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"
timeout client 5000s
option http-server-close
mode http
#Hello App
acl ACL_hello_app hdr(host) -i hello.soxprox.com
use_backend hello_app if ACL_hello_app
#Nginx App
acl ACL_nginx_app hdr(host) -i nginx.soxprox.com
use_backend nginx_app if ACL_nginx_app
With these settings I get correct round robin results from metallb

HAProxy stuck on ssl websocket (parse livequery)

My use-case is use HAProxy as ssl termination on Parse Server & Parse LiveQuery. I'm stuck on haproxy config because it can't connect websocket properly, and it's work only when connect to local (no ssl termination). Tested on iOS (react native app).
This is my haproxy config
global
maxconn 50000
log 127.0.0.1 local0
user haproxy
chroot /usr/share/haproxy
pidfile /run/haproxy.pid
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
ssl-default-bind-options no-sslv3
daemon
defaults
log global
retries 3
mode http
stats enable
stats uri /stats
stats realm Haproxy\ Statistics
stats auth xx:xx
option forwardfor
option http-server-close
option httplog
option dontlognull
frontend xxx_web
bind *:443 ssl crt /etc/haproxy/certs/xxx.com.pem
acl host_api_xxx hdr(host) -i api.xxx.com
acl is_websocket hdr(Upgrade) -i websocket
reqadd X-Forwarded-Port:\ 443
reqadd X-Forwarded-Proto:\ https
reqadd X-Forwarded-Scheme:\ https
redirect scheme https if !{ ssl_fc }
use_backend ws_xxx if is_websocket
use_backend api_xxx if host_api_xxx
backend ws_xxx
server ws_xxx 127.0.0.1:7777 check
backend api_xxx
balance roundrobin
reqadd X-Forwarded-Port:\ 443
reqadd X-Forwarded-Proto:\ https
reqadd X-Forwarded-Scheme:\ https
server api_xxx 127.0.0.1:7777 check # ssl verify none
Changing the mode to "mode tcp" should help.

Weird behavior/error trying to use HAProxy to forward requests to ELBs

We have an architecture where have several APIs on multiple hosts. Each API sits behind an AWS ELB. We want to put a proxy in front that will route requests based on URI to the ELB.
What I have so far mostly works, but about 3 out of 10 requests result in the following error (using cURL, but the problem isn't cURL):
curl: (35) Unknown SSL protocol error in connection to test-router.versal.com:-9847
I have a feeling that the ELB is the culprit. The SSL is terminated there.
Here is our HAProxy config:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 10240
user haproxy
group haproxy
daemon
debug
stats socket /var/run/haproxy.sock
log-send-hostname test-router.domain.com
description "HAProxy on test-router.domain.com"
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
option forwardfor
option httpclose
option dontlognull
option tcpka
maxconn 10240
timeout connect 10000ms
timeout client 600000ms
timeout server 600000ms
frontend public
bind *:80
bind *:443
acl elb0 path_beg /elb0
acl elb1 path_beg /elb1
use_backend elb0 if elb0
use_backend elb1 if elb1
bind 0.0.0.0:443 ssl crt /etc/ssl/cert.pem no-sslv3 ciphers AES128+EECDH:AES128+EDH
backend elb0
server server_vcat elb0.domain.com:443 ssl verify none
backend elb1
server server_laapi elb1.domain.com:443 ssl verify none
The SSL from curl is not terminated at the ELB. It is terminated at HAProxy, in your configuration...
bind 0.0.0.0:443 ssl crt /etc/ssl/cert.pem no-sslv3 ciphers AES128+EECDH:AES128+EDH
...and then an entirely different SSL session is established by HAProxy on the connection to the ELB:
server ... ssl verify none
An SSL problem with ELB could not possibly be propagated back to curl through HAProxy in this configuration.
The problem is in your HAProxy configuration, here:
bind *:443
Remove that line. It's redundant (and incorrect).
You are telling HAProxy to bind to port 443 twice: once speaking SSL, and once not speaking SSL.
So, statistically, on approximately 50% of your connection attempts, curl finds HAProxy is not speaking SSL on port 443 -- it's just speaking HTTP, and curl can't (and shouldn't) handle that gracefully.
I believe this (mis)configuration goes undetected by HAProxy, not because of an actual bug, but rather because of the way some things are implemented in the HAProxy internals, related to multi-process deployments and hot reloads, in which case it would be valid to have HAProxy bound to the same socket more than once.

HAproxy times out(504 error) with large POST bodies

I have 3 nodejs web-servers spun on an ubuntu box and HAproxy to load-balance those servers on the same box. HAproxy listens at port 80(http) and 443(https, with SSL termination). There's no SSL between the HAproxy server and the web-servers.
The POST call to one of the api without SSL, passes through with any value of content-length, but when I try to do the POST call with a content-length greater than 8055 on the HAproxy with SSL connection(port443), HAproxy times out giving a 504 Gateway Timeout error.
Also, if I give an "Expect:100 continue" header to the curl command, the server responds with some delay, which I don't want to exist. Below is how the HAproxy config file looks like:
global
stats socket /var/run/haproxy.sock mode 0777
log 127.0.0.1 local0 info
log 127.0.0.1 local1 info
chroot /usr/share/haproxy
uid nobody
gid nobody
nbproc 1
daemon
maxconn 50000
frontend localnodes:https
bind *:443 ssl crt /etc/ssl/private/443_private_ssl_in.pem no-sslv3
mode http
reqadd X-Forwarded-Proto:\ https
default_backend nodes
timeout client 30000
frontend localnodes-http
bind *:80
mode http
reqadd X-Forwarded-Proto:\ http
default_backend nodes
timeout client 30000
backend nodes
mode http
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
log global
timeout connect 3000
timeout server 30000
option httplog
option ssl-hello-chk
option httpchk GET /
http-check expect status 404
server nodejsweb01 127.0.0.1:8000 check
server nodejsweb02 127.0.0.1:8001 check
server nodejsweb03 127.0.0.1:8002 check
I have ensured that the nodejs web-servers behind have no problem, they work fine. I have tried increasing the 'timeout server' period, no effect.
Also tried a solution on this link that tells to give an option ssl ca-file to the backend nodes, as follows:
server nodejsweb01 127.0.0.1:8000 ssl ca-file /etc/ssl/certs/ca.pem check
server nodejsweb02 127.0.0.1:8001 ssl ca-file /etc/ssl/certs/ca.pem check
server nodejsweb03 127.0.0.1:8002 ssl ca-file /etc/ssl/certs/ca.pem check
but after this option HAproxy throws an error saying no servers available at the backend.
Please tell me what am I doing wrong in HAproxy conf file, so that I make the webservers respond successfully with the SSL connection
Try this minimum config:
frontend localnodes
bind *:80
bind *:443 ssl crt /etc/ssl/private/443_private_ssl_in.pem
mode http
default_backend nodes
backend nodes
mode http
option forwardfor
option httplog
server nodejsweb01 127.0.0.1:8000 check
server nodejsweb02 127.0.0.1:8001 check
server nodejsweb03 127.0.0.1:8002 check
I suspect it's those additional options.
It could also be the SSL cert file. Is your PEM file created from a self-signed cert? How is it structured?

SSL over HAProxy issue

I am trying to put an HTTPS web server behind HAProxy load balancer.
HTTPS server runs on Node.js on port 8000, had all the correct SSL certs and keys and works if I try access it directly via URL, e.g. https://ssltest.mydomain.com:8000/. I can also check SSL connection using openssl s_client -connect ssltest.mydomain.com:8000 and it shows data that seems correct.
The front end for this HTTPS server is another subdomain, e.g. app.mydomain.com and it runs on default SSL port 443.
But when I try to access https://app.mydomain.com from my browser, I get an SSL error. The above command line check also fails: openssl s_client -connect app.mydomain.com:443 -state -debug
Below is how my HA Proxy config looks like:
global
log /dev/log local0 info
log /dev/log local0 notice
maxconn 20000
user haproxy
group haproxy
daemon
spread-checks 5
defaults
log global
mode tcp
option dontlognull
retries 3
option redispatch
timeout connect 30s
timeout client 30s
timeout server 30s
timeout check 5s
balance roundrobin
frontend https_proxy
bind :443
acl is_app hdr_dom(host) -i app.mydomain.com
acl is_rest hdr_dom(host) -i rest.mydomain.com
acl is_io hdr_dom(host) -i io.mydomain.com
use_backend core_https if is_app
use_backend core_rest_api if is_rest
use_backend core_www_io if is_io
backend core_https
server www1 10.0.0.174:8000 maxconn 25 check inter 5s rise 18 fall 2
backend core_www_io
server io1 10.0.0.174:8100 maxconn 25 check inter 5s rise 18 fall 2
backend core_rest_api
server api1 10.0.0.174:8200 maxconn 25 check inter 5s rise 18 fall 2
I can't figure out why this does not work via HAProxy but works directly, and I would REALLY appreciate your suggestions.
You're confusing layer 4 and layer 7 load balancing. To separate requests using hdr_dom you need layer 7 that's only available for HTTP and as you may guess HTTPS works on layer 4.
So as haproxy can't inspect the host, none of your ifs are returning true and there is no backend selected, to fix you should add a default_backend entry.
frontend https_proxy
bind :443
acl is_app hdr_dom(host) -i app.mydomain.com
acl is_rest hdr_dom(host) -i rest.mydomain.com
acl is_io hdr_dom(host) -i io.mydomain.com
use_backend core_https if is_app
use_backend core_rest_api if is_rest
use_backend core_www_io if is_io
default_backend core_https