HAProxy http check on for ssl? - ssl-certificate

I have some web servers which are MySQL backend. An HAProxy is in front of those web servers. All the web servers are using https.
I tried to use the http check option on both http and https to make sure if the database connection was lost, the HAProxy will failover to another node. My haproxy configuration file:
global
log /dev/log local0
maxconn 4096
#debug
#quiet
user haproxy
group haproxy
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen http
bind *:80
mode http
balance roundrobin
stats enable
stats auth haproxy:haproxy
cookie JSESSIONID prefix
stick on src table https
option http-server-close
option forwardfor
default-server inter 5s fall 2
option httpchk
http-check expect ! string Database\ error
server sitea 192.168.0.20 cookie sitea check port 80
server siteb 192.168.0.21 cookie siteb check port 80
listen https
bind *:443
mode tcp
balance roundrobin
stick-table type ip size 5000k expire 2h store conn_cur
stick on src
option tcplog
option ssl-hello-chk
default-server inter 5s fall 2
option httpchk
http-check expect ! string Database\ error
server sitea 192.168.0.20:443 check ssl verify none
server siteb 192.168.0.21:443 check ssl verify none
Look at the last two lines. If I specified "ssl verify none", my HAProxy can successfully check both Apache and MySQL status. However, I can't open the webpage via https(it prompts me This site can’t provide a secure connection. ERR_SSL_PROTOCOL_ERROR).
If I remove that parameter, the webpage can be opened again, but all the https servers status become DOWN in the HAProxy.
P.S. I'm using self-signed certificate currently, because I'm still on testing.

I have found the solution: since I am using https on apache nodes, I have to copy ssl certificates content to haproxy. To do that, copy and merge both private key and the certificate content issued by the CA into one single file(In my case, I put it into /etc/haproxy/haproxy.pem).
Modify the haproxy configuration, change
bind *:443
To
bind *:443 ssl crt /etc/haproxy/haproxy.pem

Related

Weird behavior/error trying to use HAProxy to forward requests to ELBs

We have an architecture where have several APIs on multiple hosts. Each API sits behind an AWS ELB. We want to put a proxy in front that will route requests based on URI to the ELB.
What I have so far mostly works, but about 3 out of 10 requests result in the following error (using cURL, but the problem isn't cURL):
curl: (35) Unknown SSL protocol error in connection to test-router.versal.com:-9847
I have a feeling that the ELB is the culprit. The SSL is terminated there.
Here is our HAProxy config:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 10240
user haproxy
group haproxy
daemon
debug
stats socket /var/run/haproxy.sock
log-send-hostname test-router.domain.com
description "HAProxy on test-router.domain.com"
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
option forwardfor
option httpclose
option dontlognull
option tcpka
maxconn 10240
timeout connect 10000ms
timeout client 600000ms
timeout server 600000ms
frontend public
bind *:80
bind *:443
acl elb0 path_beg /elb0
acl elb1 path_beg /elb1
use_backend elb0 if elb0
use_backend elb1 if elb1
bind 0.0.0.0:443 ssl crt /etc/ssl/cert.pem no-sslv3 ciphers AES128+EECDH:AES128+EDH
backend elb0
server server_vcat elb0.domain.com:443 ssl verify none
backend elb1
server server_laapi elb1.domain.com:443 ssl verify none
The SSL from curl is not terminated at the ELB. It is terminated at HAProxy, in your configuration...
bind 0.0.0.0:443 ssl crt /etc/ssl/cert.pem no-sslv3 ciphers AES128+EECDH:AES128+EDH
...and then an entirely different SSL session is established by HAProxy on the connection to the ELB:
server ... ssl verify none
An SSL problem with ELB could not possibly be propagated back to curl through HAProxy in this configuration.
The problem is in your HAProxy configuration, here:
bind *:443
Remove that line. It's redundant (and incorrect).
You are telling HAProxy to bind to port 443 twice: once speaking SSL, and once not speaking SSL.
So, statistically, on approximately 50% of your connection attempts, curl finds HAProxy is not speaking SSL on port 443 -- it's just speaking HTTP, and curl can't (and shouldn't) handle that gracefully.
I believe this (mis)configuration goes undetected by HAProxy, not because of an actual bug, but rather because of the way some things are implemented in the HAProxy internals, related to multi-process deployments and hot reloads, in which case it would be valid to have HAProxy bound to the same socket more than once.

HAProxy health check in tcp mode on https 404 status code

I have two servers each running one Wildfly application server with one service available via https. The service is taking care of the https encryption. In front of the two servers I have an HAProxy as a load balancer in tcp mode to pass the ssl traffic through to the two services.
The HAProxy health check only checks if the server is online, not the service. If the service is not running Wildfly returns:
<html><head><title>Error</title></head><body>404 - Not Found</body></html>
which HAProxy interprets as healthy.
HAProxy config:
global
maxconn 2000
defaults
log global
mode http
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 10000
timeout server 10000
listen backend
bind *:8443
mode tcp
balance roundrobin
option httpclose
server backend1 wildfly:8443 check
server backend2 xxx.xxx.xxx.xxx:8443 check
How can I make HAProxy understand that 404 - Not Found is not healthy.
Two lines did the trick:
option httpchk /server
httpchk tells HAProxy to send an http request and check the response status
/server specifies the URI / Subdomain of my service
server backend1 wildfly:8443 check check-ssl verify none
check-ssl tells HAProxy to check via https instead of http
verify none tells HAProxy to trust the ssl certificate of the service (alternativly you can specify a .pem file)
Full HAProxy config:
global
maxconn 2000
defaults
log global
mode http
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 10000
timeout server 10000
listen backend
bind *:8443
mode tcp
balance roundrobin
option httpchk /server
server backend1 xxx.xxx.xxx.xxx:8443 check check-ssl verify none
server backend2 xxx.xxx.xxx.xxx:8443 check check-ssl verify none

HAproxy times out(504 error) with large POST bodies

I have 3 nodejs web-servers spun on an ubuntu box and HAproxy to load-balance those servers on the same box. HAproxy listens at port 80(http) and 443(https, with SSL termination). There's no SSL between the HAproxy server and the web-servers.
The POST call to one of the api without SSL, passes through with any value of content-length, but when I try to do the POST call with a content-length greater than 8055 on the HAproxy with SSL connection(port443), HAproxy times out giving a 504 Gateway Timeout error.
Also, if I give an "Expect:100 continue" header to the curl command, the server responds with some delay, which I don't want to exist. Below is how the HAproxy config file looks like:
global
stats socket /var/run/haproxy.sock mode 0777
log 127.0.0.1 local0 info
log 127.0.0.1 local1 info
chroot /usr/share/haproxy
uid nobody
gid nobody
nbproc 1
daemon
maxconn 50000
frontend localnodes:https
bind *:443 ssl crt /etc/ssl/private/443_private_ssl_in.pem no-sslv3
mode http
reqadd X-Forwarded-Proto:\ https
default_backend nodes
timeout client 30000
frontend localnodes-http
bind *:80
mode http
reqadd X-Forwarded-Proto:\ http
default_backend nodes
timeout client 30000
backend nodes
mode http
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
log global
timeout connect 3000
timeout server 30000
option httplog
option ssl-hello-chk
option httpchk GET /
http-check expect status 404
server nodejsweb01 127.0.0.1:8000 check
server nodejsweb02 127.0.0.1:8001 check
server nodejsweb03 127.0.0.1:8002 check
I have ensured that the nodejs web-servers behind have no problem, they work fine. I have tried increasing the 'timeout server' period, no effect.
Also tried a solution on this link that tells to give an option ssl ca-file to the backend nodes, as follows:
server nodejsweb01 127.0.0.1:8000 ssl ca-file /etc/ssl/certs/ca.pem check
server nodejsweb02 127.0.0.1:8001 ssl ca-file /etc/ssl/certs/ca.pem check
server nodejsweb03 127.0.0.1:8002 ssl ca-file /etc/ssl/certs/ca.pem check
but after this option HAproxy throws an error saying no servers available at the backend.
Please tell me what am I doing wrong in HAproxy conf file, so that I make the webservers respond successfully with the SSL connection
Try this minimum config:
frontend localnodes
bind *:80
bind *:443 ssl crt /etc/ssl/private/443_private_ssl_in.pem
mode http
default_backend nodes
backend nodes
mode http
option forwardfor
option httplog
server nodejsweb01 127.0.0.1:8000 check
server nodejsweb02 127.0.0.1:8001 check
server nodejsweb03 127.0.0.1:8002 check
I suspect it's those additional options.
It could also be the SSL cert file. Is your PEM file created from a self-signed cert? How is it structured?

Haproxy backend server down due to layer 6 invalid response failed ssl handashake?

The scenario is we have two servers which are in different network . We want to have ssl communication from client to front-end and from front-end to back-end !
the front-end able to get ssl traffic and terminate the ssl, after that in back-end ssl communication is not happening the error follows as
"Server nodes/web02 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration:546ms "
This is my hapoxy config file
global
log 127.0.0.1 local1 debug
maxconn 4000
daemon
uid 99
gid 99
stats socket /tmp/haproxy.stats level admin
defaults
mode http
log global
option forwardfor
option http-server-close
timeout server 5s
timeout connect 5s
timeout client 5s
frontend www-https
bind <Ip-address>:443 ssl crt /home/user/SSL/domain-name.in.pem
reqadd X-Forwarded-Proto:\ https
default_backend nodes
backend nodes
balance roundrobin
cookie JSESSIONID prefix indirect nocache
server web01 <IP-address>:8443 ssl verify none check cookie web01
server web02 <IP-address>:8443 ssl crt /home/SSL/domain-name.in.pem ca-file /home/SSL/gdig2.crt verify required check cookie web02
any help in this greatly appreciated

HAProxy with SSL (https) and Sticky Session

I need to setup Load balancer as an alternative for ELB for Amazon as they have issue in connection timeout.
Currently, Im using HAProxy and it works normally. However, I need to use SSL for users who wants to connect in https (port 443) to the backend apache servers plus sticky session.
What will be the configuration would looks like? I heard that HAProxy doesn't support SSL in native and can use stunnel or nginx / apache to handle the SSL termination.
I would appreciate anyone to share their knowledge and experiences.
Thanks.
James
To http use something like that.
Change the XXX.XXX.XXX.XXX to your IP address.
listen example-cluster XXX.XXX.XXX.XXX:80
mode http
stats enable
stats auth user:password
stick store-request src
stick-table type ip size 200k expire 2m
balance source
cookie JSESSIONID prefix
option httplog
option httpclose
option forwardfor
option persist
option redispatch
option httpchk HEAD /check.txt HTTP/1.0
server example-webl XXX.XXX.XXX.XXX:80 cookie A check
server example-web2 XXX.XXX.XXX.XXX:80 cookie B check
server example-web3 XXX.XXX.XXX.XXX:80 cookie C check
server example-web4 XXX.XXX.XXX.XXX:80 cookie D check
server example-web5 XXX.XXX.XXX.XXX:80 cookie E check
To your SSL use the mode tcp with balance source:
listen example-cluster-ssl XXX.XXX.XXX.XXX:443
mode tcp
reqadd X-Forwarded-Proto:\ https
stick store-request src
stick-table type ip size 200k expire 2m
option persist
option redispatch
option ssl-hello-chk
balance source
server example-webl XXX.XXX.XXX.XXX:443 check
server example-web2 XXX.XXX.XXX.XXX:443 check
server example-web3 XXX.XXX.XXX.XXX:443 check
server example-web4 XXX.XXX.XXX.XXX:443 check
server example-web5 XXX.XXX.XXX.XXX:443 check
Another way is your upgrade your haproxy to version 1.5, in that version have support to ssl but isn't stable yet.
Take a look at the Stud project on github, which combines extremely well with haproxy, is very performant, scalable, and uses very little resource. Many users are switching to it right now because it's simple and efficient.