HAProxy with SSL passthrough to multiple domains with multiple backends - apache

I need to setup a load balancer for all our applications.
At the moment all our applications are clustered (2-node appservers, and 1 apache on each node as well) and we do not have a LB so we just point our DNS alias to the first webserver of each node, making the second node useless (have to manually do a DNS switch in case of a failure of node1, and we don't have load balanced https queries).
Each application uses SSL with a specific domain & SSL certificate. we cannot accept to decrypt SSL and send unencrypted traffic to the backends as the LB might be located in another country etc. so we need to use passthrough.
Before anything, i just wanted to know if this is actually possible in HAProxy or not ?
I am talking about ~50 different applications. Our LB configuration would have to be HA so i guess we'll use something like keepalived with a shared VIP for HAProxy itself.
The setup would look like this i suppose :
domain-a.com-' '-> backend_dom_a -> 1.1.1.1 (app node1 dom a)
| | 1.1.1.2 (app node2 dom a)
domain-b.com-' '-> backend_dom_b -> 2.1.1.1 (app node1 dom b)
| | 2.1.1.2 (app node2 dom b)
domain-c.com-' '-> backend_dom_c -> 3.1.1.1 (app node1 dom c)
| | 3.1.1.2 (app node2 dom c)
domain-N.com-' '-> backend_dom_N -> 4.1.1.1 (app node1 dom N)
| | 4.1.1.2 (app node2 dom N)
+-> haproxy -+
Thanks for your support, best regards

FYI i'm using this configuration that works like a charm.
i have replaced the values in the files to hide our domains & hostnames, and limited the numbers of urls/backends but we have about 50 running now with the load balancer forwarding requests to many apache servers (and each apache forwards requests to tomcat servers behind)
feel free if you have any question
we use balance source to ensure session stickyness
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
daemon
user haproxy
group haproxy
log /dev/log local6 notice
log /dev/log local5 info
maxconn 50000
#chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode tcp
option tcplog
log global
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
#---------------------------------------------------------------------
# dedicated stats page
#---------------------------------------------------------------------
listen stats
mode http
bind :22222
stats enable
stats uri /haproxy?stats
stats realm Haproxy\ Statistics
stats auth <mylogin>:<mypass>
stats refresh 30s
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main_https_listen
bind <ip address>:443
mode tcp
option tcplog
log global
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
#---------------------------------------------------------------------
# Common HAProxy nodes configuration
#---------------------------------------------------------------------
# -------------------------------
# ACLs
# -------------------------------
acl acl_SIT_AT35073 req.ssl_sni -i <app_url1>.my.domain.net # SIT_AT35073 is just an internal code we use, but you can use any alias
acl acl_SIT_AT34305 req.ssl_sni -i <app_url2>.my.domain.net
acl acl_SIT_AT28548 req.ssl_sni -i <app_urlN>.my.domain.net
# -------------------------------
# Conditions
# -------------------------------
use_backend backend_SIT_AT35073 if acl_SIT_AT35073 # same here
use_backend backend_SIT_AT34305 if acl_SIT_AT34305
use_backend backend_SIT_AT28548 if acl_SIT_AT28548
#---------------------------------------------------------------------
# Backends
#---------------------------------------------------------------------
# APP 1
backend backend_SIT_AT35073
description APPNAME1
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT35073_1 <apache_server1>.my.domain.net:443 check
server server_SIT_AT35073_2 <apache_server2>.my.domain.net:443 check
# APP 2
backend backend_SIT_AT34305
description APPNAME2
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT34305_1 <apache_server3>.my.domain.net:443 check
server server_SIT_AT34305_2 <apache_server4>.my.domain.net:443 check
# APP N
backend backend_SIT_AT28548
description APPNAMEN
mode tcp
balance source
option ssl-hello-chk
server server_SIT_AT28548_1 <apache_server5>.my.domain.net:443 check
server server_SIT_AT28548_2 <apache_server6>.my.domain.net:443 check

I think you have two options:
pass the traffic through to the backend by using the TCP mode in haproxy frontend and backend. This has the benefit that your backend SSL certificate is passed through. Though you lose the possibility to have one SSL termination in your site. So I present you
Have one (usual) SSL certificate, acting as termination for your site and enable SSL between your backend and haproxy instance. This gives you the advantage that you still have only one entry point but different backends with unique certificates.
The second option might look like this:
frontend f_foo
bind :443 ssl crt /path/to/bundle
mode http
log global
use_backend b2_foo
backend be_foo
mode http
timeout connect 5s
server FOO address:port ssl check crt /path/to/client/bundle force-tlsv10 verify none
The drawback is that you need a client certificate for each backend server but that should be easily automatable.

AS more of an update answer for multi domain configs I use the below for routing different domains.
in the frontend is where you bind the port and add the certs which multiple have to be on the same line afaik.
frontend https_in
bind *:443 ssl crt /link/to/cert+key-file.pem crt /link/to/cert+key-file.pem
The acl host is where you specify the domain name and which backend to use based on that domain name.
acl host_example.com hdr(host) -i example.com
use_backend BACKEND_NAME if host_example.com
The backend where you specify the server that domain is running on.
backend BACKEND_NAME
mode http
option httpclose
option forwardfor
cookie JSESSIONID prefix
server server-name server-ip:443 check ssl verify none

Related

Openshift on VCenter UPI installation bootstrap not working

Trying to install openshift in vcenter. The bootstrap is booting up but not grabbing the bootstrap.ign from my web server.
set semanage fcontext public_content_rw_t on /var/www/openshift directory but I am getting
DEBUG Still waiting for the Kubernetes API: Get "https://api.cl.ops.local:6443/version": http: server gave HTTP response to HTTPS client
my haproxy server looks like this I am only testing with static IP addreses.
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
# utilize system-wide crypto-policies
#ssl-default-bind-ciphers PROFILE=SYSTEM
#ssl-default-server-ciphers PROFILE=SYSTEM
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
listen stats
bind :9000
mode http
stats enable
stats uri /
monitor-uri /healthz
frontend api
bind *:6443
default_backend api
mode http
backend api
balance source
server bootstrap.cl.ops.local 192.168.61.22:6443 check
# server master0 192.168.61.23:6443 check
server master0.cl.ops.local 192.168.61.23:6443 check
server master1 192.168.61.24:6443 check
server master2 192.168.61.25:6443 check
mode http
frontend api-int
bind *:22623
default_backend api-int
mode http
backend api-int
balance source
server bootstrap.cl.ops.local 192.168.61.22:22623 check # Can be removed or commented out after install completes
server master0.cl.ops.local 192.168.61.23:22623 check
# server master0 192.168.61.23:22623 check
server master1 192.168.61.24:22623 check
server master2 192.168.61.25:22623 check
mode http
frontend secure
bind *:443
default_backend secure
mode http
backend secure
balance source
server worker0 192.168.61.65:443 check
server worker1 192.168.61.66:443 check
server worker2 192.168.61.67:443 check
mode http
frontend insecure
bind *:80
default_backend insecure
mode http
backend insecure
balance source
server worker0 192.168.61.65:80 check
server worker1 192.168.61.66:80 check
server worker2 192.168.61.67:80 check
mode h 14,3 11%

Is there a way to use a forwarding proxy as a backend (including authentication) in HAProxy?

I am quite new to HAProxy and want to achieve the following setup / packet flow:
Client -> HAProxy (as reverse proxy) -> Forwarding Proxy (HAProxy, IIS, Squid...) -> Internet -> server.example.com
I would like to have encrypted connections with TLS/SSL from the Client -> HAProxy and from HAProxy -> server.example.com
This means, that the forwarding proxy needs to support the HTTP CONNECT method, to establish a TCP tunnel and transmits packets without trying to interpret them. Over this TCP port I should be able to send bytes to server.example.com - TLS/SSL encrypted, so HTTPS.
Furthermore it could be, that authentication against the forwarding proxy is needed e.g. HTTP Basic authentication.
The software stack of my test setup is as follows:
Client Firefox 76.0.1
HA-Proxy version 1.6.3 2015/12/25
Squid Cache: Version 3.5.27
I have setup this HAProxy configuration:
global
# Standard settings
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
# Tuning
maxconn 2000
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-options no-sslv3
# Ensure a secure enough DH paramset
tune.ssl.default-dh-param 2048
defaults
log global
mode http
option httplog
option dontlognull
# Redispatch 503 errors on backends, reducing the number of 503 reaching clients
option redispatch
# We want to stall clients for as long as possible, before giving
# up with 503:
timeout connect 5m
# Clients must be acceptably responsive
timeout client 1m
# Server not as much...
timeout server 5m
# HTTPS server
frontend https-in
bind :443 ssl crt-ignore-err all crt /etc/haproxy/ssl/certkey.pem
# Don't serve HTTP directly, but redirect to same URL in https
redirect scheme https code 301 if !{ ssl_fc }
default_backend backend-proxy
backend backend-proxy
# Create the Authorization / Proxy-Authorization header value
# echo -n "user:password" | base64
http-request add-header Proxy-Authorization "Basic dXNlcjpwYXNzd29yZA=="
# We need to use the CONNECT method
http-request set-method CONNECT
# The proxyserver needs to know the full server name
http-request set-path server.example.com:443
server proxy 192.168.1.1:8080
In my test setup I use a Squid server as a forwarding proxy with the following configuration:
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/squid-passwd
acl basic_client proxy_auth REQUIRED
http_access deny !basic_client
http_access allow basic_client
http_access deny all
http_port 8080
coredump_dir /var/spool/squid
Using the Squid forwarding proxy in a regular browser on the same subnet including authentication works fine.
So when the first request comes in from the Client at the HAProxy it gets forwarded via the backend backend-proxy to the Forward Proxy (Squid). The CONNECT succeeds, as I see in the Squid log.
1591166403.966 60146 192.168.1.10 TCP_TUNNEL/200 39 CONNECT server.example.com:443 test HIER_DIRECT/6.7.8.9 -
(IP addresses were replaced with generic values)
The HAProxy log shows, that the correct backend is used:
Jun 3 08:39:57 localhost haproxy[3547]: 192.168.1.20:39398 [03/Jun/2020:08:38:56.855] https-in~ backend-proxy/proxy 154/0/13/209/60375 200 39 - - ---- 0/0/0/0/0 0/0 "GET /someurl HTTP/1.1"
(IP addresses were replaced with generic values)
So far so good. But I am unable to establish a successful communication to server.example.com from my Client. I think I have to use a second/other backend, which will not mangle the requests any more (exchange method and path) but instead use the given TCP port from the forwarding proxy to transmit the request.
How can I save the 'state' of the communication to my backend / proxy server in HAProxy, so the request could be resend to another backend?
How to extract and use the TCP port from the response of the Forwarding Proxy?
Is there a way to check, if the TCP tunnel on the Forwarding Proxy is still opened or do I need to request it using CONNECT every time before I want to use it?
EDIT:
I solved the situation by using stunnel as an intermediary to handle the TCP tunnel creation with CONNECT against the Forwarding Proxy.
If you have an upstream HTTP proxy (like squid) (not a socks proxy) and you want to have haproxy accept connections and open the tunnel thru the upstream proxy (such that the haproxy clients do not support doing so themselves via http/CONNECT method) on behalf of the clients, then this functionality does not exist in haproxy today.
I crated a branch that does this via server keyword proxy-tunnel.
The below example config will behave such that clients of haproxy that connect to port 20025 (haproxy) will result in haproxy establishing a http/CONNECT tunnel via the upstream proxy 172.16.0.99:50443 to 172.16.0.2:2023:
listen SMTP-20025
bind 0.0.0.0:20025
server TEST_SERVERVIA_PROXY 172.16.0.2:2023 proxy-tunnel 172.16.0.99:50443

HAPROXY roundrobin DNS on backend server domain

I have read allot of posts and tried several things, but can't seem to get what I want working and stable.
I have HAproxy setup as a pure proxy. The IP/domain of the HAproxy passes ALL to the backend server.
My issues is that the backend server domain has 2 IPs in DNS:
1.1.1.1
2.2.2.2
When the provider switches or removes an IP, HAproxy does not update to the "new" IP and gives a backend not reachable error in the logs:
Message from syslogd#localhost at Jul 18 16:15:02 ...
haproxy[3233]: backend b-http has no server available!
But there is a valid and working server on one of the ips. A restart which forces HAproxy to do a lookup normally fixes this, but I'd prefer for it to be automatic.
On HAproxy version haproxy-1.5.18 I have:
frontend f-http
bind :80
default_backend b-http
backend b-http
option forwardfor
server web-1 domain.com:80 check
I have tried on HAproxy version haproxy-1.7.8-1 I have:
resolvers public-dns
nameserver dns2 8.8.8.8:53
nameserver dns1 8.8.4.4:53
hold valid 10s
frontend f-http
bind :80
default_backend b-http
backend b-http
option forwardfor
server web-1 domain.com:80 resolvers public-dns check
As above dig on domain.com would return 2 A records. I'm thinking that there must be some config which will continue to check the IPs for a valid/working IP and start to use that one on the fly.
Any help is very much appreciated.

HAProxy http check on for ssl?

I have some web servers which are MySQL backend. An HAProxy is in front of those web servers. All the web servers are using https.
I tried to use the http check option on both http and https to make sure if the database connection was lost, the HAProxy will failover to another node. My haproxy configuration file:
global
log /dev/log local0
maxconn 4096
#debug
#quiet
user haproxy
group haproxy
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen http
bind *:80
mode http
balance roundrobin
stats enable
stats auth haproxy:haproxy
cookie JSESSIONID prefix
stick on src table https
option http-server-close
option forwardfor
default-server inter 5s fall 2
option httpchk
http-check expect ! string Database\ error
server sitea 192.168.0.20 cookie sitea check port 80
server siteb 192.168.0.21 cookie siteb check port 80
listen https
bind *:443
mode tcp
balance roundrobin
stick-table type ip size 5000k expire 2h store conn_cur
stick on src
option tcplog
option ssl-hello-chk
default-server inter 5s fall 2
option httpchk
http-check expect ! string Database\ error
server sitea 192.168.0.20:443 check ssl verify none
server siteb 192.168.0.21:443 check ssl verify none
Look at the last two lines. If I specified "ssl verify none", my HAProxy can successfully check both Apache and MySQL status. However, I can't open the webpage via https(it prompts me This site can’t provide a secure connection. ERR_SSL_PROTOCOL_ERROR).
If I remove that parameter, the webpage can be opened again, but all the https servers status become DOWN in the HAProxy.
P.S. I'm using self-signed certificate currently, because I'm still on testing.
I have found the solution: since I am using https on apache nodes, I have to copy ssl certificates content to haproxy. To do that, copy and merge both private key and the certificate content issued by the CA into one single file(In my case, I put it into /etc/haproxy/haproxy.pem).
Modify the haproxy configuration, change
bind *:443
To
bind *:443 ssl crt /etc/haproxy/haproxy.pem

HaProxy - group tcp and http hosts dependent of each other

I have the following scenario:
Haproxy is running in front of my two groups of servers:
two http servers (active / backup)
two tcp servers (active / backup)
I now want to fail over from the active sides to the backup ones of ANY of the active services goes down (fail over HTTP and TCP at the same time).
Is there any way to do so in HAproxy? I so far was only able to fail over to one of them depending on the protocol but not both. Can these be grouped?
i was wondering if the can be done via ACLs and things like the fe_conn directive
I think haproxy's nbsrv works here. If your nbsrv count, number of healthy instances, falls below desired amount on EITHER pool switch both pools to the backup backend. Otherwise just use the default pool. Here is an example verified on 1.5.18 but should work fine on newer versions:
defaults all
timeout connect 30s
timeout client 30s
timeout server 30s
mode http
# http frontend
frontend http *:80
# use the backup service if EITHER service is down
acl use_backup nbsrv(http_service) lt 1
acl use_backup nbsrv(tcp_service) lt 1
use_backend http_service_backup if use_backup
default_backend http_service
# tcp frontend
frontend tcp_10000 *:10000
mode tcp
# use the backup service if EITHER service is down
acl use_backup nbsrv(http_service) lt 1
acl use_backup nbsrv(tcp_service) lt 1
use_backend tcp_service_backup if use_backup
default_backend tcp_service
backend tcp_service
mode tcp
# main tcp instance here
# can also include backup server here with backup directive if desired
server tcp-service1 tcp-service1.local:10000 check
backend tcp_service_backup
mode tcp
# backup tcp instance here
server tcp-service2 tcp-service2.local:10000 check
backend http_service
# main http instance here
# can also include backup server here with backup directive if desired
server http-service1 http-service1.local:80 check
backend http_service_backup
# backup http instance here
server http-service2 http-service2.local:80 check
See https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#nbsrv for more nbsrv details.