HAProxy apparently not forwarding correctly - reverse-proxy

I'm using this HAProxy config file:
defaults
mode http
frontend main
bind *:80
use_backend drewgrosscom if { hdr(host) -i drewgross.com }
use_backend drewgrosscom if { hdr(host) -i www.drewgross.com }
use_backend drewgrosscom if { hdr(host) -i drewallyngross.com }
use_backend drewgrosscom if { hdr(host) -i www.drewallyngross.com }
use_backend idoor if { hdr(host) -i idoor.drewgross.com }
backend drewgrosscom
server s1 127.0.0.1:8000
backend idoor
server s3 127.0.0.1:8002
All of the things that forward to the drewgrosscom backend work fine, but visiting idoor.drewgross.com gives me nothing. The server is definitely running and serving on port 8002, because I can visit either drewgross.com:8002 or <my-servers-ip>:8002 and see the thing I expect to see. What am I missing here?

By default, HAProxy works in tunnel mode. That means, it will inspect only the first request in a connection, no matter if it's a keep-alive connection or not. After the request has been dispatched, all further communication of this specific TCP connection will then be send to the selected backend server of the first request.
To mitigate this, you could either completely disable keep-alive
option httpclose
or enable keep-alive between the client and HAProxy but to disable it between HAProxy and the backend server
option http-server-close
A third (incomplete and still very much work-in-progress) option available in HAProxy 1.5-dev20 or newer is full keep-alive support with
option http-keep-alive
You have to use one of these options to analyze (and switch) all HTTP requests. For a production site, I'd currently recommend option http-server-close.

Related

metallb round robin not working when accessed from external HAProxy

I have a sample app running in a kubernetes cluster with 3 replicas. I am exposing the app with type=LoadBalancer using metallb.
The external ip issued is 10.10.10.11
When I run curl 10.10.10.11 I get a different pod responding for each request as you would expect from round robin. This is the behaviour I want.
I have now setup HAProxy with a backend pointing to 10.10.10.11, however each time I access the HAProxy frontend, I get the same node responding to each request. If I keep refreshing I intermittently get different pods, sometimes after 20 refreshes, sometimes after 50+ refreshes. I have tried clearing my browser history, but that has no effect.
I assume it is my HAProxy config which is the cause the problem, perhaps caching? but I have not configured any caching. I am a HAProxy newbie, so I might be missing something.
Here is my HAProxy config.
I have tried both mode tcp and mode http, but both give the same result (the same pod responding to each request)
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /home/simon/haproxy/haproxy_certs
crt-base /home/simon/haproxy/haproxy_certs
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend https
bind *:443 ssl crt /home/simon/haproxy/haproxy_certs
timeout client 60s
mode tcp
#Hello App
acl ACL_hello_app hdr(host) -i hello.xxxxxxxxxdomain2.com
use_backend hello_app if ACL_hello_app
#Nginx App
acl ACL_nginx_app hdr(host) -i nginx.xxxxxxxxxdomain1.com
use_backend nginx_app if ACL_nginx_app
backend hello_app
timeout connect 10s
timeout server 100s
mode tcp
server hello_app 10.10.10.11:80
backend nginx_app
mode tcp
server nginx_app 10.10.10.10:80
UPDATE
Upon further testing the the issue seems to be related to the timeout client, timeout connect, timeout server. I reduce these to 1 second, then I get a different POD every 1 second, however these times are so short, I also get intermittent connection failures.
So, I also have the question. Is HAProxy able to work as a reverse proxy in front of another load balancer, or do I need to use another technology such as Nginx?
I eventually found the answer. I needed to use option http-server-close in my frontend settings.
frontend https
bind *:443 ssl crt /home/simon/haproxy/haproxy_certs
http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"
timeout client 5000s
option http-server-close
mode http
#Hello App
acl ACL_hello_app hdr(host) -i hello.soxprox.com
use_backend hello_app if ACL_hello_app
#Nginx App
acl ACL_nginx_app hdr(host) -i nginx.soxprox.com
use_backend nginx_app if ACL_nginx_app
With these settings I get correct round robin results from metallb

Can't authenticate for NiFi/NiFi Registry instance sitting behind HAProxy server

I have a NiFi and NiFi Registry instance sitting behind a HAProxy server. The NiFi instances are both secured using SSL. I am not able to pass the SSL information from the proxy server to the NiFi server. I also tried SSL Passthrough but that has some other limitations further on the line.
My current HAProxy config looks like this:
frontend https_in
bind *:443 ssl crt /etc/ssl/nifi/nifi-server.pem verify required ca-file /etc/ssl/nifi/nifi-ca.cert
mode http
option httplog
option http-server-close
acl is_registry path_beg /nifi-registry
use_backend nifi-registry if is_registry
default_backend nifi
backend nifi-registry
mode http
balance roundrobin
http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-ProxyScheme https
http-request set-header X-ProxyHost xx.xxx.xxx.xx
http-request set-header X-ProxyPort 443
server registry01 172.xx.xx.xxx:18443 check ssl verify none
When I browse to https://xx.xxx.xxx.xx:443/nifi-registry and select the client certificate I get the NiFi Registry UI but not logged in with my client user. I am not able to pass my SSL information to the NiFi servers. Following the documentation I've set some headers but they don't seem to have effect.
Am I missing something here ?
EDIT
So as suggested in the comments and mentioned above I also tried SSL Passthrough in tlc mode. With this I manage to pass the SSL authentication to the NiFi servers but I get in trouble with "invalid host header" message.
My HAProxy config:
frontend http_in
bind *:80 v4v6
mode http
redirect scheme https if !{ ssl_fc }
frontend nifi_registry_in
bind *:1443 v4v6
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nifi_registry
frontend nifi_in
bind *:2443 v4v6
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nifi
backend nifi_registry
mode tcp
balance roundrobin
server registry01 xxx.xx.xx.xxx:18443 check
backend nifi
mode tcp
balance roundrobin
server nifi01 xxx.xx.xx.xxx:9443 check
When browsing to the [public-haproxy-server-domain]:1443 I get the registry ui after authentication.
When browsing to the [public-haproxy-server-domain]:2443 I get the following error after authentication.
My NiFi config is
nifi.web.https.host=xxx.xx.xx.xxx
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=[public-haproxy-server-domain]:2443
So I've figured it out and I feel really stupid on this one. On the other hand I don't find this well documented. So I will post my answer here for other people in need.
Apparently for the nifi.web.proxy.host setting to work, the nifi.web.proxy.context.path setting should also be set. Simply putting a '/' as a value for this and everything works as expected.
nifi.web.proxy.context.path=/
nifi.web.proxy.host=[public-haproxy-server-domain]:2443

HTTPS URL Redirecting not working with HaProxy

I have two domain that each support HTTP and HTTPS - I want to redirect them to the proper ports / servers. The following config works for HTTP but not HTTPS
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
# daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
acl ui hdr(host) -i app.XYZ.com
acl api hdr(host) -i api.XYZ.com
use_backend apiserverHTTP if api
use_backend uiserverHTTP if ui
frontend https-in
bind *:443
acl ui hdr(host) -i app.XYZ.com
acl api hdr(host) -i api.XYZ.com
use_backend apiserverHTTPS if api
use_backend uiserverHTTPS if ui
backend apiserverHTTP
option tcplog
balance leastconn
server apiserver-2 10.132.62.240:31000 check
server apiserver-1 10.132.62.242:31000 check
backend apiserverHTTPS
option tcplog
balance leastconn
server apiserverS-2 10.132.62.240:31001 check
server apiserverS-1 10.132.62.242:31001 check
backend uiserverHTTP
option tcplog
balance leastconn
server uiserver-1 10.132.62.240:31002 check
backend uiserverHTTPS
option tcplog
balance leastconn
server uiserverS-1 10.132.62.240:31003 check
HTTP redirect without a problem - HTTPS does no redirection of any sort. Just fails or falls to the default if set.
Try this:
frontend http-in
bind *:80
bind *:443
acl ui hdr(host) -i app.XYZ.com
acl api hdr(host) -i api.XYZ.com
acl http_80 dst_port 80
acl https_443 dst_port 443
use_backend apiserverHTTP if api http_80
use_backend uiserverHTTP if ui http_80
use_backend apiserverHTTPS if api https_443
use_backend uiserverHTTPS if ui https_443
backend apiserverHTTP
option tcplog
balance leastconn
server apiserver-2 10.132.62.240:31000 check
server apiserver-1 10.132.62.242:31000 check
backend apiserverHTTPS
option tcplog
balance leastconn
server apiserverS-2 10.132.62.240:31001 check
server apiserverS-1 10.132.62.242:31001 check
backend uiserverHTTP
option tcplog
balance leastconn
server uiserver-1 10.132.62.240:31002 check
backend uiserverHTTPS
option tcplog
balance leastconn
server uiserverS-1 10.132.62.240:31003 check

HAProxy wildcard SSL backend forward issue

I am using HAProxy 1.5-dev21. I have purchased a wildcard SSL (for example: *.foo.com).
I want all traffic from Internet port 443 will redirect to internal network according the domain name, backend servers are many web server running HTTP (for example: abc.foo.com:443 -> 192.168.10.10:80 , edf.foo.com:443 -> 192.168.10.11:80)
However, whatever the incoming domain name, HAProxy passed all traffic to default backend.
My config is working well if I not using SSL
The following is my simplified config file:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
mode http
option tcplog
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend https-in
mode tcp
bind :443 ssl crt /etc/haproxy/foo.com.pem
use_backend abc if { hdr_end(host) -i abc.foo.com }
use_backend def if { hdr_end(host) -i def.foo.com }
default_backend application-backend
backend abc
mode tcp
server Server1 192.168.10.10:80
backend def
mode tcp
server Server2 192.168.10.11:80
backend application-backend
mode tcp
server server3 192.168.10.12:80
You're using tcp mode while trying to access HTTP content.
Please turn on 'mode http' and it should work.
Baptiste
When you have of SSL, you can't use hdr_end. Here is how i do it:
frontend domain.com
bind 10.50.81.131:443 ssl crt domain.com ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP
bind 10.50.81.131:80
mode http
maxconn 300
option httpclose
option forwardfor
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
use_backend first_farm if { ssl_fc_sni sub1.domain.com }
use_backend second_farm if { ssl_fc_sni sub2.domain.com }
default_backend default_farm

SSL over HAProxy issue

I am trying to put an HTTPS web server behind HAProxy load balancer.
HTTPS server runs on Node.js on port 8000, had all the correct SSL certs and keys and works if I try access it directly via URL, e.g. https://ssltest.mydomain.com:8000/. I can also check SSL connection using openssl s_client -connect ssltest.mydomain.com:8000 and it shows data that seems correct.
The front end for this HTTPS server is another subdomain, e.g. app.mydomain.com and it runs on default SSL port 443.
But when I try to access https://app.mydomain.com from my browser, I get an SSL error. The above command line check also fails: openssl s_client -connect app.mydomain.com:443 -state -debug
Below is how my HA Proxy config looks like:
global
log /dev/log local0 info
log /dev/log local0 notice
maxconn 20000
user haproxy
group haproxy
daemon
spread-checks 5
defaults
log global
mode tcp
option dontlognull
retries 3
option redispatch
timeout connect 30s
timeout client 30s
timeout server 30s
timeout check 5s
balance roundrobin
frontend https_proxy
bind :443
acl is_app hdr_dom(host) -i app.mydomain.com
acl is_rest hdr_dom(host) -i rest.mydomain.com
acl is_io hdr_dom(host) -i io.mydomain.com
use_backend core_https if is_app
use_backend core_rest_api if is_rest
use_backend core_www_io if is_io
backend core_https
server www1 10.0.0.174:8000 maxconn 25 check inter 5s rise 18 fall 2
backend core_www_io
server io1 10.0.0.174:8100 maxconn 25 check inter 5s rise 18 fall 2
backend core_rest_api
server api1 10.0.0.174:8200 maxconn 25 check inter 5s rise 18 fall 2
I can't figure out why this does not work via HAProxy but works directly, and I would REALLY appreciate your suggestions.
You're confusing layer 4 and layer 7 load balancing. To separate requests using hdr_dom you need layer 7 that's only available for HTTP and as you may guess HTTPS works on layer 4.
So as haproxy can't inspect the host, none of your ifs are returning true and there is no backend selected, to fix you should add a default_backend entry.
frontend https_proxy
bind :443
acl is_app hdr_dom(host) -i app.mydomain.com
acl is_rest hdr_dom(host) -i rest.mydomain.com
acl is_io hdr_dom(host) -i io.mydomain.com
use_backend core_https if is_app
use_backend core_rest_api if is_rest
use_backend core_www_io if is_io
default_backend core_https