HAProxy - Add response header to indicate the server that was chosen - header

Consider the following HAProxy Config:
frontend front
default_backend default
backend default
balance roundrobin
http-response set-header X-RGN us-east-1
server app-1a app.us-east-1a.example.com:443 ssl verify none check
server app-1c app.us-east-1c.example.com:443 ssl verify none check
server app-1b app.us-east-1b.example.com:443 ssl verify none check
I would like to return a response header the indicates the server that was chosen. For example, if the frontend receives a request, it will balance roundrobin and forward the request to a backend server, when it responds, I would like to see in my browser which server was used.
The config might look something like this:
frontend front
default_backend default
backend default
balance roundrobin
http-response set-header X-RGN us-east-1
server app-1a app.us-east-1a.example.com:443 ssl verify none check
server app-1c app.us-east-1c.example.com:443 ssl verify none check
server app-1b app.us-east-1b.example.com:443 ssl verify none check
http-response set-header X-Server app-1a if server -i app-1a
http-response set-header X-Server app-1b if server -i app-1b
http-response set-header X-Server app-1c if server -i app-1c
Has anyone tried this before?

Assuming HAProxy 1.5 or later:
http-response set-header X-Server %s

Related

Can't authenticate for NiFi/NiFi Registry instance sitting behind HAProxy server

I have a NiFi and NiFi Registry instance sitting behind a HAProxy server. The NiFi instances are both secured using SSL. I am not able to pass the SSL information from the proxy server to the NiFi server. I also tried SSL Passthrough but that has some other limitations further on the line.
My current HAProxy config looks like this:
frontend https_in
bind *:443 ssl crt /etc/ssl/nifi/nifi-server.pem verify required ca-file /etc/ssl/nifi/nifi-ca.cert
mode http
option httplog
option http-server-close
acl is_registry path_beg /nifi-registry
use_backend nifi-registry if is_registry
default_backend nifi
backend nifi-registry
mode http
balance roundrobin
http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-ProxyScheme https
http-request set-header X-ProxyHost xx.xxx.xxx.xx
http-request set-header X-ProxyPort 443
server registry01 172.xx.xx.xxx:18443 check ssl verify none
When I browse to https://xx.xxx.xxx.xx:443/nifi-registry and select the client certificate I get the NiFi Registry UI but not logged in with my client user. I am not able to pass my SSL information to the NiFi servers. Following the documentation I've set some headers but they don't seem to have effect.
Am I missing something here ?
EDIT
So as suggested in the comments and mentioned above I also tried SSL Passthrough in tlc mode. With this I manage to pass the SSL authentication to the NiFi servers but I get in trouble with "invalid host header" message.
My HAProxy config:
frontend http_in
bind *:80 v4v6
mode http
redirect scheme https if !{ ssl_fc }
frontend nifi_registry_in
bind *:1443 v4v6
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nifi_registry
frontend nifi_in
bind *:2443 v4v6
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nifi
backend nifi_registry
mode tcp
balance roundrobin
server registry01 xxx.xx.xx.xxx:18443 check
backend nifi
mode tcp
balance roundrobin
server nifi01 xxx.xx.xx.xxx:9443 check
When browsing to the [public-haproxy-server-domain]:1443 I get the registry ui after authentication.
When browsing to the [public-haproxy-server-domain]:2443 I get the following error after authentication.
My NiFi config is
nifi.web.https.host=xxx.xx.xx.xxx
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=[public-haproxy-server-domain]:2443
So I've figured it out and I feel really stupid on this one. On the other hand I don't find this well documented. So I will post my answer here for other people in need.
Apparently for the nifi.web.proxy.host setting to work, the nifi.web.proxy.context.path setting should also be set. Simply putting a '/' as a value for this and everything works as expected.
nifi.web.proxy.context.path=/
nifi.web.proxy.host=[public-haproxy-server-domain]:2443

Is there a way to use a forwarding proxy as a backend (including authentication) in HAProxy?

I am quite new to HAProxy and want to achieve the following setup / packet flow:
Client -> HAProxy (as reverse proxy) -> Forwarding Proxy (HAProxy, IIS, Squid...) -> Internet -> server.example.com
I would like to have encrypted connections with TLS/SSL from the Client -> HAProxy and from HAProxy -> server.example.com
This means, that the forwarding proxy needs to support the HTTP CONNECT method, to establish a TCP tunnel and transmits packets without trying to interpret them. Over this TCP port I should be able to send bytes to server.example.com - TLS/SSL encrypted, so HTTPS.
Furthermore it could be, that authentication against the forwarding proxy is needed e.g. HTTP Basic authentication.
The software stack of my test setup is as follows:
Client Firefox 76.0.1
HA-Proxy version 1.6.3 2015/12/25
Squid Cache: Version 3.5.27
I have setup this HAProxy configuration:
global
# Standard settings
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
# Tuning
maxconn 2000
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-options no-sslv3
# Ensure a secure enough DH paramset
tune.ssl.default-dh-param 2048
defaults
log global
mode http
option httplog
option dontlognull
# Redispatch 503 errors on backends, reducing the number of 503 reaching clients
option redispatch
# We want to stall clients for as long as possible, before giving
# up with 503:
timeout connect 5m
# Clients must be acceptably responsive
timeout client 1m
# Server not as much...
timeout server 5m
# HTTPS server
frontend https-in
bind :443 ssl crt-ignore-err all crt /etc/haproxy/ssl/certkey.pem
# Don't serve HTTP directly, but redirect to same URL in https
redirect scheme https code 301 if !{ ssl_fc }
default_backend backend-proxy
backend backend-proxy
# Create the Authorization / Proxy-Authorization header value
# echo -n "user:password" | base64
http-request add-header Proxy-Authorization "Basic dXNlcjpwYXNzd29yZA=="
# We need to use the CONNECT method
http-request set-method CONNECT
# The proxyserver needs to know the full server name
http-request set-path server.example.com:443
server proxy 192.168.1.1:8080
In my test setup I use a Squid server as a forwarding proxy with the following configuration:
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 443 # https
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager
http_access allow localhost
auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/squid-passwd
acl basic_client proxy_auth REQUIRED
http_access deny !basic_client
http_access allow basic_client
http_access deny all
http_port 8080
coredump_dir /var/spool/squid
Using the Squid forwarding proxy in a regular browser on the same subnet including authentication works fine.
So when the first request comes in from the Client at the HAProxy it gets forwarded via the backend backend-proxy to the Forward Proxy (Squid). The CONNECT succeeds, as I see in the Squid log.
1591166403.966 60146 192.168.1.10 TCP_TUNNEL/200 39 CONNECT server.example.com:443 test HIER_DIRECT/6.7.8.9 -
(IP addresses were replaced with generic values)
The HAProxy log shows, that the correct backend is used:
Jun 3 08:39:57 localhost haproxy[3547]: 192.168.1.20:39398 [03/Jun/2020:08:38:56.855] https-in~ backend-proxy/proxy 154/0/13/209/60375 200 39 - - ---- 0/0/0/0/0 0/0 "GET /someurl HTTP/1.1"
(IP addresses were replaced with generic values)
So far so good. But I am unable to establish a successful communication to server.example.com from my Client. I think I have to use a second/other backend, which will not mangle the requests any more (exchange method and path) but instead use the given TCP port from the forwarding proxy to transmit the request.
How can I save the 'state' of the communication to my backend / proxy server in HAProxy, so the request could be resend to another backend?
How to extract and use the TCP port from the response of the Forwarding Proxy?
Is there a way to check, if the TCP tunnel on the Forwarding Proxy is still opened or do I need to request it using CONNECT every time before I want to use it?
EDIT:
I solved the situation by using stunnel as an intermediary to handle the TCP tunnel creation with CONNECT against the Forwarding Proxy.
If you have an upstream HTTP proxy (like squid) (not a socks proxy) and you want to have haproxy accept connections and open the tunnel thru the upstream proxy (such that the haproxy clients do not support doing so themselves via http/CONNECT method) on behalf of the clients, then this functionality does not exist in haproxy today.
I crated a branch that does this via server keyword proxy-tunnel.
The below example config will behave such that clients of haproxy that connect to port 20025 (haproxy) will result in haproxy establishing a http/CONNECT tunnel via the upstream proxy 172.16.0.99:50443 to 172.16.0.2:2023:
listen SMTP-20025
bind 0.0.0.0:20025
server TEST_SERVERVIA_PROXY 172.16.0.2:2023 proxy-tunnel 172.16.0.99:50443

HaProxy - Direct to backend server according to query string

I am using HaProxy to load balance between my 2 servers. My mobile app connects to one of this 2 servers directed by the loadbalancer. My apps mainly makes this http request to loadbalancer:
http://www.example.com/file.php?action=getdata
and this is being redirected to one of these 2 servers and there is no problem with this but I need this requests to direct to server2 always because these specific actions are stored on the database of server2.
http://www.example.com/file.php?action=gettoken OR
http://www.example.com/file.php?action=savetoken
How can I achieve this with haproxy's config file ? Currently it looks like:
frontend haproxynode
bind *:80
mode http
default_backend backendnodes
backend backendnodes
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
server node1 ip:80 check
server node2 ip:80 check
use server node2 if { url_param(action) -m str gettoken } || { url_param(action) -m str savetoken }
http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#7.3.6-url_param
Using url_sub is a little bit sloppy, since it examines the entire query string.

HAproxy times out(504 error) with large POST bodies

I have 3 nodejs web-servers spun on an ubuntu box and HAproxy to load-balance those servers on the same box. HAproxy listens at port 80(http) and 443(https, with SSL termination). There's no SSL between the HAproxy server and the web-servers.
The POST call to one of the api without SSL, passes through with any value of content-length, but when I try to do the POST call with a content-length greater than 8055 on the HAproxy with SSL connection(port443), HAproxy times out giving a 504 Gateway Timeout error.
Also, if I give an "Expect:100 continue" header to the curl command, the server responds with some delay, which I don't want to exist. Below is how the HAproxy config file looks like:
global
stats socket /var/run/haproxy.sock mode 0777
log 127.0.0.1 local0 info
log 127.0.0.1 local1 info
chroot /usr/share/haproxy
uid nobody
gid nobody
nbproc 1
daemon
maxconn 50000
frontend localnodes:https
bind *:443 ssl crt /etc/ssl/private/443_private_ssl_in.pem no-sslv3
mode http
reqadd X-Forwarded-Proto:\ https
default_backend nodes
timeout client 30000
frontend localnodes-http
bind *:80
mode http
reqadd X-Forwarded-Proto:\ http
default_backend nodes
timeout client 30000
backend nodes
mode http
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
log global
timeout connect 3000
timeout server 30000
option httplog
option ssl-hello-chk
option httpchk GET /
http-check expect status 404
server nodejsweb01 127.0.0.1:8000 check
server nodejsweb02 127.0.0.1:8001 check
server nodejsweb03 127.0.0.1:8002 check
I have ensured that the nodejs web-servers behind have no problem, they work fine. I have tried increasing the 'timeout server' period, no effect.
Also tried a solution on this link that tells to give an option ssl ca-file to the backend nodes, as follows:
server nodejsweb01 127.0.0.1:8000 ssl ca-file /etc/ssl/certs/ca.pem check
server nodejsweb02 127.0.0.1:8001 ssl ca-file /etc/ssl/certs/ca.pem check
server nodejsweb03 127.0.0.1:8002 ssl ca-file /etc/ssl/certs/ca.pem check
but after this option HAproxy throws an error saying no servers available at the backend.
Please tell me what am I doing wrong in HAproxy conf file, so that I make the webservers respond successfully with the SSL connection
Try this minimum config:
frontend localnodes
bind *:80
bind *:443 ssl crt /etc/ssl/private/443_private_ssl_in.pem
mode http
default_backend nodes
backend nodes
mode http
option forwardfor
option httplog
server nodejsweb01 127.0.0.1:8000 check
server nodejsweb02 127.0.0.1:8001 check
server nodejsweb03 127.0.0.1:8002 check
I suspect it's those additional options.
It could also be the SSL cert file. Is your PEM file created from a self-signed cert? How is it structured?

HAPROXY reqirep on Host header not forwarding

I have a web server (actually it's a CF environment, but that doesn't matter much) running behind a haproxy version 1.5-dev19 2013/06/17 that accepts requests only for a certain internal domain, let's call it: internal-address.
Meaning, that HTTP requests only work if the host header is suffixed by internal-address, (like: Host: login.internal-address).
Users from the WAN can access this web server by connecting to an external address which has ip forwarding to the internal server.
But when a user accesses the external address, the Host header is suffixed with external-address, and the web server behind the haproxy rejects the request.
I added reqirep entries in to haproxy configuration:
global
log 127.0.0.1 syslog info
daemon
user vcap
group vcap
maxconn 64000
spread-checks 4
defaults
log global
timeout connect 30000ms
timeout client 300000ms
timeout server 300000ms
frontend http-in
mode http
bind :80
option httplog
option forwardfor
reqadd X-Forwarded-Proto:\ http
default_backend http-routers
frontend https-in
mode http
bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
option httplog
option forwardfor
option http-server-close
reqadd X-Forwarded-Proto:\ https
default_backend http-routers
frontend ssl-in
mode tcp
bind :4443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
default_backend tcp-routers
backend http-routers
mode http
balance roundrobin
reqirep ^Host:\ uaa.external-address Host:\ uaa.internal-address
reqirep ^Host:\ api.external-address Host:\ api.internal-address
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
server node0 172.20.0.1:8888 check inter 1000
backend tcp-routers
mode tcp
balance roundrobin
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
server node0 172.20.0.1:8888 check inter 1000
And every request sent to uaa.external-address or api.external-address is indeed changed and the web server behind the haproxy receives the request as if the Host header is suffixed with internal-address.
But the 3rd rule:
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
Doesn't work :(
The web server's access log shows the Host header is sent from external-address:4443 , which implies that the haproxy didn't match the Host header correctly and then the web server rejects the request.
The request issued by the client is:
WEBSOCKET REQUEST: [2014-10-01T10:25:07+03:00]
GET /tail/?app=029a1269-67fe-46e2-85f7-e1b0b5d34193 HTTP/1.1
Host: wss://external-address:4443
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: [HIDDEN]
Origin: http://localhost
Authorization: [PRIVATE DATA HIDDEN]
Does anyone know what's wrong with the rule?
EDIT:
I deleted the rules from the backend and created more generic rules in the frontend, and it still doesn't work for websockets:
frontend https-in
mode http
bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
option httplog
option forwardfor
option http-server-close
reqadd X-Forwarded-Proto:\ https
default_backend http-routers
reqirep ^Host:\ (.*).external-address(.*) Host:\ \1.internal-address\2
frontend ssl-in
mode tcp
bind :4443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
default_backend tcp-routers
reqirep ^Host:\ (.*).external-address(.*) Host:\ \1.internal-address\2
Thanks in advance.
Which version of HAProxy are you running?
If 1.4, please add 'option http-server-close' into your defaults section.
By default, 1.4 is in tunnel mode, which let HAProxy analyse the first request and response and transfer subsequent requests and responses as payload.
In 1.5, it should work out of the box. HAProxy uses a new mode "http-keep-alive" which allows HAProxy to analyse all the content all the time.
Baptiste