HAPROXY reqirep on Host header not forwarding - reverse-proxy

I have a web server (actually it's a CF environment, but that doesn't matter much) running behind a haproxy version 1.5-dev19 2013/06/17 that accepts requests only for a certain internal domain, let's call it: internal-address.
Meaning, that HTTP requests only work if the host header is suffixed by internal-address, (like: Host: login.internal-address).
Users from the WAN can access this web server by connecting to an external address which has ip forwarding to the internal server.
But when a user accesses the external address, the Host header is suffixed with external-address, and the web server behind the haproxy rejects the request.
I added reqirep entries in to haproxy configuration:
global
log 127.0.0.1 syslog info
daemon
user vcap
group vcap
maxconn 64000
spread-checks 4
defaults
log global
timeout connect 30000ms
timeout client 300000ms
timeout server 300000ms
frontend http-in
mode http
bind :80
option httplog
option forwardfor
reqadd X-Forwarded-Proto:\ http
default_backend http-routers
frontend https-in
mode http
bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
option httplog
option forwardfor
option http-server-close
reqadd X-Forwarded-Proto:\ https
default_backend http-routers
frontend ssl-in
mode tcp
bind :4443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
default_backend tcp-routers
backend http-routers
mode http
balance roundrobin
reqirep ^Host:\ uaa.external-address Host:\ uaa.internal-address
reqirep ^Host:\ api.external-address Host:\ api.internal-address
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
server node0 172.20.0.1:8888 check inter 1000
backend tcp-routers
mode tcp
balance roundrobin
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
server node0 172.20.0.1:8888 check inter 1000
And every request sent to uaa.external-address or api.external-address is indeed changed and the web server behind the haproxy receives the request as if the Host header is suffixed with internal-address.
But the 3rd rule:
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
Doesn't work :(
The web server's access log shows the Host header is sent from external-address:4443 , which implies that the haproxy didn't match the Host header correctly and then the web server rejects the request.
The request issued by the client is:
WEBSOCKET REQUEST: [2014-10-01T10:25:07+03:00]
GET /tail/?app=029a1269-67fe-46e2-85f7-e1b0b5d34193 HTTP/1.1
Host: wss://external-address:4443
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: [HIDDEN]
Origin: http://localhost
Authorization: [PRIVATE DATA HIDDEN]
Does anyone know what's wrong with the rule?
EDIT:
I deleted the rules from the backend and created more generic rules in the frontend, and it still doesn't work for websockets:
frontend https-in
mode http
bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
option httplog
option forwardfor
option http-server-close
reqadd X-Forwarded-Proto:\ https
default_backend http-routers
reqirep ^Host:\ (.*).external-address(.*) Host:\ \1.internal-address\2
frontend ssl-in
mode tcp
bind :4443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
default_backend tcp-routers
reqirep ^Host:\ (.*).external-address(.*) Host:\ \1.internal-address\2
Thanks in advance.

Which version of HAProxy are you running?
If 1.4, please add 'option http-server-close' into your defaults section.
By default, 1.4 is in tunnel mode, which let HAProxy analyse the first request and response and transfer subsequent requests and responses as payload.
In 1.5, it should work out of the box. HAProxy uses a new mode "http-keep-alive" which allows HAProxy to analyse all the content all the time.
Baptiste

Related

Can't authenticate for NiFi/NiFi Registry instance sitting behind HAProxy server

I have a NiFi and NiFi Registry instance sitting behind a HAProxy server. The NiFi instances are both secured using SSL. I am not able to pass the SSL information from the proxy server to the NiFi server. I also tried SSL Passthrough but that has some other limitations further on the line.
My current HAProxy config looks like this:
frontend https_in
bind *:443 ssl crt /etc/ssl/nifi/nifi-server.pem verify required ca-file /etc/ssl/nifi/nifi-ca.cert
mode http
option httplog
option http-server-close
acl is_registry path_beg /nifi-registry
use_backend nifi-registry if is_registry
default_backend nifi
backend nifi-registry
mode http
balance roundrobin
http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-ProxyScheme https
http-request set-header X-ProxyHost xx.xxx.xxx.xx
http-request set-header X-ProxyPort 443
server registry01 172.xx.xx.xxx:18443 check ssl verify none
When I browse to https://xx.xxx.xxx.xx:443/nifi-registry and select the client certificate I get the NiFi Registry UI but not logged in with my client user. I am not able to pass my SSL information to the NiFi servers. Following the documentation I've set some headers but they don't seem to have effect.
Am I missing something here ?
EDIT
So as suggested in the comments and mentioned above I also tried SSL Passthrough in tlc mode. With this I manage to pass the SSL authentication to the NiFi servers but I get in trouble with "invalid host header" message.
My HAProxy config:
frontend http_in
bind *:80 v4v6
mode http
redirect scheme https if !{ ssl_fc }
frontend nifi_registry_in
bind *:1443 v4v6
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nifi_registry
frontend nifi_in
bind *:2443 v4v6
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
default_backend nifi
backend nifi_registry
mode tcp
balance roundrobin
server registry01 xxx.xx.xx.xxx:18443 check
backend nifi
mode tcp
balance roundrobin
server nifi01 xxx.xx.xx.xxx:9443 check
When browsing to the [public-haproxy-server-domain]:1443 I get the registry ui after authentication.
When browsing to the [public-haproxy-server-domain]:2443 I get the following error after authentication.
My NiFi config is
nifi.web.https.host=xxx.xx.xx.xxx
nifi.web.https.port=9443
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=[public-haproxy-server-domain]:2443
So I've figured it out and I feel really stupid on this one. On the other hand I don't find this well documented. So I will post my answer here for other people in need.
Apparently for the nifi.web.proxy.host setting to work, the nifi.web.proxy.context.path setting should also be set. Simply putting a '/' as a value for this and everything works as expected.
nifi.web.proxy.context.path=/
nifi.web.proxy.host=[public-haproxy-server-domain]:2443

HAProxy http check on for ssl?

I have some web servers which are MySQL backend. An HAProxy is in front of those web servers. All the web servers are using https.
I tried to use the http check option on both http and https to make sure if the database connection was lost, the HAProxy will failover to another node. My haproxy configuration file:
global
log /dev/log local0
maxconn 4096
#debug
#quiet
user haproxy
group haproxy
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen http
bind *:80
mode http
balance roundrobin
stats enable
stats auth haproxy:haproxy
cookie JSESSIONID prefix
stick on src table https
option http-server-close
option forwardfor
default-server inter 5s fall 2
option httpchk
http-check expect ! string Database\ error
server sitea 192.168.0.20 cookie sitea check port 80
server siteb 192.168.0.21 cookie siteb check port 80
listen https
bind *:443
mode tcp
balance roundrobin
stick-table type ip size 5000k expire 2h store conn_cur
stick on src
option tcplog
option ssl-hello-chk
default-server inter 5s fall 2
option httpchk
http-check expect ! string Database\ error
server sitea 192.168.0.20:443 check ssl verify none
server siteb 192.168.0.21:443 check ssl verify none
Look at the last two lines. If I specified "ssl verify none", my HAProxy can successfully check both Apache and MySQL status. However, I can't open the webpage via https(it prompts me This site can’t provide a secure connection. ERR_SSL_PROTOCOL_ERROR).
If I remove that parameter, the webpage can be opened again, but all the https servers status become DOWN in the HAProxy.
P.S. I'm using self-signed certificate currently, because I'm still on testing.
I have found the solution: since I am using https on apache nodes, I have to copy ssl certificates content to haproxy. To do that, copy and merge both private key and the certificate content issued by the CA into one single file(In my case, I put it into /etc/haproxy/haproxy.pem).
Modify the haproxy configuration, change
bind *:443
To
bind *:443 ssl crt /etc/haproxy/haproxy.pem

HAproxy times out(504 error) with large POST bodies

I have 3 nodejs web-servers spun on an ubuntu box and HAproxy to load-balance those servers on the same box. HAproxy listens at port 80(http) and 443(https, with SSL termination). There's no SSL between the HAproxy server and the web-servers.
The POST call to one of the api without SSL, passes through with any value of content-length, but when I try to do the POST call with a content-length greater than 8055 on the HAproxy with SSL connection(port443), HAproxy times out giving a 504 Gateway Timeout error.
Also, if I give an "Expect:100 continue" header to the curl command, the server responds with some delay, which I don't want to exist. Below is how the HAproxy config file looks like:
global
stats socket /var/run/haproxy.sock mode 0777
log 127.0.0.1 local0 info
log 127.0.0.1 local1 info
chroot /usr/share/haproxy
uid nobody
gid nobody
nbproc 1
daemon
maxconn 50000
frontend localnodes:https
bind *:443 ssl crt /etc/ssl/private/443_private_ssl_in.pem no-sslv3
mode http
reqadd X-Forwarded-Proto:\ https
default_backend nodes
timeout client 30000
frontend localnodes-http
bind *:80
mode http
reqadd X-Forwarded-Proto:\ http
default_backend nodes
timeout client 30000
backend nodes
mode http
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
log global
timeout connect 3000
timeout server 30000
option httplog
option ssl-hello-chk
option httpchk GET /
http-check expect status 404
server nodejsweb01 127.0.0.1:8000 check
server nodejsweb02 127.0.0.1:8001 check
server nodejsweb03 127.0.0.1:8002 check
I have ensured that the nodejs web-servers behind have no problem, they work fine. I have tried increasing the 'timeout server' period, no effect.
Also tried a solution on this link that tells to give an option ssl ca-file to the backend nodes, as follows:
server nodejsweb01 127.0.0.1:8000 ssl ca-file /etc/ssl/certs/ca.pem check
server nodejsweb02 127.0.0.1:8001 ssl ca-file /etc/ssl/certs/ca.pem check
server nodejsweb03 127.0.0.1:8002 ssl ca-file /etc/ssl/certs/ca.pem check
but after this option HAproxy throws an error saying no servers available at the backend.
Please tell me what am I doing wrong in HAproxy conf file, so that I make the webservers respond successfully with the SSL connection
Try this minimum config:
frontend localnodes
bind *:80
bind *:443 ssl crt /etc/ssl/private/443_private_ssl_in.pem
mode http
default_backend nodes
backend nodes
mode http
option forwardfor
option httplog
server nodejsweb01 127.0.0.1:8000 check
server nodejsweb02 127.0.0.1:8001 check
server nodejsweb03 127.0.0.1:8002 check
I suspect it's those additional options.
It could also be the SSL cert file. Is your PEM file created from a self-signed cert? How is it structured?

HAProxy wildcard SSL backend forward issue

I am using HAProxy 1.5-dev21. I have purchased a wildcard SSL (for example: *.foo.com).
I want all traffic from Internet port 443 will redirect to internal network according the domain name, backend servers are many web server running HTTP (for example: abc.foo.com:443 -> 192.168.10.10:80 , edf.foo.com:443 -> 192.168.10.11:80)
However, whatever the incoming domain name, HAProxy passed all traffic to default backend.
My config is working well if I not using SSL
The following is my simplified config file:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
defaults
log global
mode http
option tcplog
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend https-in
mode tcp
bind :443 ssl crt /etc/haproxy/foo.com.pem
use_backend abc if { hdr_end(host) -i abc.foo.com }
use_backend def if { hdr_end(host) -i def.foo.com }
default_backend application-backend
backend abc
mode tcp
server Server1 192.168.10.10:80
backend def
mode tcp
server Server2 192.168.10.11:80
backend application-backend
mode tcp
server server3 192.168.10.12:80
You're using tcp mode while trying to access HTTP content.
Please turn on 'mode http' and it should work.
Baptiste
When you have of SSL, you can't use hdr_end. Here is how i do it:
frontend domain.com
bind 10.50.81.131:443 ssl crt domain.com ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP
bind 10.50.81.131:80
mode http
maxconn 300
option httpclose
option forwardfor
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
use_backend first_farm if { ssl_fc_sni sub1.domain.com }
use_backend second_farm if { ssl_fc_sni sub2.domain.com }
default_backend default_farm

Unable to start domain with https

I am working on haproxy. I want to make my site open with http. I have purchased ssl certificate and install on the server.
In ha.cfg I have configured as follow :
global
tune.bufsize 32786
tune.maxrewrite 16384
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 8192
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
balance roundrobin
stats enable
stats refresh
stats uri /ssproxy_stats
stats realm Haproxy\ Statistics
stats auth haproxy:haproxy
maxconn 4000
contimeout 5000
clitimeout 50000
srvtimeout 50000
frontend http
bind *:80
acl hari path_beg /customers
acl css path_beg /assets
reqadd X-Forwarded-Proto:\ http
use_backend appointpress_app if hari
use_backend appointpress_app if css
default_backend appointpress_site
frontend https
bind *:443 ssl crt /etc/ssl/ssl.key/mydomain.crt
default_backend appointpress_site
backend appointpress_app :80
stats enable
stats auth haproxy:haproxy
cookie SERVERID insert
option httpclose
option forwardfor
server app_server ec2-elastic-domain:80 cookie haproxy_app check
backend appointpress_site :80
stats enable
stats auth haproxy:haproxy
cookie SERVERID insert
option httpclose
option forwardfor
server wordpress someip:443 cookie haproxy_site check
After running the command haproxy -f ha.cfg I am getting no error,
and when I am running url http://ec2-instance, its working fine
but while running https://ec2-instance,
I am getting error :
in chrome : Error code: ERR_SSL_PROTOCOL_ERROR
in firefox : Error code: ssl_error_rx_record_too_long
How to resolve the error
Check to make sure that your EC2 security rules allow port 443 to your running instance. A simple way to test this is to use telnet from your client:
telnet ec2-instance 443