Haproxy seemingly substitutes brotli with gzip in "Accept-Encoding" header - gzip

Im struggling to figure out why haproxy seemingly replaces br with gzip in "Accept-Encoding" header as request passes haproxy.
My application currently structured like this:
HAPROXY(tls termination) -> varnish -> apache
So I test like this:
curl -I --http2 -H 'Accept-Encoding: br' -I https://mysite.dev:31753?tru
So - sending single GET request to haproxy that strictly asks for brotly only (using curl)...
So that's what I would expect to see coming to varnish, but what is actually coming into varnish is these 2 requests:
HEAD request with br
GET request with gzip value instead...
I'm so confused - why are there 2 requests now? I did not configure compression in haproxy how can it be rewriting br to gzip.
Requests coming to varnish (I get this using tcpflow program):
172.030.000.035.41382-172.030.000.034.00080: HEAD /?tru HTTP/1.1
user-agent: curl/7.68.0
accept: */*
accept-encoding: br
host: mysite.dev:31753
x-client-ip: 192.168.10.103
x-forwarded-port: 31753
x-forwarded-proto: https
x-forwarded-for: 192.168.10.103
connection: close
172.030.000.034.41882-172.030.000.033.00080: GET /?tru HTTP/1.1
user-agent: curl/7.68.0
accept: */*
x-client-ip: 192.168.10.103
x-forwarded-port: 31753
x-forwarded-proto: https
X-Forwarded-For: 192.168.10.103, 172.30.0.35
host: mysite:31753
Accept-Encoding: gzip
X-Varnish: 328479
My haproxy config looks like so:
Haproxy
global
maxconn 1024
log stdout format raw local0
ssl-default-bind-options ssl-min-ver TLSv1.2
defaults
log global
option httplog
option http-server-close
mode http
option dontlognull
timeout connect 5s
timeout client 20s
timeout server 45s
frontend fe-wp-combined
mode tcp
bind *:31753
tcp-request inspect-delay 2s
tcp-request content accept if HTTP
tcp-request content accept if { req.ssl_hello_type 1 }
use_backend be-wp-recirc-http if HTTP
default_backend be-wp-recirc-https
backend be-wp-recirc-http
mode tcp
server loopback-for-http abns#wp-haproxy-http send-proxy-v2
backend be-wp-recirc-https
mode tcp
server loopback-for-https abns#wp-haproxy-https send-proxy-v2
frontend fe-wp-https
mode http
bind abns#wp-haproxy-https accept-proxy ssl crt /certs/fullkeychain.pem alpn h2,http/1.1
# whatever you need todo for HTTPS traffic
default_backend be-wp-real
frontend fe-wp-http
mode http
bind abns#wp-haproxy-http accept-proxy
# whatever you need todo for HTTP traffic
redirect scheme https code 301 if !{ ssl_fc }
backend be-wp-real
mode http
balance roundrobin
option forwardfor
# Send these request to check health
option httpchk
http-check send meth HEAD uri / ver HTTP/1.1 hdr Host haproxy.local
http-response del-header Server
http-response del-header via
server wp-backend1 proxy-varnish:80 check
http-request set-header x-client-ip %[src]
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
Please help if anyone knows what's happening here - extremely stumped.

nvm.upon further investigation it turned out that I was mixing up tcpflow results.
apparently it was varnish all along that was automatically encoding gzip, after I disabled it - I got br back as expected.

Related

Getting 404 when call request by haproxy (directly works fine)

I directly call a web service with url
curl http://venesh.ir/webservice/oauth/token
and I got error 403,
but when I call it by reverse proxy from some server I got 404,is it possible that haproxy change my address?
haproxy config:
frontend localhost
bind *:8081
option tcplog
mode tcp
acl isVenesh dst_port 8081
use_backend venesh if isVenesh
default_backend venesh
backend venesh
mode tcp
balance roundrobin
server web01 venesh.ir:80 check
when I call mySerevrIp:8081/webservice/oauth/token I expect getting the result that I directly call
curl http://venesh.ir/webservice/oauth/token that is 403,
but when I call curl mySerevrIp:8081/webservice/oauth/token I get error 404,
Is a problem with my haproxy or my config or is it possible that this problem is because of venesh.ir website?
It appears that http://venesh.ir/webservice/oauth/token expects the host header to be venesh.ir. You can test this from the command line. If the host header is not venesh.ir, it will return 404:
$ curl -I -H 'Host: 1.1.1.1' http://venesh.ir/webservice/oauth/token
HTTP/1.1 404 Not Found
Date: Mon, 24 Jun 2019 17:48:56 GMT
Server: Apache/2
Content-Type: text/html; charset=iso-8859-1
You can add the host header to your configuration if you change your mode to http:
frontend localhost
bind *:8081
option httplog
mode http
default_backend venesh
backend venesh
mode http
balance roundrobin
http-request set-header Host venesh.ir
server web01 venesh.ir:80 check
The answer of #mweiss was true, and an alternative way that I found is Setting HOST value to venesh.ir in my request header then the tcp reverse proxy works fine.

How to use HA Proxy server as Load Balancer

I am trying to use HA Proxy server as a load-balancer between my two web servers.
The Individual servers are up and running on the specified ports.
But I am not able to hit the web servers through HA Proxy server.
I have configured HA proxy using standard settings as below
/etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local0
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main *:80
mode http
default_backend app
backend app
balance roundrobin
option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
server app1 10.368.240.116:9091
server app2 10.368.240.317:9092
#server app3 127.0.0.1:5003 check
#server app4 127.0.0.1:5004 check
I can successfully curl the above servers from my HA Proxy host machine
curl -i "10.368.240.116:9091"
HTTP/1.1 200 OK
curl -i "10.368.240.317:9091"
HTTP/1.1 200 OK
My HA Proxy server is listening on 80 port
But when I hit it it is giving 503 error as below
curl -iv "127.0.0.1"
* About to connect() to 127.0.0.1 port 80 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
HTTP/1.0 503 Service Unavailable
< Cache-Control: no-cache
Cache-Control: no-cache
< Connection: close
Connection: close
< Content-Type: text/html
Content-Type: text/html
<
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
* Closing connection 0
Can anyone guide where am I doing wrong

Reverse proxy setup on haproxy gives 500; wget on base server & nginx works

I am trying to setup haproxy as a reverse proxy for a server. I am on Centos.
The config goes like this:
global
#log /dev/log local0
#log /dev/log local1 notice
log 127.0.0.1 local2 info
log 127.0.0.1 local2 notice
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend http_front
bind *:801
option forwardfor
stats enable
default_backend http_back
backend http_back
mode http
option httpchk
option forwardfor
http-send-name-header Host
balance roundrobin
server server1 stg-hostserv.com:80
But, if I do a wget against it, I am getting the below error.
# wget http://0.0.0.0:801
--2018-07-16 14:26:24-- http://0.0.0.0:801/
Connecting to 0.0.0.0:801... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2018-07-16 14:26:24 ERROR 500: Internal Server Error.
haproxy -f /etc/haproxy/haproxy.cfg -d
[WARNING] 197/200148 (13833) : config : frontend 'GLOBAL' has no 'bind' directive. Please declare it as a backend if this was intended.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use epoll.
Using epoll() as the polling mechanism.
00000000:http_front.accept(0004)=0006 from [127.0.0.1:60696]
00000000:http_front.clireq[0006:ffffffff]: GET / HTTP/1.1
00000000:http_front.clihdr[0006:ffffffff]: User-Agent: Wget/1.14 (linux-gnu)
00000000:http_front.clihdr[0006:ffffffff]: Accept: */*
00000000:http_front.clihdr[0006:ffffffff]: Host: 0.0.0.0:801
00000000:http_front.clihdr[0006:ffffffff]: Connection: Keep-Alive
00000000:http_back.srvrep[0006:0007]: HTTP/1.1 500 Internal Server Error
00000000:http_back.srvhdr[0006:0007]: Content-Type: text/html
00000000:http_back.srvhdr[0006:0007]: Server: Microsoft-IIS/8.0
00000000:http_back.srvhdr[0006:0007]: X-Powered-By: ASP.NET
00000000:http_back.srvhdr[0006:0007]: Date: Tue, 17 Jul 2018 12:02:00 GMT
00000000:http_back.srvhdr[0006:0007]: Connection: close
00000000:http_back.srvhdr[0006:0007]: Content-Length: 1208
00000001:http_front.clicls[0006:ffffffff]
00000001:http_front.closed[0006:ffffffff]
^C
[root#izp0w3tkx2yr8zhes26ajqz ~]#
I tried different config for the server and consistently hit 500 error. Wget to the base server works without any issues.
I setup nginix to the same thing and it works beautifully. Just haproxy does not seem to work. The customer wants it on haproxy. :)
Can you please advise where I can look at to further debug. Appreciate your assistance.
This update from nuster cache server helped solve the problem:
Does your backend Microsoft-IIS/8.0 check host header? as you set http-send-name-header Host, so request from HAProxy to stg-hostserv.com:80 looks like GET / HTTP/1.1 Host: izp0w3tkx2yr8zhes26ajqz
HAProxy worked when I set:
http-request set-header Host stg-hostserv.com

HAPROXY reqirep on Host header not forwarding

I have a web server (actually it's a CF environment, but that doesn't matter much) running behind a haproxy version 1.5-dev19 2013/06/17 that accepts requests only for a certain internal domain, let's call it: internal-address.
Meaning, that HTTP requests only work if the host header is suffixed by internal-address, (like: Host: login.internal-address).
Users from the WAN can access this web server by connecting to an external address which has ip forwarding to the internal server.
But when a user accesses the external address, the Host header is suffixed with external-address, and the web server behind the haproxy rejects the request.
I added reqirep entries in to haproxy configuration:
global
log 127.0.0.1 syslog info
daemon
user vcap
group vcap
maxconn 64000
spread-checks 4
defaults
log global
timeout connect 30000ms
timeout client 300000ms
timeout server 300000ms
frontend http-in
mode http
bind :80
option httplog
option forwardfor
reqadd X-Forwarded-Proto:\ http
default_backend http-routers
frontend https-in
mode http
bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
option httplog
option forwardfor
option http-server-close
reqadd X-Forwarded-Proto:\ https
default_backend http-routers
frontend ssl-in
mode tcp
bind :4443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
default_backend tcp-routers
backend http-routers
mode http
balance roundrobin
reqirep ^Host:\ uaa.external-address Host:\ uaa.internal-address
reqirep ^Host:\ api.external-address Host:\ api.internal-address
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
server node0 172.20.0.1:8888 check inter 1000
backend tcp-routers
mode tcp
balance roundrobin
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
server node0 172.20.0.1:8888 check inter 1000
And every request sent to uaa.external-address or api.external-address is indeed changed and the web server behind the haproxy receives the request as if the Host header is suffixed with internal-address.
But the 3rd rule:
reqirep ^Host:\ external-address:4443 Host:\ loggregator.internal-address:4443
Doesn't work :(
The web server's access log shows the Host header is sent from external-address:4443 , which implies that the haproxy didn't match the Host header correctly and then the web server rejects the request.
The request issued by the client is:
WEBSOCKET REQUEST: [2014-10-01T10:25:07+03:00]
GET /tail/?app=029a1269-67fe-46e2-85f7-e1b0b5d34193 HTTP/1.1
Host: wss://external-address:4443
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: [HIDDEN]
Origin: http://localhost
Authorization: [PRIVATE DATA HIDDEN]
Does anyone know what's wrong with the rule?
EDIT:
I deleted the rules from the backend and created more generic rules in the frontend, and it still doesn't work for websockets:
frontend https-in
mode http
bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
option httplog
option forwardfor
option http-server-close
reqadd X-Forwarded-Proto:\ https
default_backend http-routers
reqirep ^Host:\ (.*).external-address(.*) Host:\ \1.internal-address\2
frontend ssl-in
mode tcp
bind :4443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem
default_backend tcp-routers
reqirep ^Host:\ (.*).external-address(.*) Host:\ \1.internal-address\2
Thanks in advance.
Which version of HAProxy are you running?
If 1.4, please add 'option http-server-close' into your defaults section.
By default, 1.4 is in tunnel mode, which let HAProxy analyse the first request and response and transfer subsequent requests and responses as payload.
In 1.5, it should work out of the box. HAProxy uses a new mode "http-keep-alive" which allows HAProxy to analyse all the content all the time.
Baptiste

Haproxy not matching path (with express.js & Socket.IO)

My config file:
global
maxconn 4096 # Total Max Connections. This is dependent on ulimit
nbproc 2
daemon
log 127.0.0.1 local1 notice
defaults
mode http
frontend all 0.0.0.0:80
timeout client 86400000
default_backend www_backend
acl is_websocket hdr(Upgrade) -i WebSocket
acl is_websocket hdr_beg(Host) -i ws
acl is_websocket path_beg /socket.io
use_backend socket_backend if is_websocket
backend www_backend
balance roundrobin
option forwardfor # This sets X-Forwarded-For
timeout server 30000
timeout connect 4000
server server1 localhost:9001 weight 1 maxconn 1024 check
server server2 localhost:9002 weight 1 maxconn 1024 check
backend socket_backend
balance roundrobin
option forwardfor # This sets X-Forwarded-For
stats enable
timeout queue 5000
timeout server 86400000
timeout connect 86400000
server server1 localhost:5000 weight 1 maxconn 1024 check
As far as I can tell www_backend matches everything. When my web app requests http://myapp.com/socket.io/1/?t=1335831853491 it returns a 404, and the header shows the response came from Express. The odd thing is when I do curl -I http://myapp.com/socket.io/1/?t=1335831853491 it returns:
HTTP/1.1 200 OK
Content-Type: text/plain
Connection: keep-alive
When I run sudo netstat -lptu I can confirm that my socket.io process is running on port 5000. Any thoughts?
Agreed with the response above. BTW, you should not use a 1-day timeout for the TCP connection to establish (timeout connect), it makes no sense at all and will cause connections to accumulate when your server goes down. A connection (especially a local one) is
supposed to establish immediately. I tend to set a 5s timeout for connect, which is far enough even across slow networks.
Concerning the other long timeouts, I'm planning on implementing a "timeout tunnel" so that users don't have to use that large timeouts for normal traffic.
Answer found here:
https://serverfault.com/questions/248897/haproxy-access-list-using-path-dir-having-issues-with-firefox
"ust add "option http-server-close" to your defaults section and it should work."