Is it possible to have HaProxy failover when it encounters a certain http-status codes?
I have the following generic haproxy code that works fine if the tomcat server itself stops/fails. However I would like to fail-over when http-status codes 502 Bad Gateway or 500 Internal Server Error are also encountered from tomcat. The following configuration will continue to send traffic even when 500, 404 status codes are encountered in any node.
backend db01_replication
mode http
bind 192.168.0.1:80
server app1 10.0.0.19:8080 check inter 10s rise 2 fall 2
server app2 10.0.0.11:8080 check inter 10s rise 2 fall 2
server app3 10.0.0.13:8080 check inter 10s rise 2 fall 2
Thanks In Advance
I found the following HaProxy http-check expect to resolve the load-balancing based on http status codes.
# Only accept status 200 as valid
http-check expect status 200
# Consider SQL errors as errors
http-check expect ! string SQL\ Error
# Consider all http status 5xx as errors
http-check expect ! rstatus ^5
In order to fail-over when a 500 error is encountered, the HaProxy configuration would look like:
backend App1_replication
mode http
bind 192.168.0.1:80
http-check expect ! rstatus ^5
server app1 10.0.0.19:8080 check inter 10s rise 2 fall 2
server app2 10.0.0.11:8080 check inter 10s rise 2 fall 2
server app3 10.0.0.13:8080 check inter 10s rise 2 fall 2
Source
https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#http-check%20expect
Related
I have the following haproxy backend configuration.
I'm surprised that in haproxy status page the check is reported as "L6ok".
Why Layer 6 and not Layer 7 ?
backend back:lb
option httpchk GET / HTTP/1.1\r\nHost:\ www.domain.tld
default-server ca-file my.ca inter 10s maxconn 50 maxqueue 10 ssl check
server lb-01.{{ domain }} 10.10.10.30:443 weight 100
server lb-02.{{ domain }} 10.10.10.31:443 weight 100
for information, I added the expected http return code (302):
http-check expect status 302
and now checks are reported in l7:
L7OK/302 in 12ms
I created an python api on openshift online with python image. If you request all the data, it takes more than 30 seconds to respond. The server gives a 504 gateway timeout http response. How do you configure the length a response can take? > I created an annotation on the route, this seems to set proxy timeout.
haproxy.router.openshift.io/timeout: 600s
Problem remains, I now got logging. It looks like the message comes from mod_wsgi.
I want to try alter the configuration of the httpd (mod_wsgi-express process) from request-timeout 60 to request-timeout 600. Where doe you configure this. I am using base image https://github.com/sclorg/s2i-python-container/tree/master/2.7
Logging:
Timeout when reading response headers from daemon process 'localhost:8080':/tmp/mod_wsgi-localhost:8080:1000430000/htdocs
Does someone know how to fix this error on openshift online
Next to alter timeout of haproxy of the route of my app
haproxy.router.openshift.io/timeout: 600s
I altered the request-timeout and socket-timeout in app.sh of my python application. So the mod_wsgi-express server is configured with a higher timeout
ARGS="$ARGS --request-timeout 600"
ARGS="$ARGS --socket-timeout 600"
My application now wait 10 minutes before cancelling a request
I am trying to solving a scenario now using haproxy. The scenario as below
Block all IP by default
Allow only connection from a specific IP address
If any connections come from a whilelist IP, if should reject if it exceed more than 10 concurrent connection in 30 sec
I want to do this to reduce number of API calls into my server. Could any one please help me with this?
Thanks
First two things are easy, simply allow only whitelisted IP
acl whitelist src 10.12.12.23
use_backend SOMESERVER if whitelist
The third - throttling - requires to use stick-tables (there are many data type - counters conn, sess, http, rates...) as a rate counter:
# max entries count request in 60s periods
stick-table type ip size 200k expire 100s store http_req_rate(60s)
next you have to fill the table, by tracking each request eg. by IP
tcp-request content track-sc0 src
# more info at http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-tcp-request%20connection
and finally the acl:
# is there more than 5req/1min from IP
acl http_rate_abuse sc0_http_req_rate gt 5
# update use_backend condition
use_backend SOMESERVER if whitelisted !http_rate_abuse
For example some working config file with customized errors:
global
log /dev/log local1 debug
defaults
log global
mode http
option httplog
retries 3
option redispatch
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
frontend http
bind *:8181
stick-table type ip size 200k expire 100s store http_req_rate(60s)
tcp-request content track-sc0 src
acl whitelist src 127.0.0.1
acl http_rate_abuse sc0_http_req_rate gt 5
use_backend error401 if !whitelist
use_backend error429 if http_rate_abuse
use_backend realone
backend realone
server local stackoverflow.com:80
# too many requests
backend error429
mode http
errorfile 503 /etc/haproxy/errors/429.http
# unauthenticated
backend error401
mode http
errorfile 503 /etc/haproxy/errors/401.http
Note: the error handling is a bit tricky. Because above error backends are missing server entries, haproxy will throw HTTP 503, errorfile catch them and send different errors (with different codes).
Example /etc/haproxy/errors/401.http content:
HTTP/1.0 401 Unauthenticated
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>401 Unauthenticated</h1>
</body></html>
Example /etc/haproxy/errors/429.http content:
HTTP/1.0 429 Too many requests
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>429 Too many requests</h1>
</body></html>
We have 3 Machines
mod_jk with load balancer
first Worker on tomcat8
2nd Worker on tomcat8
everything works as expected but, when one of the tomcat is being shutting down the status page on the load balancer still shows that the state of this worker is OK/IDLE.
Any ideas how to force the status page to check the real status of the worker?
Related Materials
worker.properties
\### Define worker names
worker.list=status,loadbalancer
\### Declare Tomcat server 1
worker.worker1.port=8409
worker.worker1.host=centureapp1
worker.worker1.type=ajp13
worker.worker1.lbfactor=1
\### Declare Tomcat server 2
worker.worker2.port=8410
worker.worker2.host=centureapp2
worker.worker2.type=ajp13
worker.worker2.lbfactor=1
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=worker1,worker2
worker.loadbalancer.sticky_session=1
worker.status.type=status
~
By default balancer maintenance runs every 60 seconds. So, you will see the state of this worker after 60 seconds.
I have Apache + Tomcat setup with mod_jk on 2 servers. Each server has its own Apache+Tomcat pair, and every request is being served by Tomcat load balancing workers on 2 servers.
I have a question about how Apache's maxClient and Tomcat's maxThread should be set.
The default numbers are,
Apache: maxClient=150, Tomcat: maxThread=200
In this configuration, if we have only 1 server setup, it would work just fine as Tomcat worker never receives the incoming connections more than 150 at once. However, if we are load balancing between 2 servers, could it be possible that Tomcat worker receives 150 + (some number from another server) and make the maxThread overflow as SEVERE: All threads (200) are currently busy?
If so, should I set Tomcat's maxThread=300 in this case?
Thanks
Setting maxThreads to 300 should be fine - there are no fixed rules. It depends on whether you see any connections being refused.
Increasing too much causes high memory consumption but production Tomcats are known to run with 750 threads. See here as well. http://java-monitor.com/forum/showthread.php?t=235
Have you actually got the SEVERE error? I've tested on our Tomcat 6.0.20 and it throws an INFO message when the maxThreads is crossed.
INFO: Maximum number of threads (200) created for connector with address null and port 8080
It does not refuse connections until the acceptCount value is crossed. The default is 100.
From the Tomcat docs http://tomcat.apache.org/tomcat-5.5-doc/config/http.html
The maximum queue length for incoming
connection requests when all possible
request processing threads are in use.
Any requests received when the queue
is full will be refused. The default
value is 100.
The way it works is
1) As the number of simultaneous requests increase, threads will be created up to the configured maximum (the value of the maxThreads attribute).
So in your case, the message "Maximum number of threads (200) created" will appear at this point. However requests will still be queued for service.
2) If still more simultaneous requests are received, they are queued up to the configured maximum (the value of the acceptCount attribute).
Thus a total of 300 requests can be accepted without failure. (assuming your acceptCount is at default of 100)
3) Crossing this number throws Connection Refused errors, until resources are available to process them.
So you should be fine until you hit step 3