So I'm running Tomcat(8.0) behind Apache(2.4) on Windows Server 2012 and using ProxyPass to pass through all traffic. Everything works fine, but whenever I do nothing for 60 seconds, and then hit the server again, i get a 8-20 second delay, like apache is creating a new process to handle the request.
My configuration is pretty much the default that comes with Apache Haus, with the addition of the proxy stuff, which I believe is the culprit:
ProxyPass /static/ !
ProxyPass / http://localhost:8088/
ProxyPassReverse / http://localhost:8088/
I added the
/static/ !
exemption to see if same problem would happen on static files being served, and apparently it does. I further narrowed it down by commenting out all the ProxyPass stuff, and verifying my static file always loads fast. Then i uncommented ProxyPass stuff, and only requested my static file, and it again always returned fast. But once I hit a URL that takes me through the proxy, wait a minute, then hit it again, something goes horribly wrong. Below is network monitor output for two requests, first of the static file being requested a second time after a 1 minute delay before proxy use, the other after the proxy had been used twice with delay between proxy requests.
3501 4:17:48 PM 10/21/2015 104.2752287 httpd.exe HTTP HTTP:Request, GET /static/index.html
3502 4:17:48 PM 10/21/2015 104.2760830 httpd.exe HTTP HTTP:Response, HTTP/1.1, Status: Not modified, URL: /static/index.html
After (8 seconds to return):
24232 4:26:13 PM 10/21/2015 608.7355960 httpd.exe HTTP HTTP:Request, GET /static/index.html
24775 4:26:20 PM 10/21/2015 616.0896861 httpd.exe HTTP HTTP:Response, HTTP/1.1, Status: Not modified, URL: /static/index.html
I'm noticing more of this SynReTransmit line after it was initially broken, not sure if it's relevant:
24226 4:26:13 PM 10/21/2015 608.7286692 httpd.exe TCP TCP:[SynReTransmit #24107]Flags=......S., SrcPort=61726, DstPort=HTTP(80), PayloadLen=0, Seq=1157444168, Ack=0, Win=8192 ( Negotiating scale factor 0x2 ) = 8192
But basically every call, be it to static file or over proxy, if it's been over 60 seconds since the last call, will take forever to get a response!
Any ideas?
UPDATE:
I was running a slightly older version of Apache, 2.4.12, but updating to latest, 2.4.17, didn't fix it. I've tried all sorts of keepalive settings, nothing seems to help. On another forum i was directed at this apache dev thread which has a proposed patch for what sounds like a similar issue, guess I'll wait for an apache update:
http://marc.info/?l=apache-httpd-dev&m=144543644225945&w=2
Try explicitly tuning the ProxyReceiveBufferSize:
# For increase throughput (bytes)
ProxyReceiveBufferSize 2048
In httpd config, add these follow lines:
AcceptFilter http none
AcceptFilter https none
EnableSendfile Off
EnableMMAP off
right after this line:
Listen 80
My response get less than 2 time but it still quite slow than normal.
From https://www.apachelounge.com/viewtopic.php?p=26601
I was using Apache httpd as reverse proxy and it was drastically slow (2 mins to load a single web page). But, as soon as changed the hostname to IP address it was super fast.
before:
ProxyPass "/home" "http://hostname.domain.com:port/home"
After:
ProxyPass "/home" "http://ip:port/home"
Hope it helps someone.
I'm trying to setup Apache as a load balancer for 2 Tomcat instances with session affinity.
The goal is to have the session stick to one server but to have next session (when it's changed by the backend server) to go to the next available server (let's say using round-robin algorithm for easier implementation). When using the "jvmRoute" in Tomcat and an equivalent "route" in Apache the actual value that does the routing is the route name which does not change and all requests are routed always to the same backend server for a single client.
I found out so far that there's an chicken/egg problem when using just the JSESSIONID cookie. Let's consider the following setup:
2 Tomcat servers listening on ports 8009 and 8010 (AJP13)
1 Apache server with the following configuration
<Proxy balancer://hello-cluster>
BalancerMember ajp://127.0.0.1:8009/hello
BalancerMember ajp://127.0.0.1:8010/hello
</Proxy>
ProxyPass /hello balancer://hello-cluster stickysession=JSESSIONID
And here's the scenario:
The first request has no cookie so Apache selects the next available server in the load balancer to handle the request.
The backend Tomcat server sets JSESSIONID but does not note the actual value being returned.
The next request comes in, Apache notes that there's no backend server noted for the given JSESSIONID so it selects the next available, which in this case the other one as served the first request
Tomcat notices that the value of JSESSIONID is invalid so it creates a new one.
Apache does not take a note that the JSESSIONID has changed to pin it down to that backend server.
Back to pt. 3
Is there a way to convince Apache to note the value returned by Tomcat?
maybe if you try with tomcat session replication. I found this interesting post:
http://www.bradchen.com/blog/2012/12/tomcat-auto-failover-using-apache-memcached
.
You could try too with redis:
http://shivganesh.com/2013/08/15/setup-redis-session-store-apache-tomcat-7/
Let me know your experience please.
I am using mod_proxy_balancer to manage failover of backend servers. Backend servers may return an error code instead of timing out when some other backend service fails such as NFS and we want such servers also to be marked as failed nodes. Hence we are using failonstatus directive.
<Proxy balancer://avatar>
ProxySet failonstatus=503
BalancerMember http://active/ retry=30
# the hot standby
BalancerMember http://standby/ status=+H retry=0
</Proxy>
Currently the failover works perfectly with one glitch. When active node fails the user gets a 503 error and from the next request the Standby server takes over.
I dont want even a single request to fail though. Cant mod_proxy failover with out ever returning an error to the client? If active node fails I want mod_proxy to try the Standby for the same request and not just from the subsequent request!
I think you asked this on the Apache HTTPd mailing list but sadly didn't get a satisfactory answer. I've asked almost the same question in ServerFault so I'm joining them together.
https://serverfault.com/questions/414024/apache-httpd-workers-retry
There is a new module that accomplishes what you are asking
https://httpd.apache.org/docs/2.4/mod/mod_proxy_hcheck.html
I have a few HTTP/PHP servers fronted by an Apache/mod_proxy_balancer load balancer, with a typical cluster setup as below:
<Proxy balancer://mycluster>
BalancerMember http://192.168.1.10
BalancerMember http://192.168.1.11
BalancerMember http://192.168.1.12
BalancerMember http://192.168.1.13
</Proxy>
My question: is there a way to configure Apache such that some specific requests can be sent to all members of the cluster instead of being the standard proxied to one?
I ask because each member in my cluster uses a local data cache based on XCache. Each member has a http-accessible script to unset specific cache item on itself. On some rare events, I need to clear the same cache entry on all servers.
I could make a separate bash/curl script to hit each server in sequence, but since the cluster definition is in my httpd.conf, it'd be easier to not have to copy it somewhere else, and also, I'm hoping to retain a curl-able endpoint on the load balancer itself to do a global cache clearing.
Note: I am not asking about memcache. I'm already using memcache for other things, where servers need to have synchronized storage. In this case I use Xcache for almost-persistent, no-sync-required, data caching.
I am using the following Apache config to forward requests to a Tomcat server:
ProxyPass /myapp ajp://localhost:8009/myapp max=2
This is a simplified config, but is enough to reproduce the issue, which is that the max parameter has no effect. If I through 10 concurrent requests to Apache, all 10 are forwarded to Tomcat at the same time, while I would like to have them forwarded 2 by 2. Should I use something other than the max parameter for this?
The max=2 failed to limit the number of requests concurrently forwarded to Tomcat because I was running this on UNIX, and my Apache came preconfigured with prefork MPM, which creates one process per request. The max applies per process, hence doesn't have the desired effect.
If you are in this situation and need to limit the number concurrent request forwarded to Tomcat, then you'll need to replace your Apache with a worker or event MPM Apache, in the config set ServerLimit to 1, and ThreadsPerChild and MaxClients to the same value, which will be the total number of concurrent connections your Apache will be able to process. You can find more information about this in this section documenting the recommended Apache configuration for Orbeon Forms.
service apache2 restart