I am getting crazy with mod_cache!
My current setup:
Apache 2.4 on Ubuntu with:
- mpm_worker
- mod_jk
- mod_cache
- mod_cache_disk
- mod_expires
- mod_deflate
HTTP & HTTPS Request are balanced by modjk to 5 Tomcat AppServers.
I want to cache media assets served by Tomcat Instances, but getting trouble with mod_cache.
My current cache configuration:
CacheRoot /srv/volatile/cache
CacheDirLevels 3
CacheDirLength 2
CacheEnable disk /medias
# Currently active:
CacheQuickHandler off
CacheLock on
CacheLockPath /tmp/mod_cache-lock
CacheLockMaxAge 5
CacheIgnoreHeaders Set-Cookie
# This is another configuration i tried:
#CacheIgnoreNoLastMod On
#CacheIgnoreCacheControl On
#CacheIgnoreQueryString On
#CacheIgnoreHeaders Set-Cookie
I checked a lot of tutorials und guides, but without success.
The first request is getting served fine by Apache, following request are failing with weird errors:
Recalled cached URL info header https://...
Recalled headers for URL https://...
Adding CACHE_SAVE filter for /medias...
Adding CACHE_REMOVE_URL filter for /medias...
cache: /media... responded with an uncacheable 304, retrying the request. Reason: contradiction: 304 Not Modified, but Content-Length modified
cache: Removing url https://
Deleting /srv/volatile/cache/.../.header from cache.
Deleting /srv/volatile/cache/.../.data from cache.
Deleting directory /srv/volatile/cache/.../7n from cache.
URL https://... failed the size check (0 < 1)
I thought that it could be a load balancing issue, because of walk threw multiple nodes or something like that, but the behavior isn't deterministic. Tests with apache-bench & jmeter showed me that 60-70% of request failed, so that not exactly every X request failed.
If the CacheIgnoreCacheControl On option is set, requests threw jmeter & apache-bench didn't fail, but access threw Browser failed.
Someone any idea ?
Thanks
Taulant
I think, this is covered by mod_cache bug 56881, which is fixed in Apache 2.4.11.
Related
I have an issue where my Apache server is returning an Upgrade H2 header response sporadically. This causes the following issue with Safari and HTTP2:
http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [h2]
HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
I have seen similar issues on stackoverflow but none of the proposed solutions have worked for me. I have tried setting "Header unset Upgrade" in the httpd.conf file as well as the ssl.conf and virtualhost specific configuration files. I also tried to unset the header in a location directive in those files.
I have removed the h2c protocol from the mod_http2.c configuration in httpd.conf based on other people having this problem but it did not work by itself or in conjunction with the other changes above.
Protocols h2 http/1.1
I have no idea why the Header unset Upgrade statement is not working. We are using Apache as a reverse proxy with a tomcat application backend. We also have an AWS ALB in front of Apache. Does anyone know why this would not be working or another way to remove the response header "Upgrade"?
Behind Apache 2.2 httpd server, i have a java REST-based application with this kind of service :
GET /countries/canada
I'd like to enable mod_mem_cache to speed up these services like that :
CacheEnable mem /countries/*
Is it possible to use wildcard ?
If not, what is there any solution to do that ?
In fact, we don't need wildcard.
For example, if a defined the following configuration :
ProxyPass /jsontest/ http://echo.jsontest.com/
ProxyPassReverse /jsontest/ http://echo.jsontest.com/
<IfModule mod_cache.c>
<IfModule mod_mem_cache.c>
CacheEnable mem /jsontest/
CacheDefaultExpire 300
MCacheSize 1024000
MCacheMaxObjectCount 10000
MCacheMinObjectSize 1
MCacheMaxObjectSize 2048000
CacheStorePrivate On
CacheIgnoreNoLastMod On
CacheStoreNoStore On
CacheIgnoreCacheControl On
</IfModule>
</IfModule>
I can request :
http://localhost/jsontest/
or
http://localhost/jsontest/titi/tutu
and apache will store each response and serve them :
Incoming request is asking for an uncached version of /jsontest/, but we have been configured to ignore it and serve cached content anyway
mem_cache: Cached url: http://localhost:80/jsontest/?
Incoming request is asking for an uncached version of /jsontest/, but we have been configured to ignore it and serve cached content anyway
mod_cache.c(312): cache: serving /jsontest/
Incoming request is asking for an uncached version of /jsontest/titi/tutu, but we have been configured to ignore it and serve cached content anyway
mod_cache.c(753): cache: Caching url: /jsontest/titi/tutu
Incoming request is asking for an uncached version of /jsontest/titi/tutu, but we have been configured to ignore it and serve cached content anyway
mod_cache.c(312): cache: serving /jsontest/titi/tutu
Be carefull :
With CacheStorePrivate, CacheIgnoreNoLastMod, CacheStoreNoStore and CacheIgnoreCacheControl, i disabled all default behavior and force cache !!!
You have to ask yourself if it is your requirement.
I'm currently in the process of configuring Apache httpd to direct all traffic for '/api/foo' to a specific Tomcat instance via AJP (using mod_proxy_ajp). For this purpose I have the following ProxyPass rule in my httpd configuration file:
ProxyPass /api/foo ajp://localhost:9999/api/foo connectiontimeout=300 timeout=300 retry=3
This Tomcat instance has the following connector defined in its server.xml:
<Connector port="9999" protocol="AJP/1.3" redirectPort="9443"/>
With this configuration I reach my Tomcat instance correctly when visiting /api/foo. However, when the URL becomes bigger than 300 characters I can't seem to reach my Tomcat instance, sometimes.
The first hour it could be that I get a timeouts and the other hour it could be that everything comes through just fine.
When I get timeouts, I see the following errors in my httpd error log:
[error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[error] ajp_read_header: ajp_ilink_receive failed
[error] (70007)The timeout specified has expired: proxy: read response failed from 127.0.0.1:9999 (localhost)
And the following result in my httpd access log:
"GET /api/foo/barrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr HTTP/1.1" 503 323 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:33.0) Gecko/20100101 Firefox/33.0"
The strange thing is that this request doesn't seem to reach my Tomcat instance. I don't see anything reaching my application logging, and I also don't see the request in my Tomcat access log.
Does anyone have an idea what could be the cause of this problem? The timeout is set to more than 1 minute, so I'd expect the request to at least reach my Tomcat instance...
As a last note, I've tried setting the AJP connector's packageSize to 65536 and Apache httpd's LimitRequestFieldSize and ProxyIOBufferSize to 65536. This didn't solve the problem.
In the absence of a better answer I though I would share my experience with a similar and possibly the same problem and give you a recommendation to change the connector you are using.
I have also received the same ajp_read_header: ajp_ilink_receive failed errors while using mod_proxy_ajp in slightly different environments using apache 2.2 & tomcat 6.
I've tried many various and recommended fixes including the connector timeout and other connector settings when we encountered our issue.
From what I've read, some of these will work in different scenarios and the error ajp_ilink_receive failed message seems to be quite generic. In your case this occurs when the URL is longer than 300 characters - well sometimes at least..
After trying many of the different connector properties both from within the tomcat server.xml and the apache ProxyPass settings appended after the ajp:// URI I pretty much gave up going down this path. Since then I have tried mod_proxy and mod_jk which both seem fix the issue or at the very least there is no more ajp error. But this wouldn't happen with mod_proxy anyway because it doesnt connect with ajp.
So I suggest to change your tomcat connector to either mod_proxy or mod_jk. There are various write ups on the differences but the pages here and here on tomcat expect are a reasonably good source of information. You;ll want to choose between mod_proxy and mod_jk based on your setup. mod_jk, while different, is actually not too hard to set up (but you may have to build the mod_jk.so for apache).
Summarising the different connectors;
mod_jk: highest recommendation. ajp connector. purpose designed. different configuration setup that resides outside apache. difficult to configure when connecting a webapp with a certian name to a different url name.
mod_proxy: http connector. reportedly more stable than mod_proxy_ajp. problems with passing ssl variables?
mod_proxy_ajp: least recommended but most common. ajp connector. extension of mod_proxy. easy implementation.
If this doesn't work for some reason. Others have had success looking into problems on the tomcat side such as long running queries.
We are having problems with our web server (which is configured ssl -> apache -> jetty) randomly rejecting multipart upload POST requests with a 400 Bad Request error code. The apache error log (on info level) shows the following two errors:
[info] [client x1.y1.z1.w1] (70007)The timeout specified has expired: SSL input filter read failed.
[error] proxy: pass request body failed to x.y.z.w:8087 from x1.y1.z1.w1
[info] [client x1.y1.z1.w1] Connection closed to child 74 with standard shutdown
or
[info] [client x2.y2.z2.w2] (70014)End of file found: SSL input filter read failed.
[error] proxy: pass request body failed to x.y.z.w:8087 from x2.y2.z2.w2
[info] [client x2.y2.z2.w2] Connection closed to child 209 with standard shutdown
both cases result from the client side in a 400 Bad Request. Sometimes our jetty server doesn't even see the request meaning that it gets rejected on apaches side, sometimes it starts processing it only to be rejected (this manifests itself as a MultipartException in our UploadFilter)
We have mod_proxy setup to use a fallback load balancing scheme but the logs show that a fallback has not yet been triggered causing me to believe this is not the cause of the problem.
I tried setting SetEnv proxy-sendcl 1 but that didn't change anything.
The upload requests are arount 1mb. Only these multipart file POST requests fail, we have multiple GET requests comming in at the same time and they always work as expected.
If anyone can share any advice or suggestions I would be very grateful! Thank you
If you are using some ajp-enabled backend server, like Tomcat, you may try using mod_proxy_ajp instead of mod_proxy_http. I had a similar problem on a heavy upload app and I fixed it by changing
ProxyPass /myapp http://localhost:8080/myapp
ProxyPassReverse /myapp http://localhost:8080/myapp
by
ProxyPass /myapp ajp://localhost:8009/myapp
ProxyPassReverse /myapp ajp://localhost:8009/myapp
It's also required to enable the ajp connector on tomcat side, of course.
Hope it helps!
Please check this one: https://issues.apache.org/bugzilla/show_bug.cgi?id=44592
The problem could be caused by the HTTP keepalive (KeepAliveTimeout directive) that is killing an established connection with a slow client (tcp latency, slow request body creation, etc).
Try to raise the KeepAliveTimeout in your apache conf, but don't keep it too high to avoid killin' your server (DOS).
You may be seeing this due to timeouts resulting from Apache trying to buffer the entire POST body before passing it through.
Enabling proxy-sendcl may exacerbate this, since this can force Apache to spool a large POST to disk just to calculate the Content-Length when "the original body was sent with chunked encoding (and is large)".
To avoid this, set the environment variable proxy-sendchunked.
After fixing the main problem with the upload, I was still getting some weird logs like:
[reqtimeout:info] [pid 18164:tid 140462752990976] [client 201.76.162.37:41473] AH01382: Request header read timeout
I was able to reduce drastically the frequency it coccurs by increasing the limits of mod_reqtimeout, changing the values of RequestReadTimeout parameter. You can change the default values or redeclare this parameter on your VirtualHost.
This is a rather complicated scenario, so I would highly appreciate any pointer to the correct direction.
So I have setup apache on server A to proxy https traffic το server B, that is a plone site behind varnish and apache.
I connect to A and can browse the site on https, everything is fine. However, problems start when I upload files, via plone's POST forms. I can upload small files (~1 MB), but when I try to upload a 50MB file, I wait all the time till the file is uploaded, and when the indication is 100%, I get a Bad gateway (The proxy server received an invalid response from an upstream server.)
It seems to me that something timeouts between the communication of A and B and instead of being redirected to the correct url, I get a Bad gateway, not to mention that the file is not uploaded.
On the apache log I see
[error] proxy: pass request body failed
As suggested on other threads, I've experimented with the following values with no luck
force-proxy-request-1.0
proxy-nokeepalive
KeepAlive
KeepAliveTimeout
proxy-initial-not-pooled
Timeout
ProxyTimeout
Sooooo..any suggestions? Thanks a million in advance!
Did you check the varnish configuration? varnish has some timeouts of its own, I am familiar with send_timeout which usually breaks downloads if they fail to finish within a few seconds (Varnish really isn't any good for large downloads, because you end doing stupid things like configuring send_timeout=7200 to make it work).
Also, set first_byte_timeout to a larger number for that backend, because a large file upload might delay plone's response just enough to cause this.
Setting the Timeout and KeepAliveTimeout in the apache virtual host file worked for me.
Example:
Timeout 3600
KeepAliveTimeout 50