Apache multipart POST "pass request body failed" - apache

Question

We are having problems with our web server (which is configured ssl -> apache -> jetty) randomly rejecting multipart upload POST requests with a 400 Bad Request error code. The apache error log (on info level) shows the following two errors:
[info] [client x1.y1.z1.w1] (70007)The timeout specified has expired: SSL input filter read failed.
[error] proxy: pass request body failed to x.y.z.w:8087 from x1.y1.z1.w1
[info] [client x1.y1.z1.w1] Connection closed to child 74 with standard shutdown
or
[info] [client x2.y2.z2.w2] (70014)End of file found: SSL input filter read failed.
[error] proxy: pass request body failed to x.y.z.w:8087 from x2.y2.z2.w2
[info] [client x2.y2.z2.w2] Connection closed to child 209 with standard shutdown
both cases result from the client side in a 400 Bad Request. Sometimes our jetty server doesn't even see the request meaning that it gets rejected on apaches side, sometimes it starts processing it only to be rejected (this manifests itself as a MultipartException in our UploadFilter)
We have mod_proxy setup to use a fallback load balancing scheme but the logs show that a fallback has not yet been triggered causing me to believe this is not the cause of the problem.
I tried setting SetEnv proxy-sendcl 1 but that didn't change anything.
The upload requests are arount 1mb. Only these multipart file POST requests fail, we have multiple GET requests comming in at the same time and they always work as expected.
If anyone can share any advice or suggestions I would be very grateful! Thank you

Answer 1

If you are using some ajp-enabled backend server, like Tomcat, you may try using mod_proxy_ajp instead of mod_proxy_http. I had a similar problem on a heavy upload app and I fixed it by changing
ProxyPass /myapp http://localhost:8080/myapp
ProxyPassReverse /myapp http://localhost:8080/myapp
by
ProxyPass /myapp ajp://localhost:8009/myapp
ProxyPassReverse /myapp ajp://localhost:8009/myapp
It's also required to enable the ajp connector on tomcat side, of course.
Hope it helps!

Answer 2

Please check this one: https://issues.apache.org/bugzilla/show_bug.cgi?id=44592
The problem could be caused by the HTTP keepalive (KeepAliveTimeout directive) that is killing an established connection with a slow client (tcp latency, slow request body creation, etc).
Try to raise the KeepAliveTimeout in your apache conf, but don't keep it too high to avoid killin' your server (DOS).

Answer 3

You may be seeing this due to timeouts resulting from Apache trying to buffer the entire POST body before passing it through.
Enabling proxy-sendcl may exacerbate this, since this can force Apache to spool a large POST to disk just to calculate the Content-Length when "the original body was sent with chunked encoding (and is large)".
To avoid this, set the environment variable proxy-sendchunked.

Answer 4

After fixing the main problem with the upload, I was still getting some weird logs like:
[reqtimeout:info] [pid 18164:tid 140462752990976] [client 201.76.162.37:41473] AH01382: Request header read timeout
I was able to reduce drastically the frequency it coccurs by increasing the limits of mod_reqtimeout, changing the values of RequestReadTimeout parameter. You can change the default values or redeclare this parameter on your VirtualHost.