Varnish/Apache Random 503 Errors - apache

Question

I'm running cPanel on a Rackspace cloud server that's hosting about 30 websites. I'm using Varnish 3.0.3 + Apache 2.2.23. I'm getting random 503 errors. If I restart the httpd process, they go away. I checked the Apache error_log, and I don't see anything related to a 503 error.
Here's my Varnish config:
backend default {
.host = "198.61.161.65";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}
Here's my Varnish startup parameters:
-a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -p http_max_hdr=256 -p http_resp_hdr_len=8192 -p http_range_support=on -s malloc,768M
I ran varnishlog and recorded some of the errors. They all have TxResponse=Service Unavailable in them. To me, this means Apache isn't responding to the request from Varnish in a timely manner. Here's an example:
17 SessionOpen c 74.133.75.136 54227 :80
17 ReqStart c 74.133.75.136 54227 1219297893
17 RxRequest c GET
17 RxURL c /
17 RxProtocol c HTTP/1.1
17 RxHeader c Host: www.archerytrade.org
17 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:20.0) Gecko/20100101 Firefox/20.0
17 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
17 RxHeader c Accept-Language: en-US,en;q=0.5
17 RxHeader c Accept-Encoding: gzip, deflate
17 RxHeader c DNT: 1
17 RxHeader c Connection: keep-alive
17 VCL_call c recv lookup
17 VCL_call c hash
17 Hash c /
17 Hash c www.archerytrade.org
17 VCL_return c hash
17 VCL_call c miss fetch
17 Backend c 18 default default
17 FetchError c http first read error: -1 0 (Success)
17 VCL_call c error deliver
17 VCL_call c deliver deliver
17 TxProtocol c HTTP/1.1
17 TxStatus c 503
17 TxResponse c Service Unavailable
17 TxHeader c Server: Varnish
17 TxHeader c Content-Type: text/html; charset=utf-8
17 TxHeader c Retry-After: 5
17 TxHeader c Content-Length: 441
17 TxHeader c Accept-Ranges: bytes
17 TxHeader c Date: Wed, 17 Apr 2013 01:39:52 GMT
17 TxHeader c X-Varnish: 1219297893
17 TxHeader c Age: 0
17 TxHeader c Via: 1.1 varnish
17 TxHeader c Connection: close
17 Length c 441
17 ReqEnd c 1219297893 1366162792.398471832 1366162792.541639328 0.000244379 0.143036604 0.000130892
17 SessionClose c error
17 StatSess c 74.133.75.136 54227 0 1 1 0 0 0 257 441
I've tried adding the following to my default.vcl file:
sub vcl_recv {
set req.grace = 15s;
...
}
sub vcl_fetch {
if (beresp.status == 500 || beresp.status == 503) {
set beresp.saintmode = 10s;
return(restart);
}
set beresp.grace = 1h;
...
}
Around the time of some 503 errors, I found PHP Fatal error, Premature end of script headers, and Request exceeded the limit of 10 internal redirects due to probable configuration error, in the Apache error_log.
I've tried increasing/decreasing the memory for Varnish and the timeout values. I've tried adding the higher http_max_hdr and http_resp_hdr_len values.

Answer 1

We're using APC on this server, and the allotted memory size was only 96M. After increasing the memory a few times and tracking the usage, we settled on 312M. The cache hit rate is 99.7% and the fragmentation is < 0.2%. The Apache 503 errors have almost completely stopped.

Answer 2

Following is what helped me fixing the issue:
After executing following command, I knew about the error:
$ varnishlog -b -q 'FetchError'
Error was "- FetchError http first read error: EOF", details can be found at varnish site
Then after setting up "first_byte_timeout", the issue was fixed:
backend default {
.host = "127.0.0.1";
.port = "8888";
.connect_timeout = 1s; # Wait a maximum of 1s for backend connection (Apache, Nginx, etc...)
.first_byte_timeout = 5s; # Wait a maximum of 5s for the first byte to come from your backend
.between_bytes_timeout = 2s; # Wait a maximum of 2s between each bytes sent
}
Details at Varnish 503 Solutions