Setup Varnish cache on LAMP
When visiting my website -> www.arintoker.com getting the error below.
Error 503 Service Unavailable
Service Unavailable
Guru Meditation:
XID: 529248319
when I run varnishlog I get the following output
132 FetchError - http first read error: -1 11 (Resource temporarily unavailable)
133 BackendClose - default
132 VCL_call - error
132 VCL_return - deliver
132 VCL_call - deliver
132 VCL_return - deliver
132 TxProtocol - HTTP/1.1
132 TxStatus - 503
132 TxResponse - Service Unavailable
132 TxHeader - Server: Varnish
132 TxHeader - Content-Type: text/html; charset=utf-8
132 TxHeader - Retry-After: 5
132 TxHeader - Content-Length: 418
132 TxHeader - Accept-Ranges: bytes
132 TxHeader - Date: Sat, 27 Aug 2016 20:07:36 GMT
132 TxHeader - X-Varnish: 529248853
132 TxHeader - Age: 17
132 TxHeader - Via: 1.1 varnish
132 TxHeader - Connection: close
132 Length - 418
132 ReqEnd - 529248853 1472328439.180813074 1472328456.191231966 0.000165224 17.010340691 0.000078201
132 SessionClose - error
When I setup Varnish Cache I followed the guide on DigitalOcean(link)
*Let me know what other reports/resources could be helpful for resolving this issue. Thanks in advance for any help!
Old post but I manage to solve this error by increasing the timeout.
sudo nano /etc/varnish/default.vcl
and having the following config settings:
backend default {
.host = "127..0.0.1";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}
Although 600s as timeout is probably too much, you can have much lower setting which can work for you. For further help read this thread.
PS: I am on Ubuntu 14.04.
I guess Varnish can not talk to your backend. Maybe this can help:
http://www.technoreply.com/solving-dreaded-varnish-503-error/
I was actually able to solve this. My website is a WordPress website so it often that the system receives a brute force attack on the /wp-admin panel. But this time I also noticed the same memory and CPU > 90% alerts from New Relic.
So I ran netstat -natp | grep varnishto figure out which IP was attacking me.
then proceeded to permanently block the IP with my server firewall.
This resolved the issue. Hope this helps someone!
Related
We are getting random 503 responses from Varnish and when looking deeper in varnishlog we see that the backend server (Apache) is reported as a 503 but there is nothing in the apache access log.
How can we trace the cause of this issue?
From varnishlog
* << Request >> 42221068
- Begin req 42221065 rxreq
- Timestamp Start: 1553163507.132639 0.000000 0.000000
- Timestamp Req: 1553163507.132639 0.000000 0.000000
- ReqStart 172.31.19.159 14390
- ReqMethod POST
- ReqURL /path/to/request
- ReqProtocol HTTP/1.1
- ReqHeader X-Forwarded-For: 62.103.214.39
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-Port: 443
- ReqHeader Host: server.com
- ReqHeader X-Amzn-Trace-Id: Root=1-5c9364f3-26a603e3ce2f87d9923ed27e
- ReqHeader Content-Length: 53
- ReqHeader Accept: application/json
- ReqHeader Authorization: Bearer xxx
- ReqHeader Content-Type: application/json; charset=utf-8
- ReqHeader Cookie: PHPSESSID=o1f3010cuch8iu055stl6q1s03
- ReqUnset X-Forwarded-For: 62.103.214.39
- ReqHeader X-Forwarded-For: 62.103.214.39, 172.31.19.159
- VCL_call RECV
- VCL_return pass
- VCL_call HASH
- VCL_return lookup
- VCL_call PASS
- VCL_return fetch
- Link bereq 42221069 pass
- Storage malloc Transient
- Timestamp ReqBody: 1553163507.952175 0.819536 0.819536
- Timestamp Fetch: 1553163507.952297 0.819659 0.000123
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Thu, 21 Mar 2019 10:18:27 GMT
- RespHeader Server: Varnish
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 42221068
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/5.2)
- VCL_call DELIVER
- RespUnset Age: 0
- RespUnset Server: Varnish
- RespUnset X-Varnish: 42221068
- RespUnset Via: 1.1 varnish (Varnish/5.2)
- VCL_return deliver
- Timestamp Process: 1553163507.952322 0.819683 0.000024
- RespHeader Content-Length: 285
- RespHeader Connection: keep-alive
- Timestamp Resp: 1553163507.952341 0.819703 0.000020
- ReqAcct 415 53 468 194 285 479
- End
** << BeReq >> 42221069
-- Begin bereq 42221068 pass
-- Timestamp Start: 1553163507.132720 0.000000 0.000000
-- BereqMethod POST
-- BereqURL /path/to/request
-- BereqProtocol HTTP/1.1
-- BereqHeader X-Forwarded-Proto: https
-- BereqHeader X-Forwarded-Port: 443
-- BereqHeader Host: server.com
-- BereqHeader X-Amzn-Trace-Id: Root=1-5c9364f3-26a603e3ce2f87d9923ed27e
-- BereqHeader Content-Length: 53
-- BereqHeader Accept: application/json
-- BereqHeader Authorization: Bearer 3pa8ot18dlmpad2s34a5gt3p08jg0ml5
-- BereqHeader Content-Type: application/json; charset=utf-8
-- BereqHeader Cookie: PHPSESSID=o1f3010cuch8iu055stl6q1s03
-- BereqHeader X-Forwarded-For: 62.103.214.39, 172.31.19.159
-- BereqHeader X-Varnish: 42221069
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- BackendOpen 32 reload_20190304_175528_18074.live1 172.31.11.111 8081 172.31.31.198 39914
-- BackendStart 172.31.11.111 8081
-- Timestamp Bereq: 1553163507.952205 0.819485 0.819485
-- FetchError HTC status -1
-- BackendClose 32 reload_20190304_175528_18074.live1
-- Timestamp Beresp: 1553163507.952235 0.819514 0.000029
-- Timestamp Error: 1553163507.952237 0.819517 0.000003
-- BerespProtocol HTTP/1.1
-- BerespStatus 503
-- BerespReason Service Unavailable
-- BerespReason Backend fetch failed
-- BerespHeader Date: Thu, 21 Mar 2019 10:18:27 GMT
-- BerespHeader Server: Varnish
-- VCL_call BACKEND_ERROR
-- BerespHeader Content-Type: text/html; charset=utf-8
-- BerespHeader Retry-After: 5
-- VCL_return deliver
-- Storage malloc Transient
-- ObjProtocol HTTP/1.1
-- ObjStatus 503
-- ObjReason Backend fetch failed
-- ObjHeader Date: Thu, 21 Mar 2019 10:18:27 GMT
-- ObjHeader Server: Varnish
-- ObjHeader Content-Type: text/html; charset=utf-8
-- ObjHeader Retry-After: 5
-- Length 285
-- BereqAcct 451 53 504 0 0 0
-- End
From what we understand, Varnish is reporting a connection closed (EOF) from Apache but again Apache is not reporting anything on it's side.
We need to somehow get to the bottom of this as it is effecting service agreement.
Any ideas on tracing and solving the problem?
You don't have an Apache access log because the request is not fully handled by Apache as there is a problem before.
This kind of problem could lead to this situation:
network issue
Apache error, like too many requests, invalid request or some internal error
System error, like too many open files for the Apache user
Where you can look for traces:
Apache error log
System error log
Run a tcpdump (tshark/wireshark are good tools for this) and look for errors
I'm puzzled why this FE doesn't seem to connect me to the BE through my HAproxy:
defaults
mode http
log global
option httplog
option dontlognull
source 0.0.0.0 usesrc clientip # transparent proxy mode
frontend fe-kb
bind :8081 ssl crt /etc/haproxy/ssl/ssl-key.pem
default_backend be-kb
backend be-kb
server afnB afnB:1080 check
I get this in HA http log:
Jan 9 17:25:04 localhost haproxy[17266]: <ip redacted>:51396 [09/Jan/2016:17:24:44.544] fe-kb~ be-kb/afnB 31/0/-1/-1/20036 503 212 - - cC-- 0/0/0/0/3 0/0 "GET / HTTP/1.1"
I can connect fine from HAproxy CLI (selinux is disabled):
[root#hapA ~]# telnet afnB 1080
Trying 10.45.69.14...
Connected to afnB.
Escape character is '^]'.
GET / HTTP/1.0
HTTP/1.1 200 OK
Server: nginx/1.9.9
Date: Sat, 09 Jan 2016 16:40:44 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Wed, 09 Dec 2015 15:05:19 GMT
Connection: close
ETag: "5668432f-264"
Accept-Ranges: bytes
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
Figured it out, the default route wasn't through my HAproxy and as I was trying to do transparency of the source IPs well ;)
Thanks for watching!
'm trying to use the If-Modified-Since command in Telnet. I want to get a 304 Not modified statut code. I tried this but it don't work, I get a 400 bad request error
telnet lemonde.fr 80
GET /index.html HTTP/1.1
User-Agent: Mozilla/5.0
From: yahoo.com
Accept: text/html,text/plain,application/*
Host: www.lemonde.fr
If-Modified-Since: Wed, 19 Oct 2015 10:50:00 GMT
<linefeed>
I got as a result
HTTP/1.0 400 Bad request Cache-Control: no-cache Connection: close
Content-Type: text/html
You need to define the HTTP port. So try:
telnet lemonde.fr 80
The default telnet port is 23. So you won't be able to communicate with the HTTP server.
I'm running cPanel on a Rackspace cloud server that's hosting about 30 websites. I'm using Varnish 3.0.3 + Apache 2.2.23. I'm getting random 503 errors. If I restart the httpd process, they go away. I checked the Apache error_log, and I don't see anything related to a 503 error.
Here's my Varnish config:
backend default {
.host = "198.61.161.65";
.port = "8080";
.connect_timeout = 600s;
.first_byte_timeout = 600s;
.between_bytes_timeout = 600s;
}
Here's my Varnish startup parameters:
-a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u varnish -g varnish -S /etc/varnish/secret -p http_max_hdr=256 -p http_resp_hdr_len=8192 -p http_range_support=on -s malloc,768M
I ran varnishlog and recorded some of the errors. They all have TxResponse=Service Unavailable in them. To me, this means Apache isn't responding to the request from Varnish in a timely manner. Here's an example:
17 SessionOpen c 74.133.75.136 54227 :80
17 ReqStart c 74.133.75.136 54227 1219297893
17 RxRequest c GET
17 RxURL c /
17 RxProtocol c HTTP/1.1
17 RxHeader c Host: www.archerytrade.org
17 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:20.0) Gecko/20100101 Firefox/20.0
17 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
17 RxHeader c Accept-Language: en-US,en;q=0.5
17 RxHeader c Accept-Encoding: gzip, deflate
17 RxHeader c DNT: 1
17 RxHeader c Connection: keep-alive
17 VCL_call c recv lookup
17 VCL_call c hash
17 Hash c /
17 Hash c www.archerytrade.org
17 VCL_return c hash
17 VCL_call c miss fetch
17 Backend c 18 default default
17 FetchError c http first read error: -1 0 (Success)
17 VCL_call c error deliver
17 VCL_call c deliver deliver
17 TxProtocol c HTTP/1.1
17 TxStatus c 503
17 TxResponse c Service Unavailable
17 TxHeader c Server: Varnish
17 TxHeader c Content-Type: text/html; charset=utf-8
17 TxHeader c Retry-After: 5
17 TxHeader c Content-Length: 441
17 TxHeader c Accept-Ranges: bytes
17 TxHeader c Date: Wed, 17 Apr 2013 01:39:52 GMT
17 TxHeader c X-Varnish: 1219297893
17 TxHeader c Age: 0
17 TxHeader c Via: 1.1 varnish
17 TxHeader c Connection: close
17 Length c 441
17 ReqEnd c 1219297893 1366162792.398471832 1366162792.541639328 0.000244379 0.143036604 0.000130892
17 SessionClose c error
17 StatSess c 74.133.75.136 54227 0 1 1 0 0 0 257 441
I've tried adding the following to my default.vcl file:
sub vcl_recv {
set req.grace = 15s;
...
}
sub vcl_fetch {
if (beresp.status == 500 || beresp.status == 503) {
set beresp.saintmode = 10s;
return(restart);
}
set beresp.grace = 1h;
...
}
Around the time of some 503 errors, I found PHP Fatal error, Premature end of script headers, and Request exceeded the limit of 10 internal redirects due to probable configuration error, in the Apache error_log.
I've tried increasing/decreasing the memory for Varnish and the timeout values. I've tried adding the higher http_max_hdr and http_resp_hdr_len values.
We're using APC on this server, and the allotted memory size was only 96M. After increasing the memory a few times and tracking the usage, we settled on 312M. The cache hit rate is 99.7% and the fragmentation is < 0.2%. The Apache 503 errors have almost completely stopped.
Following is what helped me fixing the issue:
After executing following command, I knew about the error:
$ varnishlog -b -q 'FetchError'
Error was "- FetchError http first read error: EOF", details can be found at varnish site
Then after setting up "first_byte_timeout", the issue was fixed:
backend default {
.host = "127.0.0.1";
.port = "8888";
.connect_timeout = 1s; # Wait a maximum of 1s for backend connection (Apache, Nginx, etc...)
.first_byte_timeout = 5s; # Wait a maximum of 5s for the first byte to come from your backend
.between_bytes_timeout = 2s; # Wait a maximum of 2s between each bytes sent
}
Details at Varnish 503 Solutions
I have a .net CF proxy that issues a POST like so:
POST http://192.168.0.172:8080/MIS.WOService/Service.svc HTTP/1.1
Content-Type: text/xml; charset=utf-8
SOAPAction: http://tempuri.org/IWOService/LogIntoServer
Content-Length: 276
Connection: Keep-Alive
Expect: 100-continue
Host: 192.168.0.172:8080
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><LogIntoServer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://tempuri.org/"><emp_pin>55555</emp_pin></LogIntoServer></s:Body></s:Envelope>
The calls always fail with the server returning:
HTTP/1.1 404 Not Found
Date: Thu, 09 Dec 2010 16:07:14 GMT
Server: Microsoft-IIS/6.0
X-AspNet-Version: 4.0.30319
Cache-Control: private
Content-Length: 0
However if, in fiddler I change the POST to a GET it works...
I have the configuration for the website to allow ALL verbs on the svc extension and the " verify file exist" is un-checked.
Does anyone know what could be causing this behavior by IIS 6 ?
When I inspect the IIS logs, it shows:
#Date: 2010-12-09 16:24:34
#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs- username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
2010-12-09 16:24:34 W3SVC1793831887 192.168.0.172 POST /MIS.WOService/Service.svc - 8080 - 192.168.200.67 - 404 0 0
2010-12-09 16:25:14 W3SVC1793831887 192.168.0.172 POST /MIS.WOService/Service.svc - 8080 - 192.168.200.67 - 404 0 0
2010-12-09 16:25:27 W3SVC1793831887 192.168.0.172 POST /MIS.WOService/Service.svc - 8080 - 192.168.200.67 - 404 0 0
2010-12-09 16:25:48 W3SVC1793831887 192.168.0.172 GET /MIS.WOService/Service.svc - 8080 - 192.168.200.67 - 200 0 0
2010-12-09 16:27:16 W3SVC1793831887 192.168.0.172 POST /MIS.WOService/Service.svc - 8080 - 192.168.200.67 - 404 0 0
Apparently I needed to put the name as url as such:
http://192.168.0.172:8080/MIS.WOService/Service.svc/basic
and then the server started allowing the svc to receive the soap messages...
Thanks
Greg
I watched in fiddler what the wcfTestClient sent as requests...
http://192.168.0.172:8080/MIS.WOService/Service.svc/basic
and then the server started allowing the svc to receive the soap messages...