I can't figure out, for the life of me, how to get varnish to ignore caching of 500 internal server errors. Basically, if someone hits varnish and is returned a 500 internal server error, I want varnish to not cache that page (set a 0s ttl/grace period?). I'm using varnish 3.0.3 and here's my VCL. By default, I want to cache the pages for 30 days.
sub vcl_fetch {
# Set 30-day TTL
set beresp.ttl = 2592000 s;
set beresp.grace = 15d; /* The max amount of time to keep object in cache */
if (beresp.status == 301 || beresp.status == 302) {
return (hit_for_pass);
}
# Serve pages from the cache should we get a sudden error and re-check in one minute
if (beresp.status >= 500) {
set beresp.grace = 1s;
set beresp.ttl = 1s;
return (hit_for_pass);
}
# Unset the "etag" header (suggested)
unset beresp.http.etag;
return(deliver);
}
So, in english: if a 500 internal server is returned... the X-CACHE should show a MISS. When I refresh the page, if it is still 500 internal server, then it should again show a MISS. If the page is successfully delivered, it should show a HIT.
By default Varnish will only cache the following status codes[1]:
200: OK
203: Non-Authoritative Information
300: Multiple Choices
301: Moved Permanently
302: Moved Temporarily
307: Temporary Redirect
410: Gone
404: Not Found
Note that the first time the page is successfully delivered you will still get a MISS
[1] http://book.varnish-software.com/3.0/VCL_Basics.html#the-initial-value-of-beresp-ttl
Related
I would like to use Varnish for Symfony and for my eZ Platform CMS.I have followed this tutorial to set up my Varnish : http://book.varnish-software.com/4.0/chapters/Getting_Started.html#exercise-install-varnish
So I have the following working server :
Varnish listening on port 80
Varnish Uses backend on localhost:8080
Apache listening on localhost:8080
I have also setted up my eZ Platform ezplatform.yml and ezplatform.conf, to disable the default cache and enable varnish (I guess).
I added these two line to ezplatform.conf folling the documentation https://doc.ez.no/display/TECHDOC/Using+Varnish:
SetEnv USE_HTTP_CACHE 0
SetEnv TRUSTED_PROXIES "0.0.0.0"
I put 0.0.0.0 for Varnish server IP address because netstat -nlpt retreive me the following addresses for Varnish servers :
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 11151/varnishd
tcp 0 0 127.0.0.1:1234 0.0.0.0:* LISTEN 11151/varnishd
So I guess this is the right value.
Then, I added the following lines to my ezplatform.yml (checked the documentation above) :
ezpublish:
http_cache:
purge_type: http
siteaccess:
list: [site]
groups:
http_cache:
purge_servers: 0.0.0.0:80
Varnish and httpd restarted well. Then, I checked if Varnish was used by local website, checking the HTTP headers, and I got this :
Via: "1.1 varnish-v4"
X-Varnish: "32808"
Which is, I guess, a good progress.
Anyaway, In the Symfony profiler, I still have the following intels :
Cache Information
Default Cache default
Available Drivers Apc, BlackHole, FileSystem, Memcache, Redis, SQLite
Total Requests 349
Total Hits 349
Cache Service: default
Drivers FileSystem
Calls 349
Hits 349
Doctrine Adapter false
Cache In-Memory true
Is it normal to still get this ? Shouldn't it be something like Default Cache : varnish instead of default ? How can I check if my Varnish is currently working on my site instead of the symfony default cache ?
Btw, here is my vcl file :
#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and https://www.varnish-cache.org/trac/wiki/VCLExamples for more examples.
# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;
import directors;
# Default backend definition. Set this to point to your content server.
backend default {
.host = "127.0.0.1";
.port = "8080";
}
sub vcl_init {
new backs = directors.hash();
backs.add_backend(default, 1);
}
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
set req.http.X-cookie = req.http.cookie;
if (!req.http.Cookie ~ "Logged-In") {
unset req.http.Cookie;
}
if (req.url ~ "\.(png|gif|jpg|png|css|js|html)$") {
unset req.http.Cookie;
}
}
sub vcl_backend_response {
# Happens after we have read the response headers from the backend.
#
# Here you clean the response headers, removing silly Set-Cookie headers
# and other mistakes your backend does.
}
sub vcl_deliver {
# Happens when we have all the pieces we need, and are about to send the
# response to the client.
#
# You can do accounting or modifying the final object here.
}
Even if it is not finish, I don't get how the Symfony default cache is still working, since I have disabled it in the configuration file.
Thank you for your help.
if you see something like
Via: "1.1 varnish-v4"
X-Varnish: "32808"
your varnish is working. Why disable the symfony cache though? You can use both... Might or might not make sense. That pretty much depends on the program logic.
If you want to gain more insight into your varnish during developement you can add the following lines:
sub vcl_deliver {
if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
# show how ofthe the object created a hit so far (reset on miss)
set resp.http.X-Cache-Hits = obj.hits;
}
I have a webpage hosted on Amazon S3 but I don't want the http response code to be 200. The page is a maintenance page that I'll redirect traffic to when I take our main website down for maintenance.
I want the Amazon S3 page to include a response header of:
HTTP/1.1 503 Service unavailable
Amazon give the ability to add some metadata to the S3 Object but there is nothing for the http status code.
Is it possible?
Not sure which browsers, or crawlers, that supports this. But you could potentially use the meta http-equiv status meta tag to accomplish this.
<meta http-equiv="status" content="503 Service Unavailable" />
The specification says to treat it in the same way as if 503 had been sent as the status code.
I believe you can get Cloudfront to do this. I haven't tested it yet, but try this:
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/custom-error-pages.html
You cannot customize the status code for S3 responses.
You can use API Gateway as a proxy to your S3 website error page where you can customize status codes returned.
Until Amazon allow a custom status code from S3, here is a workaround using nginx.
We watch for the existence of a specific file, that acts as a "ON switch" for maintenance mode. If found, we proxy_pass requests to S3 - The trick is to return 503 but redirect processing of 503 status codes to a nginx "named location".
Example nginx conf file (just the relevant bits shown):
server {
...
# Redirect processing of 503 status codes to a nginx "named location".
error_page 503 #maintenance;
# "Maintenance Mode" is off by default - Use a nginx variable to track state.
set $maintenance off;
# Switch on "Maintenance Mode" if a certain file exists.
if (-f /var/www/app/maintenanceON) {
set $maintenance on;
}
if ($maintenance = on) {
# For Maintenance mode Google recommend using status code: "503 Service unavailable".
return 503;
}
...
location #maintenance {
# Redirect the request to a static maintenance page hosted in Amazon S3.
# Note: Use proxy_pass instead of rewrite so we keep the 503 code (otherwise nginx serves a 302 code)
rewrite ^(.*)$ /index.html break;
proxy_pass http://bucketname.s3-website-us-east-1.amazonaws.com;
}
}
I'm running Magento Enterprise 1.9 with APC caching on Apache (Debian).
If I browse to a non-existent URL on my site, I receive a 404 response from the server and am presented with my 404 page as expected. However, if I then visit that same URL again, this time (and all subsequent times) I receive a "200 OK" status from the server, despite the URL still being invalid and the 404 page still being presented.
Examples:
$ curl -I http://www.example.com/some-nonexistent-URL
$ HTTP/1.1 404 Not Found
$ curl -I http://www.example.com/some-nonexistent-URL
$ HTTP/1.1 200 OK
I am assuming that this is because the resposne to that URL is being cached somehow so the server sends out a "200 OK" cached version of the 404 page (!).
How can I prevent this?
This was Magento's Full Page Cache fault.
I discovered Magento checks for the presence of a cookie called NO_CACHE, so I just put $_COOKE["NO_CACHE"] = true; at the point before the 404 action is called (in my case, /app/code/core/Mage/Cms/controllers/IndexController.php).
I also added a cache-control: no-cache, must revalidate header, and the same cookie as above into the 404 page template itself for good measure.
I've tried reading through many other questions on how to properly configure nginx to serve static assets for rails 3.2 but no matter what I attempt the asset path being loaded by my browser does not match the asset version specified in manifest.yml after precompile and as a result all my assets are not found.
My nginx config is as follows:
location ~ ^/assets/ {
# Per RFC2616 - 1 year maximum expiry
# http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
expires 1y;
add_header Cache-Control public;
gzip_static on;
# Some browsers still send conditional-GET requests if there's a
# Last-Modified header or an ETag header even if they haven't
# reached the expiry date sent in the Expires header.
add_header Last-Modified "";
add_header ETag "";
break;
}
I've also checked the root path in nginx is correct. I'm using nginx with unicorn via a Unix Domain Socket.
First time I set this up everything loaded fine. Then I modified an asset and re-deployed. That asset was then broken. I then bumped the asset version and now all assets are broken. I've tried clearing my local cache in case that was causing problems but it didn't help.
I'm starting to tear my hair out at this point, any help would be greatly appreciated.
After more investigation I found the unicorn process wasn't properly restarting when updating.
I have a rails 3.0 app, and it has a good amount of traffic, the app it's running through the combination of Nginx and Unicorn. The thing is that unicorn and it's workers consume a lot of resources, and because of the nature of my app, a lot of records are pulled from the database, then it's like serving almost static files generated with those database records
I was wondering if you can generate this kind of static files, cache them, serve them through nginx instead of the app through unicorn to use less resources and kind of reloading the cache after 1000 request
I'm beginning my research about that, I don't know a lot of server configuration, so I hope you guys have any advise for me, it would be great!
thanks!
I assume you mean How do I serve my static assets from nginx rather than Unicorn
I just solved this problem and here is a snippet of my nginx.conf
# Prefer to serve static files directly from nginx to avoid unnecessary
# data copies from the application server.
try_files $uri/index.html $uri.html $uri #app;
# Set Far Future Cache on Static Assets
# All requests starting with /xyz/ where xyz is
# one of the options below (~* == case insensitive)
location ~* ^/(images|javascripts|stylesheets)/ {
# Per RFC2616 - 1 year maximum expiry
# http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
expires 1y;
add_header Cache-Control public;
# Some browsers still send conditional-GET requests if there's a
# Last-Modified header or an ETag header even if they haven't
# reached the expiry date sent in the Expires header.
add_header Last-Modified "";
add_header ETag "";
break;
}
location #app { ... }
I am using Rails 3.0.10 so you my need something like ^/assets/ instead. The ~* directive tells nginx to do a case in-sensitive reg-ex compare. Also you don't need to escape the backslashes as you would in other languages.
Here is the Nginx documentation on that: http://wiki.nginx.org/HttpCoreModule#location