We have varnish configured as below:
we have probe validation with apache host and port, but using a context to a backend (application server / mod jk).
we are not using cluster and load balance configuration.
backend default {
.host = "127.0.0.1";
.port = "80";
.max_connections = 300;
.probe = {
.url = "/webapp-context/healthcheck";
.interval = 60s;
.timeout = 20s;
.window = 5;
.threshold = 3;
}
.first_byte_timeout = 5s;
.connect_timeout = 5s;
.between_bytes_timeout = 1s;
}
we have varnish cache only for specific contexts
we dont have varnish cache for staticfiles (www.domain.com/staticfiles/*), because all static files are on DocumentRoot (Apache).
sub vcl_recv {
// do not cache static files
if ( req.url ~ "^(/staticfiles)" ) {
return(pass);
}
// create cache
if ( req.url ~ "^(/content/)" ) {
unset req.http.Cookie;
return(hash);
}
...
...
}
So, my problem is: we have configured varnish to do "pass" for the static files context. And now, when our backend is sick after a probe validation, all staticfiles context is getting HTTP 503 error, but the html pages are still ok on Varnish cache, but without staticfiles.
Are there any way to configure Varnish to keep serving all static files from Apache, even that the application server is down?
You can setup additional backend definition that will not have health check specified. So your VCL will include something like this:
backend static {
.host = "127.0.0.1";
.port = "80";
.max_connections = 300;
}
# .. your default backend with probe here
sub vcl_recv {
# ...
// do not cache static files
if ( req.url ~ "^(/staticfiles)" ) {
set req.backend_hint = static;
return(pass);
}
# ,,,
}
Related
How to configure varnish(or if someone can hint me to squid im fine) to cache requests from backend, but connect to backend through http_proxy
So I try to:
backend default {
.host = "10.1.1.1";
.port = "8080";
}
backend corp_proxy {
.host = "proxy";
.port = "8080";
}
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
set req.backend_hint = corp_proxy;
set req.url ="http://" + req.http.host + req.url;
}
Varnish (or other web caching proxies) caches a request, based on its cache-related headers (like Cache-Control).
Unfortunately, many web applications don't set these headers correctly. So we should use a more aggressive approach to cache some well-known items, e.g., pictures, .js or .css files.
Moreover, this line set req.url ="http://" + req.http.host + req.url; is not required, because Varnish sends the request, as is, to your designated backend.
Here is my recommended config:
backend corp_proxy {
.host = "proxy";
.port = "8080";
}
sub vcl_recv {
// Determine backend
if ( req.http.host ~ ".example.com" ) {
set req.backend_hint = corp_proxy;
// Determine cacheable items
if( req.url ~ "\.(css|js|jpg|jpeg|png|gif|ico) {
unset req.http.Cookie;
unset req.http.Cache-Control;
}
}
}
sub vcl_backend_response {
if( bereq.http.host ~ ".example.com" ) {
if (bereq.url ~ "\.(css|jsjpg|jpeg|png|gif|ico)") {
set beresp.ttl = 20m; // I opt for 20 minutes of caching by your mileage may vary
}
}
i want to add an varnish proxy infront of my SX server i have two droplets ip:192.168.0.100 ip: 192.168.0.101 both works in lan but while restarting varnish shows a compile error
nano /etc/varnish/default.vcl
# define our first nginx server
backend nginx01 {
.host = "192.168.0.100";
.port = "80";
}
# define our second nginx server
backend nginx02 {
.host = "192.168.0.101";
.port = "80";
}
# configure the load balancer
director nginx round-robin {
{ .backend = nginx01; }
{ .backend = nginx02; }
}
# When a request is made set the backend to the round-robin director named nginx
sub vcl_recv {
set req.backend = nginx;
}
while restarting varnish shows an error
root#Sproxy:~# service varnish restart
Message from VCC-compiler:
directors are now in directors VMOD.
('input' Line 30 Pos 1)
director nginx round-robin {
########--------------------
Running VCC-compiler failed, exited with 2
VCL compilation failed
* Syntax check failed, not restarting
Despite the example given by #Redithion, the right configuration for Varnish 4.0 seems to be :
vcl 4.0;
import directors;
backend nginx01 {
.host = "192.168.0.100";
.port = "80";
}
backend nginx02 {
.host = "192.168.0.101";
.port = "80";
}
sub vcl_init {
new nginx = directors.round_robin();
nginx.add_backend(nginx01);
nginx.add_backend(nginx02);
}
sub vcl_recv {
set req.backend_hint = nginx.backend();
}
It seems you are using Varnish 4.0. In this version directors has been moved to VMODs
Here is an example based in what I got from here:
vcl 4.0;
# define first nginx server
backend nginx01 {
.host = "192.168.0.100";
.port = "80";
}
# define second nginx server
backend nginx02 {
.host = "192.168.0.101";
.port = "80";
}
sub vcl_init {
new cluster1 = directors.round_robin();
cluster1.add_backend(nginx01, 1.0);
cluster1.add_backend(nginx02, 1.0);
}
sub vcl_recv {
set req.backend_hint = cluster1.backend();
}
Tip: Don't forget the "vcl 4.0;" statement!
I am running a basic lamp server with apache on port 80, and varnish on port 81. I am attempting to exclude a subdomain of the primary site entirely, however I have had no luck in doing so thus far, and I'm not sure why.
As you can see below, I have a rule in place to A) skip logged in users on the subdomain, and B) skip the subdomain entirely. Neither of these seem to work however. Is there something wrong with my vcl configuration?
backend default {
.host = "my.server.ip.address";
.port = "80";
}
sub vcl_recv {
call identify_device;
# Allow the back-end to serve up stale content if it is responding slowly.
set req.grace = 2m;
# Always cache the following file types for all users.
if ( req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$" ) {
unset req.http.cookie;
}
# Don't serve cached pages to logged in users
if ( req.http.cookie ~ "wordpress_logged_in" || req.url ~ "vaultpress=true" ) {
return( pass );
}
#Lets skip the logged in users on subdomain too!
if ( req.http.cookie ~ "dmr_user" ) {
return (pass);
}
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pass);
}
#Following for WooCommerce and comments
if (req.url ~ "^/(cart|my-account|checkout|addons|wp-comments-post)") {
return (pass);
}
#Lets skip the logged in users on entries too!
if ( req.http.cookie ~ "dmr_user" ) {
return (pass);
}
if ( req.url ~ "\?add-to-cart=" ) {
return (pass);
}
# Drop any cookies sent to WordPress.
if ( ! ( req.url ~ "wp-(login|admin)" ) ) {
unset req.http.cookie;
}
}
sub vcl_fetch {
if (beresp.ttl < 180s) {
set beresp.ttl = 180s;
}
if (!(req.url ~ "wp-(login|admin)")) {
unset beresp.http.set-cookie;
}
}
sub vcl_hash {
hash_data(req.http.X-Device);
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}
You are only skipping the processing of the subdomain halfway into your handling, ie instructions are executed in order. Moving the skip domain check directly at the top of sub vcl_recv should ensure no other rules gets executed against requests to that subdomain.
Well guys, it turns out that what I needed was to use pipe instead of pass.
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pass);
}
is now
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pipe);
}
I also went ahead and moved it up to the top of the config. Altogether it works like a charm now. Thanks to everyone for their help!
I want to exclude a single virtual host from varnish 3 config
Cache: [www].domain.tld
Dont Cache: host.domain.tld
### default.vcl
backend foo { .host = "domain.tld"; .port = "8880"; }
backend bar { .host = "host.domain.tld"; .port = "8880";}
# ...
if (req.url == "host.domain.tld") { set req.backend = bar; }
if (req.url == "host.domain.tld") { return(pass); }
if (req.http.Cookie && req.url == "host.domain.tld") { return(pass); }
# ...
### httpd.conf
Listen 8880
<VirtualHost vhost.domain.tld:8880>
DocumentRoot /var/www/foo/
ServerName vhost.doman.tld
</VirtualHost>
The request never reaches the virtual host. I guess the problem is that I request on port 80 and the host listens on port 8880.
What I can do to fix this?
req.url doesn't contain the domain. You want something like this instead:
sub vcl_recv {
if (req.http.host == "host.domain.tld") {
set req.backend = bar;
return (pass);
}
}
This will set the backend for any request going to the domain "host.domain.tld" to "bar" and pass directly to the backend (bypassing the cache).
Im trying to provide gzip compression using varnish cache. But when I set content-encoding as gzip using my below mentioned configuration for varnish (default.vcl). Browser failed to download those content for which i set content-encoding as gzipped.
Varnish configuration file:
backend default {
.host = "127.0.0.1";
.port = "9000";
}
backend socketIO {
.host = "127.0.0.1";
.port = "8083";
}
acl purge {
"127.0.0.1";
"192.168.15.0"/24;
}
sub vcl_fetch {
/* If the request is for pictures, javascript, css, etc */
if (req.url ~ "^/public/" || req.url ~ "\.js"){
unset req.http.cookie;
set beresp.http.Content-Encoding= "gzip";
set beresp.ttl = 86400s;
set beresp.http.Cache-Control = "public, max-age=3600";
/*set the expires time to response header*/
set beresp.http.expires=beresp.ttl;
/* marker for vcl_deliver to reset Age: */
set beresp.http.magicmarker = "1";
}
if (!beresp.cacheable) {
return (pass);
}
return (deliver);
}
sub vcl_deliver {
if (resp.http.magicmarker) {
/* Remove the magic marker */
unset resp.http.magicmarker;
/* By definition we have a fresh object */
set resp.http.age = "0";
}
if(obj.hits > 0) {
set resp.http.X-Varnish-Cache = "HIT";
}else {
set resp.http.X-Varnish-Cache = "MISS";
}
return (deliver);
}
sub vcl_recv {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For ", " client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# Pass requests that are not GET or HEAD
if (req.request != "GET" && req.request != "HEAD") {
return(pass);
}
#pipe websocket connections directly to Node.js
if (req.http.Upgrade ~ "(?i)websocket") {
set req.backend = socketIO;
return (pipe);
}
# Properly handle different encoding types
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|js|css)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}
# allow PURGE from localhost and 192.168.15...
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
return (lookup);
}
return (lookup);
}
sub vcl_hit {
if (req.request == "PURGE") {
purge_url(req.url);
error 200 "Purged.";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
purge_url(req.url);
error 200 "Purged.";
}
}
sub vcl_pipe {
if (req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
}
}
Response Header:
Cache-Control:public, max-age=3600
Connection:keep-alive
Content-Encoding:gzip
Content-Length:11520
Content-Type:application/javascript
Date:Fri, 06 Apr 2012 04:53:41 GMT
ETag:"1330493670000--987570445"
Last-Modified:Wed, 29 Feb 2012 05:34:30 GMT
Server:Play! Framework;1.2.x-localbuild;dev
Via:1.1 varnish
X-Varnish:118464579 118464571
X-Varnish-Cache:HIT
age:0
expires:86400.000
Any suggestion on how to fix it and how to provide gzip compression using varnish.
Varnish Cache 3.0 does most of the handling of Accept-Encoding automatically and you shouldn't mess with it.
Basically, if you want Varnish to compress an object just set beresp.do_gzip in vcl_fetch and it will compress it before storing it in cache. Uncompression happens automatically when needed.
The content-length seems to be wrong, try un-setting it. Also, why are you using the beresp instead of obj in vcl_fetch?
...
obj.http.Content-Encoding="gzip";
remove obj.http.Content-Length;
...
Check and see if you are with apache, and have mod_deflate enabled,
try disabling it.