gzip compression using varnish cache - reverse-proxy

Im trying to provide gzip compression using varnish cache. But when I set content-encoding as gzip using my below mentioned configuration for varnish (default.vcl). Browser failed to download those content for which i set content-encoding as gzipped.
Varnish configuration file:
backend default {
.host = "127.0.0.1";
.port = "9000";
}
backend socketIO {
.host = "127.0.0.1";
.port = "8083";
}
acl purge {
"127.0.0.1";
"192.168.15.0"/24;
}
sub vcl_fetch {
/* If the request is for pictures, javascript, css, etc */
if (req.url ~ "^/public/" || req.url ~ "\.js"){
unset req.http.cookie;
set beresp.http.Content-Encoding= "gzip";
set beresp.ttl = 86400s;
set beresp.http.Cache-Control = "public, max-age=3600";
/*set the expires time to response header*/
set beresp.http.expires=beresp.ttl;
/* marker for vcl_deliver to reset Age: */
set beresp.http.magicmarker = "1";
}
if (!beresp.cacheable) {
return (pass);
}
return (deliver);
}
sub vcl_deliver {
if (resp.http.magicmarker) {
/* Remove the magic marker */
unset resp.http.magicmarker;
/* By definition we have a fresh object */
set resp.http.age = "0";
}
if(obj.hits > 0) {
set resp.http.X-Varnish-Cache = "HIT";
}else {
set resp.http.X-Varnish-Cache = "MISS";
}
return (deliver);
}
sub vcl_recv {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For ", " client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# Pass requests that are not GET or HEAD
if (req.request != "GET" && req.request != "HEAD") {
return(pass);
}
#pipe websocket connections directly to Node.js
if (req.http.Upgrade ~ "(?i)websocket") {
set req.backend = socketIO;
return (pipe);
}
# Properly handle different encoding types
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|js|css)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}
# allow PURGE from localhost and 192.168.15...
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
return (lookup);
}
return (lookup);
}
sub vcl_hit {
if (req.request == "PURGE") {
purge_url(req.url);
error 200 "Purged.";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
purge_url(req.url);
error 200 "Purged.";
}
}
sub vcl_pipe {
if (req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
}
}
Response Header:
Cache-Control:public, max-age=3600
Connection:keep-alive
Content-Encoding:gzip
Content-Length:11520
Content-Type:application/javascript
Date:Fri, 06 Apr 2012 04:53:41 GMT
ETag:"1330493670000--987570445"
Last-Modified:Wed, 29 Feb 2012 05:34:30 GMT
Server:Play! Framework;1.2.x-localbuild;dev
Via:1.1 varnish
X-Varnish:118464579 118464571
X-Varnish-Cache:HIT
age:0
expires:86400.000
Any suggestion on how to fix it and how to provide gzip compression using varnish.

Varnish Cache 3.0 does most of the handling of Accept-Encoding automatically and you shouldn't mess with it.
Basically, if you want Varnish to compress an object just set beresp.do_gzip in vcl_fetch and it will compress it before storing it in cache. Uncompression happens automatically when needed.

The content-length seems to be wrong, try un-setting it. Also, why are you using the beresp instead of obj in vcl_fetch?
...
obj.http.Content-Encoding="gzip";
remove obj.http.Content-Length;
...

Check and see if you are with apache, and have mod_deflate enabled,
try disabling it.

Related

Redirect HTTPS on multidomain Varnish

i have got two domain based on same framework (magento2)
domain1.it
domain2.com
I would like to redirect them to their respective SSL version.
https://domain1.it
https://domain2.com
Domain 1 is correctly configured to redirect to HTTPS and my varnish Config file is:
sub vcl_recv {
if ( (req.http.host ~ "^(?i)www.domain1.it" || req.http.host ~ "^(?i)domain1.it") && req.http.X-Forwarded-Proto !~ "(?i)https") {
return (synth(750, ""));
}
sub vcl_synth {
if (resp.status == 750) {
set resp.status = 301;
set resp.http.Location = "https://domain1.it" + req.url;
return(deliver);
}
the problem is the synth always redirect to the same domain.
I should add an if condition where i could call a subroutines that redirect to https for domain2
For the love of everything that is good, please stop using otherworldly status codes, 301 and 302 are perfectly fine, clearer and save you a line.
I would advise against using x-forwarded-proto and use an SSL/TLS terminator that supports the PROXY protocol, but since this is what you have, here you go:
sub vcl_recv {
if (req.http.X-Forwarded-Proto !~ "https") {
set req.http.location = "https://" + req.http.host + req.url;
return(synth(301));
}
}
sub vcl_synth {
if (resp.status == 301 || resp.status == 302) {
set resp.http.location = req.http.location;
return (deliver);
}
}
relevant link: https://info.varnish-software.com/blog/rewriting-urls-with-varnish-redirection
Bitnami Engineer here. I just reviewed the Varnish documentation and found this:
sub vcl_recv {
if (client.ip != "127.0.0.1" && std.port(server.ip) == 80 && req.http.host ~ "^(?i)example.com") {
set req.http.x-redir = "https://" + req.http.host + req.url;
return(synth(850, "Moved permanently"));
}
}
sub vcl_synth {
if (resp.status == 850) {
set resp.http.Location = req.http.x-redir;
set resp.status = 302;
return (deliver);
}
}
This is useful when you want to redirect the clients to an SSL-version of your site. More info here:
https://varnish-cache.org/trac/wiki/VCLExampleRedirectInVCL

varnish or squid reverse proxy behind corp proxy

How to configure varnish(or if someone can hint me to squid im fine) to cache requests from backend, but connect to backend through http_proxy
So I try to:
backend default {
.host = "10.1.1.1";
.port = "8080";
}
backend corp_proxy {
.host = "proxy";
.port = "8080";
}
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
set req.backend_hint = corp_proxy;
set req.url ="http://" + req.http.host + req.url;
}
Varnish (or other web caching proxies) caches a request, based on its cache-related headers (like Cache-Control).
Unfortunately, many web applications don't set these headers correctly. So we should use a more aggressive approach to cache some well-known items, e.g., pictures, .js or .css files.
Moreover, this line set req.url ="http://" + req.http.host + req.url; is not required, because Varnish sends the request, as is, to your designated backend.
Here is my recommended config:
backend corp_proxy {
.host = "proxy";
.port = "8080";
}
sub vcl_recv {
// Determine backend
if ( req.http.host ~ ".example.com" ) {
set req.backend_hint = corp_proxy;
// Determine cacheable items
if( req.url ~ "\.(css|js|jpg|jpeg|png|gif|ico) {
unset req.http.Cookie;
unset req.http.Cache-Control;
}
}
}
sub vcl_backend_response {
if( bereq.http.host ~ ".example.com" ) {
if (bereq.url ~ "\.(css|jsjpg|jpeg|png|gif|ico)") {
set beresp.ttl = 20m; // I opt for 20 minutes of caching by your mileage may vary
}
}

Varnish randomly forwarding requests to backend

I run a varnish-3.0.5 instance in front of an Apache/2.4.10 server. Both on the same Ubuntu 12.04.4 machine. The Varnish server randomly forwards some requests to the backend server even it has the page in cache already.
When I do a simple request:
GET http://www.example.com/
Accept: */*
Accept-Encoding: gzip, deflate
Host: www.example.com
User-Agent: runscope/0.1
I sometimes receive a correct response from cache:
Accept-Ranges: bytes
Age: 5973
Cache-Control: private, no-cache, no-store, must-revalidate
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 10015
Content-Type: text/html; charset=utf-8
Date: Fri, 24 Oct 2014 15:26:35 GMT
Expires: -1
Last-Modified: Fri, 24 Oct 2014 13:47:02 GMT
Server: Apache
Vary: Accept-Encoding
Via: 1.1 varnish
X-Varnish: 10531909 10507062
And sometimes (more often) a response directly from Apache:
Cache-Control: no-cache
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 10015
Content-Type: text/html; charset=utf-8
Date: Fri, 24 Oct 2014 15:28:00 GMT
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Last-Modified: Fri, 24 Oct 2014 15:28:00 GMT
Server: Apache
Set-Cookie: [truncated]
Vary: Accept-Encoding
I'm sure the item is still in the cache, because after few responses directly from Apache I receive a response from Varnish where the Age header contains a higher number than the one from the previous answer from Varnish.
My VCL file:
# Default backend definition. Set this to point to your content server.
backend default {
.host = "127.0.0.1";
.port = "8080";
}
# List of IPs allowed to erase things from cache
acl purge {
"localhost";
"127.0.0.1"/8;
}
sub vcl_recv {
# Purge item from the cache
if (req.request == "PURGE") {
if (!client.ip ~ purge) {
error 405 "Not allowed.";
}
return (lookup);
}
# Setting Forwarding header
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
## Pipeline ajax requests except of requests for [truncated]
if (req.http.X-Requested-With == "XMLHttpRequest" && req.url !~ "^/[truncated]") {
return(pipe);
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.request != "GET" && req.request != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
## Remove GCLID param from googles adds
set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"
### always cache these items:
## JavaScript
if (req.request == "GET" && req.url ~ "\.(js)") {
return (lookup);
}
## images
if (req.request == "GET" && req.url ~ "\.(gif|jpg|jpeg|bmp|png|tiff|tif|ico|img|tga|wmf)$") {
return (lookup);
}
## various other content pages
if (req.request == "GET" && req.url ~ "\.(css|html)$") {
return (lookup);
}
## multimedia
if (req.request == "GET" && req.url ~ "\.(svg|swf|ico|mp3|mp4|m4a|ogg|mov|avi|wmv)$") {
return (lookup);
}
## xml
if (req.request == "GET" && req.url ~ "\.(xml)$") {
return (lookup);
}
## Do not cache POST requests
if (req.request == "POST") {
return (pipe);
}
## Do not cache the [truncated]
if (req.url ~ "^/[truncated]") {
return (pipe);
}
## Cache the [truncated]
if (req.url ~ "^/[truncated]") {
return (lookup);
}
## Cache the [truncated]
if (req.url ~ "^/[truncated]") {
return (lookup);
}
## Do not cache the [truncated]
if (req.url ~ "^/[truncated]") {
return (pipe);
}
## Exceptions for [truncated]
if (req.url ~ "^/[truncated]") {
return (pass);
}
## Exceptions for [truncated]
if (req.url ~ "^/[truncated]") {
return (pipe);
}
return (lookup);
}
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
## Make a request to check the logged in status of the current user
if (req.request == "GET" && req.url !~ "\.(js|css|html|gif|jpg|jpeg|bmp|png|tiff|tif|ico|img|tga|wmf|svg|swf|ico|mp3|mp4|m4a|ogg|mov|avi|wmv|xml)$" && req.http.cookie ~ "cache-logged-in=1") {
hash_data("logged-in");
}
return (hash);
}
#
sub vcl_hit {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
return (deliver);
}
#
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 200 "Purged.";
}
return (fetch);
}
#
sub vcl_fetch {
## If the request to the backend returns a code other than 200, restart the loop
## If the number of restarts reaches the value of the parameter max_restarts,
## the request will be error'ed. max_restarts defaults to 4. This prevents
## an eternal loop in the event that, e.g., the object does not exist at all.
## this rule also allows for 301's and 302's redirects...
## Unset cookies sent from the backend if they are not necessary
if (req.url ~ "[truncated]")) {
if (beresp.http.set-cookie !~ "cache-logged-in") {
unset beresp.http.set-cookie;
}
}
if (beresp.status != 200 && beresp.status != 403 && beresp.status != 404 && beresp.status != 301 && beresp.status != 302 && beresp.status != 303) {
return (restart);
}
# if I cant connect to the backend, ill set the grace period to be 7200 seconds to hold onto content
set beresp.ttl = 7200s;
set beresp.grace = 7200s;
# If the header X-nocache is present, do not cache the item
if (beresp.http.x-nocache) {
return (hit_for_pass);
}
if (beresp.status == 404) {
set beresp.ttl = 0s;
}
if (beresp.status >= 500) {
set beresp.ttl = 0s;
}
if (req.request == "GET" && req.url ~ "\.(gif|jpg|jpeg|bmp|png|tiff|tif|ico|img|tga|wmf)$") {
set beresp.ttl = 86400s;
}
else if (req.request == "GET" && req.url ~ "\.(css|html)$") {
## various other content pages
set beresp.ttl = 86400s;
}
else if (req.request == "GET" && req.url ~ "\.(js)$") {
set beresp.ttl = 86400s;
}
else if (req.request == "GET" && req.url ~ "\.(xml)$") {
set beresp.ttl = 86400s;
}
else if (req.request == "GET" && req.url ~ "\.(svg|swf|ico|mp3|mp4|m4a|ogg|mov|avi|wmv)$") {
## multimedia
set beresp.ttl = 86400s;
}
else {
set beresp.http.Expires = "-1";
set beresp.http.Cache-Control = "private, no-cache, no-store, must-revalidate";
}
return(deliver);
}
The requests forwarded to apache are not identical to the ones already in varnish, thats the simple explanation. As to how they differ there are multiple options for which you need to see the full requests, but could include:
- cookies (notice that the apache requests are setting cookies, so thats where i would start).
- difference in casing of the url (your hash is case sensitive).
unset req.http.cookie in vcl_recv.

Varnish not ignoring subdomain despite vcl rules

I am running a basic lamp server with apache on port 80, and varnish on port 81. I am attempting to exclude a subdomain of the primary site entirely, however I have had no luck in doing so thus far, and I'm not sure why.
As you can see below, I have a rule in place to A) skip logged in users on the subdomain, and B) skip the subdomain entirely. Neither of these seem to work however. Is there something wrong with my vcl configuration?
backend default {
.host = "my.server.ip.address";
.port = "80";
}
sub vcl_recv {
call identify_device;
# Allow the back-end to serve up stale content if it is responding slowly.
set req.grace = 2m;
# Always cache the following file types for all users.
if ( req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$" ) {
unset req.http.cookie;
}
# Don't serve cached pages to logged in users
if ( req.http.cookie ~ "wordpress_logged_in" || req.url ~ "vaultpress=true" ) {
return( pass );
}
#Lets skip the logged in users on subdomain too!
if ( req.http.cookie ~ "dmr_user" ) {
return (pass);
}
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pass);
}
#Following for WooCommerce and comments
if (req.url ~ "^/(cart|my-account|checkout|addons|wp-comments-post)") {
return (pass);
}
#Lets skip the logged in users on entries too!
if ( req.http.cookie ~ "dmr_user" ) {
return (pass);
}
if ( req.url ~ "\?add-to-cart=" ) {
return (pass);
}
# Drop any cookies sent to WordPress.
if ( ! ( req.url ~ "wp-(login|admin)" ) ) {
unset req.http.cookie;
}
}
sub vcl_fetch {
if (beresp.ttl < 180s) {
set beresp.ttl = 180s;
}
if (!(req.url ~ "wp-(login|admin)")) {
unset beresp.http.set-cookie;
}
}
sub vcl_hash {
hash_data(req.http.X-Device);
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}
You are only skipping the processing of the subdomain halfway into your handling, ie instructions are executed in order. Moving the skip domain check directly at the top of sub vcl_recv should ensure no other rules gets executed against requests to that subdomain.
Well guys, it turns out that what I needed was to use pipe instead of pass.
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pass);
}
is now
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pipe);
}
I also went ahead and moved it up to the top of the config. Altogether it works like a charm now. Thanks to everyone for their help!

varnish rails devise logout configuration

I am using varnish in front end as caching server and apache2 with passenger is running on back-end for my rails3 application.I am using devise gem for my authentication but i am not able to lougout.
I think varnish has a option to clear logged in user session or cookie.I want to know the varnish configuration so that i can logout.
Here is my varnish configuration :
backend default {
.host = "127.0.0.1";
.port = "8080";
}
sub vcl_recv {
if(req.url ~ "my" || req.request == "POST" || req.request == "PUT" || req.request == "DELETE") {
return (pass);
}
return (lookup);
}
sub vcl_fetch {
if(req.url ~ "logout" || req.url ~ "sign_out"){
unset beresp.http.Set-Cookie;
}
if (req.request == "GET") {
unset beresp.http.Set-Cookie;
set beresp.cacheable = true;
set beresp.ttl = 360m;
}
if (req.url ~ "images/" || req.url ~ "javascripts" || req.url ~ "stylesheets"){
set beresp.ttl = 360m;
}
}
I suppose that you have to include the logout urls in tou recv function because otherwise you are not telling rails to kill the session.
i would try with something like:
sub vcl_recv {
if(req.url ~ "logout" ||req.url ~ "my" || req.request == "POST" || req.request == "PUT" || req.request == "DELETE") {
return (pass);
}
return (lookup);
}