varnish or squid reverse proxy behind corp proxy - reverse-proxy

How to configure varnish(or if someone can hint me to squid im fine) to cache requests from backend, but connect to backend through http_proxy
So I try to:
backend default {
.host = "10.1.1.1";
.port = "8080";
}
backend corp_proxy {
.host = "proxy";
.port = "8080";
}
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
set req.backend_hint = corp_proxy;
set req.url ="http://" + req.http.host + req.url;
}

Varnish (or other web caching proxies) caches a request, based on its cache-related headers (like Cache-Control).
Unfortunately, many web applications don't set these headers correctly. So we should use a more aggressive approach to cache some well-known items, e.g., pictures, .js or .css files.
Moreover, this line set req.url ="http://" + req.http.host + req.url; is not required, because Varnish sends the request, as is, to your designated backend.
Here is my recommended config:
backend corp_proxy {
.host = "proxy";
.port = "8080";
}
sub vcl_recv {
// Determine backend
if ( req.http.host ~ ".example.com" ) {
set req.backend_hint = corp_proxy;
// Determine cacheable items
if( req.url ~ "\.(css|js|jpg|jpeg|png|gif|ico) {
unset req.http.Cookie;
unset req.http.Cache-Control;
}
}
}
sub vcl_backend_response {
if( bereq.http.host ~ ".example.com" ) {
if (bereq.url ~ "\.(css|jsjpg|jpeg|png|gif|ico)") {
set beresp.ttl = 20m; // I opt for 20 minutes of caching by your mileage may vary
}
}

Related

Redirect HTTPS on multidomain Varnish

i have got two domain based on same framework (magento2)
domain1.it
domain2.com
I would like to redirect them to their respective SSL version.
https://domain1.it
https://domain2.com
Domain 1 is correctly configured to redirect to HTTPS and my varnish Config file is:
sub vcl_recv {
if ( (req.http.host ~ "^(?i)www.domain1.it" || req.http.host ~ "^(?i)domain1.it") && req.http.X-Forwarded-Proto !~ "(?i)https") {
return (synth(750, ""));
}
sub vcl_synth {
if (resp.status == 750) {
set resp.status = 301;
set resp.http.Location = "https://domain1.it" + req.url;
return(deliver);
}
the problem is the synth always redirect to the same domain.
I should add an if condition where i could call a subroutines that redirect to https for domain2
For the love of everything that is good, please stop using otherworldly status codes, 301 and 302 are perfectly fine, clearer and save you a line.
I would advise against using x-forwarded-proto and use an SSL/TLS terminator that supports the PROXY protocol, but since this is what you have, here you go:
sub vcl_recv {
if (req.http.X-Forwarded-Proto !~ "https") {
set req.http.location = "https://" + req.http.host + req.url;
return(synth(301));
}
}
sub vcl_synth {
if (resp.status == 301 || resp.status == 302) {
set resp.http.location = req.http.location;
return (deliver);
}
}
relevant link: https://info.varnish-software.com/blog/rewriting-urls-with-varnish-redirection
Bitnami Engineer here. I just reviewed the Varnish documentation and found this:
sub vcl_recv {
if (client.ip != "127.0.0.1" && std.port(server.ip) == 80 && req.http.host ~ "^(?i)example.com") {
set req.http.x-redir = "https://" + req.http.host + req.url;
return(synth(850, "Moved permanently"));
}
}
sub vcl_synth {
if (resp.status == 850) {
set resp.http.Location = req.http.x-redir;
set resp.status = 302;
return (deliver);
}
}
This is useful when you want to redirect the clients to an SSL-version of your site. More info here:
https://varnish-cache.org/trac/wiki/VCLExampleRedirectInVCL

Varnish HTTP 503 - backend sick - apache static files not cached

We have varnish configured as below:
we have probe validation with apache host and port, but using a context to a backend (application server / mod jk).
we are not using cluster and load balance configuration.
backend default {
.host = "127.0.0.1";
.port = "80";
.max_connections = 300;
.probe = {
.url = "/webapp-context/healthcheck";
.interval = 60s;
.timeout = 20s;
.window = 5;
.threshold = 3;
}
.first_byte_timeout = 5s;
.connect_timeout = 5s;
.between_bytes_timeout = 1s;
}
we have varnish cache only for specific contexts
we dont have varnish cache for staticfiles (www.domain.com/staticfiles/*), because all static files are on DocumentRoot (Apache).
sub vcl_recv {
// do not cache static files
if ( req.url ~ "^(/staticfiles)" ) {
return(pass);
}
// create cache
if ( req.url ~ "^(/content/)" ) {
unset req.http.Cookie;
return(hash);
}
...
...
}
So, my problem is: we have configured varnish to do "pass" for the static files context. And now, when our backend is sick after a probe validation, all staticfiles context is getting HTTP 503 error, but the html pages are still ok on Varnish cache, but without staticfiles.
Are there any way to configure Varnish to keep serving all static files from Apache, even that the application server is down?
You can setup additional backend definition that will not have health check specified. So your VCL will include something like this:
backend static {
.host = "127.0.0.1";
.port = "80";
.max_connections = 300;
}
# .. your default backend with probe here
sub vcl_recv {
# ...
// do not cache static files
if ( req.url ~ "^(/staticfiles)" ) {
set req.backend_hint = static;
return(pass);
}
# ,,,
}

varnish backend VCC-compiler failed

i want to add an varnish proxy infront of my SX server i have two droplets ip:192.168.0.100 ip: 192.168.0.101 both works in lan but while restarting varnish shows a compile error
nano /etc/varnish/default.vcl
# define our first nginx server
backend nginx01 {
.host = "192.168.0.100";
.port = "80";
}
# define our second nginx server
backend nginx02 {
.host = "192.168.0.101";
.port = "80";
}
# configure the load balancer
director nginx round-robin {
{ .backend = nginx01; }
{ .backend = nginx02; }
}
# When a request is made set the backend to the round-robin director named nginx
sub vcl_recv {
set req.backend = nginx;
}
while restarting varnish shows an error
root#Sproxy:~# service varnish restart
Message from VCC-compiler:
directors are now in directors VMOD.
('input' Line 30 Pos 1)
director nginx round-robin {
########--------------------
Running VCC-compiler failed, exited with 2
VCL compilation failed
* Syntax check failed, not restarting
Despite the example given by #Redithion, the right configuration for Varnish 4.0 seems to be :
vcl 4.0;
import directors;
backend nginx01 {
.host = "192.168.0.100";
.port = "80";
}
backend nginx02 {
.host = "192.168.0.101";
.port = "80";
}
sub vcl_init {
new nginx = directors.round_robin();
nginx.add_backend(nginx01);
nginx.add_backend(nginx02);
}
sub vcl_recv {
set req.backend_hint = nginx.backend();
}
It seems you are using Varnish 4.0. In this version directors has been moved to VMODs
Here is an example based in what I got from here:
vcl 4.0;
# define first nginx server
backend nginx01 {
.host = "192.168.0.100";
.port = "80";
}
# define second nginx server
backend nginx02 {
.host = "192.168.0.101";
.port = "80";
}
sub vcl_init {
new cluster1 = directors.round_robin();
cluster1.add_backend(nginx01, 1.0);
cluster1.add_backend(nginx02, 1.0);
}
sub vcl_recv {
set req.backend_hint = cluster1.backend();
}
Tip: Don't forget the "vcl 4.0;" statement!

Varnish not ignoring subdomain despite vcl rules

I am running a basic lamp server with apache on port 80, and varnish on port 81. I am attempting to exclude a subdomain of the primary site entirely, however I have had no luck in doing so thus far, and I'm not sure why.
As you can see below, I have a rule in place to A) skip logged in users on the subdomain, and B) skip the subdomain entirely. Neither of these seem to work however. Is there something wrong with my vcl configuration?
backend default {
.host = "my.server.ip.address";
.port = "80";
}
sub vcl_recv {
call identify_device;
# Allow the back-end to serve up stale content if it is responding slowly.
set req.grace = 2m;
# Always cache the following file types for all users.
if ( req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$" ) {
unset req.http.cookie;
}
# Don't serve cached pages to logged in users
if ( req.http.cookie ~ "wordpress_logged_in" || req.url ~ "vaultpress=true" ) {
return( pass );
}
#Lets skip the logged in users on subdomain too!
if ( req.http.cookie ~ "dmr_user" ) {
return (pass);
}
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pass);
}
#Following for WooCommerce and comments
if (req.url ~ "^/(cart|my-account|checkout|addons|wp-comments-post)") {
return (pass);
}
#Lets skip the logged in users on entries too!
if ( req.http.cookie ~ "dmr_user" ) {
return (pass);
}
if ( req.url ~ "\?add-to-cart=" ) {
return (pass);
}
# Drop any cookies sent to WordPress.
if ( ! ( req.url ~ "wp-(login|admin)" ) ) {
unset req.http.cookie;
}
}
sub vcl_fetch {
if (beresp.ttl < 180s) {
set beresp.ttl = 180s;
}
if (!(req.url ~ "wp-(login|admin)")) {
unset beresp.http.set-cookie;
}
}
sub vcl_hash {
hash_data(req.http.X-Device);
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
} else {
set resp.http.X-Cache = "MISS";
}
}
You are only skipping the processing of the subdomain halfway into your handling, ie instructions are executed in order. Moving the skip domain check directly at the top of sub vcl_recv should ensure no other rules gets executed against requests to that subdomain.
Well guys, it turns out that what I needed was to use pipe instead of pass.
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pass);
}
is now
#skip subdomain.domain.com
if (req.http.host ~ "subdomain.domain.com") {
return (pipe);
}
I also went ahead and moved it up to the top of the config. Altogether it works like a charm now. Thanks to everyone for their help!

How to exclude virtual host from varnish 3?

I want to exclude a single virtual host from varnish 3 config
Cache: [www].domain.tld
Dont Cache: host.domain.tld
### default.vcl
backend foo { .host = "domain.tld"; .port = "8880"; }
backend bar { .host = "host.domain.tld"; .port = "8880";}
# ...
if (req.url == "host.domain.tld") { set req.backend = bar; }
if (req.url == "host.domain.tld") { return(pass); }
if (req.http.Cookie && req.url == "host.domain.tld") { return(pass); }
# ...
### httpd.conf
Listen 8880
<VirtualHost vhost.domain.tld:8880>
DocumentRoot /var/www/foo/
ServerName vhost.doman.tld
</VirtualHost>
The request never reaches the virtual host. I guess the problem is that I request on port 80 and the host listens on port 8880.
What I can do to fix this?
req.url doesn't contain the domain. You want something like this instead:
sub vcl_recv {
if (req.http.host == "host.domain.tld") {
set req.backend = bar;
return (pass);
}
}
This will set the backend for any request going to the domain "host.domain.tld" to "bar" and pass directly to the backend (bypassing the cache).