Understanding reasons why Varnish isn't delivering cache'd version [closed] - apache

Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 19 days ago.
Improve this question
I'm new to Varnish, and have recently set it up on my server—running MediaWiki—but I'm a little confused when trying to debug why it isn't delivering pages from the cache. I'm also using Ezoic Ads, which is notorious for delivering cookies and stuff like that.
I'm 90% sure that Ezoic is the reason pages aren't being delivered from the cache, as when I disable it, I am (or at least I'm sure I am) served a cache'd version of a page from the server. I'm just not entirely sure what Ezoic is doing that is stopping cache'd versions being served.
I'm using this as my default.vcl which is the same used by Wikipedia and other MediaWiki sites that use Varnish. I'm pretty sure from this I can gather that the settings tell Varnish to ignore all cookies unless they're session or token cookies, so I've got an idea that the cookies aren't the issue, but I could be wrong.
I've ran varnishlog -g request -q "(VCL_call eq 'MISS' or VCL_call eq 'PASS') and ReqUrl ~ '^/wiki/'" to see a log of why some pages are being passed back to the webserver instead of being served by Varnish, and this pastebin is an example of one of the requests.
Which part of the output is the reason that the request was passed back to the webserver? I can't seem to find any documentation that explains exactly where in the log to look for the reasoning as to why the request is being passed.

Unfortunately you only shared the BeReq backend log transaction. The actual decision-making is done in a Req client request log transaction.
It would be helpful to add this to https://pastebin.com/4VAJ8cex, however the BeReq already indicates some cookie-related issues.
The cookie header that is sent to the backend still contains a lot of tracking cookies, as you can see below:
-- BereqHeader Cookie: _ga=GA1.1.2033892632.1674510549; __gads=ID=6956ea7a576443c1-226b89437bdb000b:T=1674510549:S=ALNI_MbFWR631GOAzUPplF2CQE_vU79FlA; ezosuibasgeneris-1=ca67b864-64e7-47a2-622a-58234d258f12; ezCMPCCS=false; _pk_id.52.ebfe=890ac89b1e44b8b0.1674510619.;
This is the VCL code you use to handle cookies:
if (req.http.Authorization || req.http.Cookie ~ "([sS]ession|Token)=") {
return (pass);
}
So while the decision to cache or not to cache is not impacted by the VCL code, these unsanitized tracking cookies still affect the way the objects are stored in the cache.
Cache variations
And this is all related to the following header that the application is returning:
Vary: Accept-Encoding,DNT,Cookie
This Vary header creates a cache variations for each value of the request header that is mentioned.
Accept-Encoding creates separate cached objects per URL for plain text responses and for Gzip encoded responses
DNT creates a variation for every value that request header has. However, the header doesn't seem to appear in the request
Cookie creates a variation for each value of the cookie header
The fact that Cookie is varied on, will result in a lot of versions of each page, because the tracking cookies often get different values.
The solution
My advice would be to modify the Vary header and remove the Cookie value from it and maybe also the DNT value. As matter of fact, you can remove the Vary header completely and rely on Varnish to send the proper Vary: Accept-Encoding.
If you don't know how to configure this in your application or web server, you can also strip it in VCL:
sub vcl_backend_response {
if(beresp.http.Vary ~ "Cookie") {
unset beresp.http.Vary;
}
}
This vcl_backend_response subroutine can be added before the one in your current VCL template.
An alternative solution
If removing the Vary header is hard or causes unwanted side effects, you can also sanitize to cookies and strip off the tracking cookies.
See https://www.varnish-software.com/developers/tutorials/removing-cookies-varnish for an official tutorial on how to remove cookies.
However, in your case, this would be the VCL code:
sub vcl_recv {
if (req.http.Cookie) {
set req.http.Cookie = ";" + req.http.Cookie;
set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
set req.http.Cookie = regsuball(req.http.Cookie, ";([sS]ession|Token)=", "; \1=");
set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
if (req.http.cookie ~ "^\s*$") {
unset req.http.cookie;
}
}
}

Related

I want to add the ID in the API [duplicate]

I am working on an app using Vue js.
According to my setting I need to pass to a variable to my URL when setting change.
<!-- language: lang-js -->
$.get('http://172.16.1.157:8002/firstcolumn/' + c1v + '/' + c1b, function (data) {
// some code...
});
But when my app hit on URL, it shows the following message.
Failed to load http://172.16.1.157:8002/firstcolumn/2017-03-01/2017-10-26: Redirect from 'http://172.16.1.157:8002/firstcolumn/2017-03-01/2017-10-26' to 'http://172.16.1.157:8002/firstcolumn/2017-03-01/2017-10-26/' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8080' is therefore not allowed access.
In addition to what awd mentioned about getting the person responsible for the server to reconfigure (an impractical solution for local development) I use a change-origin chrome plugin like this:
Moesif Orign & CORS Changer (use to be free but now wants a work email address >_>)
Allow CORS: Access-Control-Allow-Origin
You can make your local dev server (ex: localhost:8080) to appear to be coming from 172.16.1.157:8002 or any other domain.
In case the 2nd plugin link breaks in the future or the plugin writer decides to capitalize off the fame of this thread, open your browser's
plugin marketplace and search "allow cors", there's going to be a
bunch of them.
Thanks all, I solved by this extension on chrome.
Allow CORS: Access-Control-Allow-Origin
If you have control over your server, you can use PHP:
<?PHP
header('Access-Control-Allow-Origin: *');
?>
Ask the person maintaining the server at http://172.16.1.157:8002/ to add your hostname to Access-Control-Allow-Origin hosts, the server should return a header similar to the following with the response-
Access-Control-Allow-Origin: yourhostname:port
Using npm:
To allow cross-origin requests install 'cors':
npm i cors
Add this in the server-side:
let cors = require("cors");
app.use(cors());
When you have this problem with Chrome, you don't need an Extension.
Start Chrome from the Console:
chrome.exe --user-data-dir="C:/Chrome dev session" --disable-web-security
Maybe you have to close all Tabs in Chrome and restart it.
I will assume that you're a front-end developer only and that you don't have access to the backend of the application (regarding the tags of the question).
Short answer on how to properly solve this in your case? You can't, you'll need somebody else.
What is this about?
You need to understand that CORS is a security thing, it's not just here to annoy you just for fun.
It's purpose is to mainly prevent the usage of a (malicious) HTTP call from a non-whitelisted frontend to your backend with some critical mutation.
You could give a look to this YouTube video or any other one really, but I recommend a visual video because text-based explanation can be quite hard to understand.
You also need to understand that if you use Postman or any other tool to try your API call, you will not get the CORS issue. The reason being that those tools are not Web frontends but rather some server-based tools.
Hence, don't be surprised if something is working there but not in your Vue app, the context is different.
Now, how to solve this?
Depending of the framework used by your backend team, the syntax may be quite different but overall, you'll need to tell them to provide something like Access-Control-Allow-Origin: http://localhost:3000 (or any other port you'll be using).
PS: Using Access-Control-Allow-Origin: * would be quite risky because it would allow anybody to access it, hence why a stricter rule is recommended.
If you're using a service, like an API to send SMS, payment, some Google console or something else really, you'll need to allow your localhost in the dashboard of the service. Ask for credentials to your manager or Tech Lead.
If you have access to the backend, you could it yourself as shown here (ExpressJS in this example): https://flaviocopes.com/cors/
How to hack it in a dirty way?
If you're in a damn hurry and want to get something really dirty, you could use a lot of various hacks a listed in the other answers, here's a quick list:
use any extension who is able to create a middleware and forward the request to the backend (it will work because it's not directly coming from your frontend)
force your browser to disable CORS, not sure how this would actually solve the issue
use a proxy, if you're using Nuxt2, #nuxtjs/proxy is a popular one but any kind of proxy (even a real backend will do the job)
any other hack related somehow to the 3 listed above...
At the end, solving the CORS issue can be done quite fast and easily. You only need to communicate with your team or find something on your side (if you have access to the backend/admin dashboard of some service).
I heavily do recommend trying get it right from the beginning because it's related to security and that it may be forgotten down the road...
The approved answer to this question is not valid.
You need to set headers on your server-side code
app.use((req,res,next)=>{
res.setHeader('Access-Control-Allow-Origin','*');
res.setHeader('Access-Control-Allow-Methods','GET,POST,PUT,PATCH,DELETE');
res.setHeader('Access-Control-Allow-Methods','Content-Type','Authorization');
next();
})
You can also try a chrome extension to add these headers automatically.
Hello If I understood it right you are doing an XMLHttpRequest to a different domain than your page is on. So the browser is blocking it as it usually allows a request in the same origin for security reasons. You need to do something different when you want to do a cross-domain request. A tutorial about how to achieve that is Using CORS.
When you are using postman they are not restricted by this policy. Quoted from Cross-Origin XMLHttpRequest:
Regular web pages can use the XMLHttpRequest object to send and receive data from remote servers, but they're limited by the same origin policy. Extensions aren't so limited. An extension can talk to remote servers outside of its origin, as long as it first requests cross-origin permissions.
To add the CORS authorization to the header using Apache, simply add the following line inside either the <Directory>, <Location>, <Files> or <VirtualHost> sections of your server config (usually located in a *.conf file, such as httpd.conf or apache.conf), or within a .htaccess file:
Header set Access-Control-Allow-Origin "*"
And then restart apache.
Altering headers requires the use of mod_headers. Mod_headers is enabled by default in Apache, however, you may want to ensure it's enabled.
I had the same problem in my Vue.js and SpringBoot projects. If somebody work with spring you can add this code:
#Bean
public FilterRegistrationBean simpleCorsFilter() {
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
CorsConfiguration config = new CorsConfiguration();
config.setAllowCredentials(true);
// *** URL below needs to match the Vue client URL and port ***
config.setAllowedOrigins(Collections.singletonList("http://localhost:8080"));
config.setAllowedMethods(Collections.singletonList("*"));
config.setAllowedHeaders(Collections.singletonList("*"));
source.registerCorsConfiguration("/**", config);
FilterRegistrationBean bean = new FilterRegistrationBean<>(new CorsFilter(source));
bean.setOrder(Ordered.HIGHEST_PRECEDENCE);
return bean;
}
I found solution in this article Build a Simple CRUD App with Spring Boot and Vue.js
You are making a request to external domain 172.16.1.157:8002/ from your local development server that is why it is giving cross origin exception.
Either you have to allow headers Access-Control-Allow-Origin:* in both frontend and backend or alternatively use this extension cors header toggle - chrome extension unless you host backend and frontend on the same domain.
Try running this command in your terminal and then test it again.
curl -H "origin: originHost" -v "RequestedResource"
Eg:
If my originHost equals https://localhost:8081/ and my RequestedResource equals https://example.com/
My command would be as below:
curl -H "origin: https://localhost:8081/" -v "https://example.com/"
If you can notice the following line then it should work for you.
< access-control-allow-origin: *
Hope this helps.
Do specify #CrossOrigin(origins = "http://localhost:8081")
in Controller class.
You can solve this temporarily by using the Firefox add-on, CORS Everywhere. Just open Firefox, press Ctrl+Shift+A , search the add-on and add it!
You won't believe this,
Make sure to add "." at the end of the "url"
I got a similar error with this code:
fetch(https://itunes.apple.com/search?term=jack+johnson)
.then( response => {
return response.json();
})
.then(data => {
console.log(data.results);
}).catch(error => console.log('Request failed:', error))
The error I got:
Access to fetch at 'https://itunes.apple.com/search?term=jack+johnson'
from origin 'http://127.0.0.1:5500' has been blocked by CORS policy:
No 'Access-Control-Allow-Origin' header is present on the requested
resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
But I realized after a lot of research that the problem was that I did not copy the
right URL address from the iTunes API documentation.
It should have been
https://itunes.apple.com/search?term=jack+johnson.
not
https://itunes.apple.com/search?term=jack+johnson
Notice the dot at the end
There is a huge explanation about why the dot is important quoting issues about DNS and character encoding but the truth is you probably do not care. Try adding the dot it might work for you too.
When I added the "." everything worked like a charm.
I hope it works for you too.
install:
npm i cors
Then include cors():
app.get("/list",cors(),(req,res) =>{
});
In addition to the Berke Kaan Cetinkaya's answer.
If you have control over your server, you can do the following in ExpressJs:
app.use(function(req, res, next) {
// update to match the domain you will make the request from
res.header("Access-Control-Allow-Origin", "YOUR-DOMAIN.TLD");
res.header("Access-Control-Allow-Methods", "GET,HEAD,OPTIONS,POST,PUT");
res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
next();
});
https://enable-cors.org/server_expressjs.html
I tried this code,and that works for me.You can see the documentation in this link
var io = require("socket.io")(http, {
cors: {
origin: "*",
methods: ["GET", "POST"]
}
})
The reason that I came across this error was that I hadn't updated the path for different environments.
you have to customize security for your browser or allow permission through customizing security. (it is impractical for your local testing)
to know more about please go through the link.
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
These errors may be caused due to follow reasons, ensure the following steps are followed. To connect the local host with the local virtual machine(host). Here, I'am connecting http://localhost:3001/ to the http://abc.test Steps to be followed:
1.We have to allow CORS, placing Access-Control-Allow-Origin: in header of request
may not work. Install a google extension which enables a CORS request.*
2.Make sure the credentials you provide in the request are valid.
3.Make sure the vagrant has been provisioned. Try vagrant up --provision this make the localhost connect to db of the homestead.
Try changing the content type of the header. header:{ 'Content-Type' : 'application/x-www-form-urlencoded; charset=UTF-8;application/json' }
this point is very important.
Another solution to this problem in a specific scenario :
If
AWS APIGW is your backend with authentication enabled and
authentication fails,
your browser may end up complaining about CORS even if CORS is enabled in APIGW. You also need to enable CORS for 4XX as follows
API:YourAPI > Resources > /YourResource > Actions > Enable CORS > Gateway Responses for yourAPI check Default 4XX
Authentication will still fail but it won't look like CORS is the root cause
$.get('https://172.16.1.157:8002/firstcolumn/' + c1v + '/' + c1b, function (data) {
// some code...
});
Just put "https" .

Confusion on the 'Access-Control-Allow-Origin' header with apache

Lets say I have my website named SiteA.com running on an Apache web server. I have defined the ff. below on my httpd.conf file:
Header set Access-Control-Allow-Origin "CustomBank.com"
Questions:
Does this mean only CustomBank.com can access my site (SiteA.com) directly? or does it mean only my site (SiteA.com) can access the CustomBank.com domain directly? I am confused if this setting is for inbound or outbound.
In reality I don't have any CORS requirement needed for my site, so I didn't implement the setting mentioned above, the one below shows up in my response header.
Access-Control-Allow-Origin: *
Penetration Testing team said this setting is overly permissive. Do I just need to remove it? if not what should I do?
It means javascript loaded from CustomBank.com can make requests to your site (the site whose configuration has changed) via XMLHTPRequest in the background.
Since XMLHTTPRequest will send a users existing session cookie with your site, malicious scripts could do all kinds of nefarious/misleading things on behalf of your user. That's why * is not normally a suitable fix.
The restrictions apply to other script-like invocations that are more esoteric that you can read about in the specs.

Apache-2.2 Set-Cookie on logic from a response header

I need to set a cookie based on a response header (as opposed to a request header). The response header is set by a SOAP call to a backend - and is out of apaches control.
I've looked into SetEnvIf, but it states that it investigate request headers only. mod_rewrite's {HTTP:parm} construct also seems to apply to request headers only.
Request coming in
Response header is generated by backend
Apache investigates respond header FooBar
Apache add Set-Cookie if the respond header FooBar value matches "string"
Any ideas out there?
It looks like this can be done with mod_headers, but unfortunately only with Apache 2.4, since expressions were only added in 2.4. You would do something like:
Header set Set-Cookie "cookie-contents-here" "expr=%{resp:Content-Type} =~ m|application/pdf|"
If you can't upgrade to 2.4, you might consider putting Varnish Cache in front of your Apache install. It's a powerful HTTP processor and can easily handle modifying the response for you. You could also implement caching with it and increase the performance of your site, but it can just be used as a pass-through HTTP processor if you don't want to do that. Perhaps there's a simpler solution but that would work.
Another option could be to put a layer in between Apache and your back-end, such as a PHP script, that handles passing the call to the back-end and modifying the headers on the way back out. Probably not great for performance though; upgrading Apache or implementing Varnish Cache would be better.
If you're using a separate back-end out of Apache's control, then you might take Apache out of the loop completely and go straight from Varnish Cache to your back-end.
Hope the ideas help.

Can Apache/nginx gzip server response if it's already chunked?

I have a server REST API that answer some JSON response. I want to chunk it on the server to increase response time.
Is there a way for a reverse proxy like Apache or Nginx or any other, to intercept this response, and gzip the chunks, and send it back to the client as chunked?
I got something working by gzipping the content before chunking it directly inside my API server, and I'm just wondering if there's any other option available to me that would increase response time of my server.
I think that this is possible according to some other stack overflow questions that I have seen answered.
https://serverfault.com/questions/159313/enabling-nginx-chunked-transfer-encoding/187573#187573
According to the above, it is possible to disable proxy_buffering in your nginx configuration, and supports gzipping output if configured.
As noted in the page, there are possible disadvantages and you should test to ensure that this action is appropriate.

Where should I set HTTP headers, such as Expires?

I want to deploy an app using Sinatra on Phusion Passenger w/ nginx. If I want to set the Expires header on my static content - stylesheets, say - there are appear to be three places where I could accomplish this.
In my Sinatra app, using the API
With Rack middleware
In the server config for my deployment
Which of these methods is the best place for setting HTTP headers?
After talking though and answering this question and seeing the comment above, I think I have figured out the answer to my own question.
The whole point of nginx actually removes the first two options.
That leads to Option #3. This is where all the other content config is set, such as gzip compression.