As part of a test application, I'm making HTTP requests to webpages. On receiving a response, I save the current date/time (GMT) and ETag header for subsequent requests. However, for some strange reason, some host servers are not validating the If-Modified-Since and If-None-Match headers on subsequent requests.
One such example is this webpage: www.foxsports.com/nba/cleveland-cavaliers-team-news (running Apache). It always returns the full body with a 200-Ok HTTP status, when a 304-NotModified status is expected, hence ignoring the If-Modified-Since and If-None-Match headers sent in the request. I tested it using curl and online Hurl.
Any ideas why the sent request headers are not validated by some host servers?
Related
I don't know if I am not correctly understanding how the caching aspect of ETags work if there is some other issue I am dealing with, but I'll walk you through my situation.
From my understanding ETags are a unique hash that is created based on the file information and they are sent as part of Response header to uniquely identify the file. If the file is updated then the info is changed and hence the ETag for the file is also changed.
In my project, I need a fresh JS file to be fetched everytime I make changes to the file. I can't use version tags or unique hashes as part of the file name. I thought an ETag would work where
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 200
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "a7-58c3bb52101c4"
......
myFile.js
Client <------------------ SERVER
// myFile.js has not been changed
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 304
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "a7-58c3bb52101c4"
......
Client uses cached version of file
// myFile has been changed
Http Request
GET myFile.js
Client ------------------> SERVER
Http Response 200
Http Response Header
accept-ranges: bytes
cache-control: max-age=86400, public
etag: "88-58c3a1cb8474f" // new etag generated
......
myFile.js
Client <------------------ SERVER
So, if you request the file again and no changes have been made..the etag will remain the same and you'll get a 304 will indicate that the cached version should be used.
If the file has been changed the etag will be different as well and a fresh copy of the file will be sent by the server.
This is how I expected it to work.
MY PROBLEM:
When I update myFile.js it seems like I never get the new ETag has back. It just defaults to the cahced version of the file. If I clear the cache then I get the latest file and the new ETag. This to me seems to defeat the purpose. Is this how it works or am I understanding something incorrectly here?
I am attempting to get some headers sent from my server to my front end via a fetch request.
In the controller function, I am explicitly sending some headers like this:
exports.getItems = async (ctx) => {
ctx.set('Search-type', 'category');
};
In postman, when I make a get request to my server I get these headers:
Connection →keep-alive
Content-Length →6442
Content-Type →application/json; charset=utf-8
Date →Thu, 19 Apr 2018 16:10:54 GMT
Search-type →category
However, when I try to access the header in the fetch request from the front end, I can only log the Content-Type. How do I get Search-type from my fetch?
After some googling, I found this issue on github which seems very similar to mine. This led me to another github issue page with the suggestion that I need to 'expose some explicitly needed headers'.
In the koa/cors documentation, there is an option allowHeaders Access-Control-Allow-Headers what I want to know is, how do I expose the headers so I can get them on my front end?
In the response to the GET, in addition to adding the Access-Control-Allow-Origin response header, you also need to include the Access-Control-Expose-Headers: <comma-separated-list-of-headers> response header.
If that header isn't returned by the server, even though the headers are sent by the server to the browser, the browser blocks any non-standard headers from being accessed by JavaScript. So you can see Content-Type (because it's a 'standard' response header), but not Search-type.
Basically, you need to ensure that the server responds with this
Access-Control-Expose-Headers: Search-type
(in addition to any other CORS response headers, like Access-Control-Allow-Origin, of course).
We have configured NTLM authentication using SSPI on apache due to which the authentication is three steps, where there are two 401 responses followed by 201/200 response.
Now in IE browser, this breaks because of - Why "Content-Length: 0" in POST requests?
Apache web server sends a 400 bad request response due to empty post request due to which POST on the server breaks.
How can I configure Apache to not treat this as 400 BAD request and process it normally?
I want to fragment one http requestto one or more http requests using apache proxy(i.e mod_proxy or hooking http get request message.
For example,
Client sends the http Get request with Content-Range : 0 - 4096bytes
Apache proxy server receives the this request message
Proxy sends the 2 http Get request messages with each Content-Range header
one HTTP Get Request message Content-Range : 0 - 2047
another HTTP Get Request message Content-Range : 2048 - 4096
Is it possibe to divide one request message to one or more request messages changing the content-range header value?
I don't believe there is any way with Apache to split a request on the byte level and then send part of it to one place and then part of it to another. Specifically because the breaking the request apart can well... break the request.
Also, the receiving end would now need to piece the split request back together or the proxy would have to do it.
Here's what happens on the local server when application invokes HTTP request on local IIS.
request.Credentials = CredentialCache.DefaultNetworkCredentials;
request.PreAuthenticate = true;
request.KeepAlive = true;
When I execute the request, I can see the following series of HTTP calls in Fiddler:
Request without authorization header, results in 401 with WWW-Authenticate NTLM+Negotiate
Request with Authorization: Negotiate (Base64 string 1), results in 401 with WWW-Authenticate: Negotiate (Base64 string 2)
Request with Authorization: Negotiate (Base64 string 3), results in 401 with WWW-Authenticate: Negotiate (Base64 string 4)
Request with Authorization: Negotiate (Base64 string 3), results in 401 with WWW-Authenticate NTLM+Negotiate
Apparently the client and the server (both running on the same machine) are trying to handshake, but in the end authorization fails.
What is strange is that if I disable Windows authentication of the site and enable Basic authentication and send user/pwd explicitly, it all works. It also works if I use NTLM authentication and try to access the site from the browser specifying my credentials.
Well, after several hours of struggling I figured what the problem was. In order to be able to inspect network traffic in Fiddler I defined a Fiddler rule:
if (oSession.HostnameIs("MYAPP")) { oSession.host = "127.0.0.1"; }
Then I used "MYAPP" instead of "localhost" in the Web app reference, and Fiddler happily displayed all session information.
But server security was far less happy, so this alias basically broke challenge-response authentication on the local server. Once I replaced the alias with "localhost", it all worked.