When a server gives Cache-Control: max-age=4320000,
Is the freshness considered 4320000 seconds after the time of request, or after the last modified date?
RFC 2616 section 14.9.3:
When the max-age
cache-control directive is present in a cached response, the response
is stale if its current age is greater than the age value given (in
seconds) at the time of a new request for that resource. The max-age
directive on a response implies that the response is cacheable (i.e.,
"public") unless some other, more restrictive cache directive is also
present.
It is always based on the time of request, not the last modified date. You can confirm this behavior by testing on the major browsers.
tl;dr: the age of a cached object is either the time it was stored by any cache or now() - "Date" response header, whichever is bigger.
Full response:
The accepted response is incorrect. The mentioned rfc 2616 states on section 13.2.4 that:
In order to decide whether a response is fresh or stale, we need to compare its freshness lifetime to its age. The age is calculated as described in section 13.2.3.
And on section 13.2.3 it is state that:
corrected_received_age = max(now - date_value, age_value)
date_value is the response header Date:
HTTP/1.1 requires origin servers to send a Date header, if possible, with every response, giving the time at which the response was generated [...] We use the term "date_value" to denote the value of the Date header.
age_value is for how long the item is stored on any cache:
In essence, the Age value is the sum of the time that the response has been resident in each of the caches along the path from the origin server, plus the amount of time it has been in transit along network paths.
This is why good cache providers will include a header called Age every time they cache an item, to tell any upstream caches for how long they cached the item. If an upstream cache decides to store that item, its age must start with that value.
A practical example: a item is stored on the cache. It was stored 5 days ago, and when this item was fetched, the response headers included:
Date: Sat, 1 Jan 2022 11:05:05 GMT
Cache-Control: max-age={30 days in seconds}
Age: {10 days in seconds}
Assuming now() is Feb 3 2022, the age of the item must be calculated like (rounding up a bit for clarity):
age_value=10 days + 5 days (age when received + age on this cache)
now - date_value = Feb 3 2022 - 1 Jan 2022 = 34 days
The corrected age is the biggest value, that is 34 days. That means that the item is expired and can't be used, since max-age is 30 days.
The RFC presents a tiny additional correction that compensates for the request latency (see section 3, "corrected_initial_age").
Unfortunately not all cache servers will include the "Age" response header, so it is very important to make sure all responses that use max-age also include the "date" header, allowing the age to always be calculated.
Related
I went to GitHub issues to raise a support ticket but thought of asking the question first to avoid noise.
This is what the docs says-
Omit the version completely or use "latest" to load the latest one (not recommended for production usage):
/npm/jquery#latest/dist/jquery.min.js
/npm/jquery/dist/jquery.min.js
According to the doc, either we can latest or omit it completely to load the latest version. But I'm seeing a difference-
With latest added (URL 1 - U1)
Example- https://cdn.jsdelivr.net/npm/#letscooee/web-sdk#latest/dist/sdk.min.js
It loads the last released version that is cached for 24 hours. That means if we release v2 & v3 within 24 hours, the above URL will still show v1.
The caching period is 1 week.
Without latest (URL 2 - U2)
Example- https://cdn.jsdelivr.net/npm/#letscooee/web-sdk/dist/sdk.min.js
While we omit the latest completely, this loads the latest release immediately i.e. v3 and the caching period is also 1 week.
I have requested for the purge API as per their docs but I believe this behaviour is not aligning with their docs.
Tried to Google the cause and read their docs 3 times. Am I missing something?
Edit 1
After reading Martin's answer, I did the following-
(To view the images, open them in new tab & remove t before .png)
Step Taken
# Time
U1
U2
Purge Cache
12:39:00 UTC
Purged
Purged
See Age Header
# 12:40 UTC
0
0
See Date Header
# 12:40 UTC
Sun, 12 Sep 2021 12:40:25 GMT
Sun, 12 Sep 2021 12:40:31 GMT
Headers
12:41:00 UTC
Result
12:41:00 UTC
Points to latest release 0.0.3
Points to latest release 0.0.3
Publish new NPM release 0.0.4
12:48:00 UTC
Refresh both the URLs
12:49:00 UTC
Shows old release 0.0.3
Shows latest release 0.0.4
The last step shows that I was wrong here. This is working as expected (i.e. showing 0.0.3 only) as per the docs
The caching time is the same in both cases - 12 hours at the CDN level and 7 days in the browser: cache-control: public, max-age=604800, s-maxage=43200
That doesn't necessarily mean both URLs will always return the same content because both the CDN and your browser calculate the expiration for each URL independently, based on when it was first retrieved, so the CDN may serve different versions for up to 12 hours after the release.
Seems to me both the links point to the same sdk URL.
As per how cdns work would be to mention the version of the sdk for example:
<script src="https://unpkg.com/three#0.126.0/examples/js/loaders/GLTFLoader.js"></script>
or as per below which will always point to the latest version of the sdk:
<script src="https://cdn.rawgit.com/mrdoob/three.js/master/examples/js/loaders/GLTFLoader.js"></script>
I have a very static site, basically HTML and some Javascript on S3. I serve this through Cloudfront. My usage has gone up a bit plus one of my Javascript files is pretty large.
So what can I do to cut down the costs of serving those files? they need have very good uptime as it has thousands of active users all over the world.
This is the usage for yesterday:
Looking at other questions about this it seems like changing headers can help but I thought I already had caching enabled. This is what curl returns if I get one of those files:
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< content-type: text/html
< content-length: 2246
< date: Fri, 03 Apr 2020 20:28:47 GMT
< last-modified: Fri, 03 Apr 2020 15:21:11 GMT
< x-amz-version-id: some string
< etag: "83df2032241b5be7b4c337f0857095fc"
< server: AmazonS3
< x-cache: Miss from cloudfront
< via: 1.1 somestring.cloudfront.net (CloudFront)
< x-amz-cf-pop: some string
< x-amz-cf-id: some string
This is what the cache is configured as on CloudFront:
This is what S3 says when I use curl to query the file:
< HTTP/1.1 200 OK
< x-amz-id-2: some string
< x-amz-request-id: some string
< Date: Fri, 03 Apr 2020 20:27:22 GMT
< x-amz-replication-status: COMPLETED
< Last-Modified: Fri, 03 Apr 2020 15:21:11 GMT
< ETag: "83df2032241b5be7b4c337f0857095fc"
< x-amz-version-id: some string
< Accept-Ranges: bytes
< Content-Type: text/html
< Content-Length: 2246
< Server: AmazonS3
So what can I do? I don't often update the files and when I do I don't mind if it takes a day or two for the change to propagate.
Thanks.
If your goal is to reduce CloudFront costs, then it's worth reviewing how it is charged:
Regional Data Transfer Out to Internet (per GB): From $0.085 to $0.170 (depending upon location of your users)
Regional Data Transfer Out to Origin (per GB): From $0.020 to $0.160 (data going back to your application)
Request Pricing for All HTTP Methods (per 10,000): From $0.0075 to $0.0090
Compare that to Amazon S3:
GET Requests: $0.0004 per 1000
Data Transfer: $0.09 per GB (Also applies for traffic coming from Amazon EC2 instances)
Therefore, some options for you to save money are:
Choose a lower Price Class that restricts which regions send traffic "out". For example, Price Class 100 only sends traffic from USA and Europe, which has lower Data Transfer costs. This will reduce Data Transfer costs for other locations, but will give them a lower quality of service (higher latency).
Stop using CloudFront and serve content directly from S3 and EC2. This will save a bit on requests (about half the price), but Data Transfer would be a similar cost to Price Class 100.
Increase the caching duration for your objects. However, the report is showing 99.9%+ hit rates, so this won't help much.
Configure the objects to persist longer in user's browsers so less requests are made. However, this only works for "repeat traffic" and might not help much. It depends on app usage. (I'm not familiar with this part. It might not work in conjunction with CloudFront. Hopefully other readers can comment.)
Typically, mosts costs are related to the volume of traffic. If you app is popular, those Data Transfer costs will go up.
Take a look at your bills and try to determine which component is leading to most of the costs. Then, it's a trade-off between service to your customers and costs to you. Changing the Price Class might be the best option for now.
I have an API where I have created a token and it has replied with.
expires_in=7776000
created_at=1463347242
expires_in seems to be seconds (also what the spec calls for) 7776000 / 60 / 60 / 24 = 90 days
However I have no idea how to even begin to decode the created_at response and the endpoint doesn't have any documentation on it.
The created_at value is just the unix timestamp in seconds the token was created at. In your case it is the RFC 2822 date Sunday, 15-May-16 21:20:42 UTC. For this field I couldn't find a specification, but it seems to be a common field to many implementations. With it you can calculate the absolute timestamp of expiration: expires_at = created_at + expires_in
I am attempting to write a document to RavenDB with an expiration 20 minutes in the future. I am not using the .NET client, just curl. My request looks like this:
PUT /databases/FRUPublic/docs/test/123 HTTP/1.1
Host: ravendev
Connection: close
Accept-encoding: gzip, deflate
Content-Type: application/json
Raven-Entity-Name: tests
Raven-Expiration-Date: 2012-07-31T22:23:00
Content-Length: 14
{"data":"foo"}
In the studio I see my document saved with Raven-Expiration-Date set exactly 20 minutes from Last-Modified, however, within 5 minutes the document is deleted.
I see this same behavior (deleted in 5 minutes) if I increase the expiration date. If I set an expiration date in the past the document deletes immediately.
I am using build 960. Any ideas about what I'm doing wrong?
I specified the time to 10 millionth of a second and now documents are being deleted just as I would expect.
For example:
Raven-Expiration-Date: 2012-07-31T22:23:00.0000000
The date have to be in UTC, and it looks like you are sending local time.
When using the Omniture Data Warehouse API Explorer ( https://developer.omniture.com/en_US/get-started/api-explorer#DataWarehouse.Request ), the following request provides an 'Date_Granularity is invalid response'. Does anyone have experience with this? The API documentation ( https://developer.omniture.com/en_US/documentation/data-warehouse/pdf ), states that the following values are acceptable: "none, hour, day, week, month, quarter, year."
{
"Breakdown_List":[
"evar14",
"ip",
"evar64",
"evar65",
"prop63",
"evar6",
"evar16"
],
"Contact_Name":"[hidden]",
"Contact_Phone":"[hidden]",
"Date_From":"12/01/11",
"Date_To":"12/14/11",
"Date_Type":"range",
"Email_Subject":"[hidden]",
"Email_To":"[hidden]",
"FTP_Dir":"/",
"FTP_Host":"[hidden]",
"FTP_Password":"[hidden]",
"FTP_Port":"21",
"FTP_UserName":"[hidden]",
"File_Name":"test-report",
"Metric_List":[ ],
"Report_Name":"test-report",
"rsid":"[hidden]",
"Date_Granularity":"hour",
}
Response:
{
"errors":[
"Date_Granularity is invalid."
]
}
Old question, just noticing it now.
Data Warehouse did not support the Hour granularity correctly until Jan 2013 (the error you saw was a symptom of this). Then it was corrected for date ranges less then 14 days. In the July 2013 maintenance release of v15 the 14 day limit should be gone. But I have not verified that myself.
As always the more data you request the longer the DW processing will take. So I recommend keeping ranges to a maximum of a month and uncompressed file sizes to under a 1GB, though I hear 2 GB should now be supported.
If you still have issues please let us know.
Thanks C.