I'm using Fine Uploader with S3 and I have a client whose computer time is off, resulting in an S3 RequestTimeTooSkewed error. Ideally, my client would have the right time, but I'd like to have my app be robust to this situation.
I've seen this post - https://github.com/aws/aws-sdk-js/issues/399 on how to automatically retry the request. You take the ServerTime from the error response and use that as the time in the response. An alternative approach would just be to get the time from a reliable external source every time, avoiding the need for a retry. However, I'm not sure how to hook either approach into S3 Fine Uploader. Does anyone have an idea of how to do this?
A solution was provided in Fine Uploader 5.5 to address this very situation. From the S3 feature documentation:
If the clock on the machine running Fine Uploader is too far off of the current date, S3 may reject any requests sent from this machine. To overcome this situation, you can include a clock drift value, in milliseconds, when creating a new Fine Uploader instance. One way to set this value is to subtract the current time according to the browser from the current unix time according to your server. For example:
var uploader = new qq.s3.FineUploader({
request: {
clockDrift: SERVER_UNIX_TIME_IN_MS - Date.now()
}
})
If this value is non-zero, Fine Uploader S3 will use it to pad the x-amz-date header and the policy expiration date sent to S3.
Related
I've set the TTL max, min and default all to 0 (on the "Default Cache Behavior Settings" page), thinking this would mean that when I upload a new file called events.html to S3 it would replace the old events.html page, but I'm still seeing the cached version after a few hours.
I am just trying to update the content on some of my webpages.
If you want to invalidate your cache with new update in s3, you need to do it explicitly with putobject event. You can call the lambda to invalidate CF cache.
Here is example: https://blog.miguelangelnieto.net/posts/Automatic_Cloudfront_invalidation_with_Amazon_Lambda.html
Be aware that with above approach, if you refresh cache more than 1000 files in a month you have to pay extra invalidation fees. Refer CF pricing.
Also with TTL, you can do it but it will happen after the TTL value is elapsed and you have to clear the browser cache to view it.
i'm trying to get my google page speed insights rating to be decent, but there are some external files that i would want to be cached aswell, anyone knows what would be the best way to deal with this?
https://s.swiftypecdn.com/cc.js (5 minutes)
https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js (60 minutes)
https://pagead2.googlesyndication.com/pagead/osd.js (60 minutes)
https://www.google-analytics.com/plugins/ua/linkid.js (60 minutes)
https://hey.hellobar.com/…d5837892514411fd16abbb3f71f0d400607f8f0b (2 hours)
https://www.google-analytics.com/analytics.js (2 hours)
Copy to your server and serve locally or from CDN, with different browser cache settings. Update GA scripts periodically with cronjob or something similar.
On Wordpress there are plugins that can do that for you, like this one: Above The Fold; they call this feature Javascript localization.
On the other hand, I use Google Pagespeed Module on server and it's directive MapProxyDomain in combination with Alternative async tracking snippet. That seems most elegant for me.
This should be enough for you to start solving your problem.
set cache-control to external resources?
You can't control the headers sent from a server that you don't control.
In other words, either host a copy yourself or there's nothing you can do about it.
Thanks
There is no solution for those files. If those files are CDN like bootstrap cdn you can copy those files locally into your host but if those request are generated on runtime than you can do nothing about it. :)
You can make your own cache
Place some files to the browser's localStorage (after the first time they arrive from the distant server) and next time you can serve them from the local copy. This way, you store things right where they're needed - the only thing to be careful with is updating them, you need a way to replace these files when it's time for a new version.
If you don't want to do this from scratch, here are some Javascript libraries:
https://www.sitepoint.com/9-javascript-libraries-working-with-local-storage/
Check out this lsCache for example, it looks super practical:
lscache.set('greeting', 'Hello World!', 2); // 2 minute expiration
lscache.get('greeting'); // returns "Hello World!"
lscache.remove('greeting'); // remove
lscache.flush(); // flush the entire cache
lscache.flushExpired(); // flush only expired items
Working on an app for a client that will asynchronously receive a request, reply immediately, then go out and fetch a set of large files to perform work on, and finally upload the results to S3 minutes or hours later.
Can we know ahead of time what the eventual url of the file on S3 will be? I'm thinking of creating a hash based on the filename and some other metadata that we know at the incoming request initialization and using that as the name of the S3 file. Is there a predictable pattern of S3 host plus bucket plus file name, or is it something that we don't know until the file upload is complete?
I'm entertaining the idea of returning the eventual S3 filename to the initial request, with the expectation that on the client's end they can periodically check the url for the result. In addition, I'm considering requiring the client to pass a callback url in with the request. The app will then hit that url later with the success/fail status of the work.
Thanks.
The URL of a file uploaded to S3 can be entirely determined by you - it's purely dependent on the bucket and key name. Specifically, it's of the form:
http://s3.amazonaws.com/BUCKETNAME/KEYNAME
(Or certain other formats, depending. It's still completely predictable.)
So long as you pick the key name ahead of time, you'll know what the eventual URL will be.
I'm using signed POST to upload file directly to amazon S3. I had some trouble with the signature of the policy using PHP but finally fixed it and here is the sample of code.
This xhr request is send in javascript and I'm waiting for an answer from amazon. At first I was using success_action_status setting it to 201 to get the XML response.
What I'd like to do is using the success_action_redirect to call a script on my server to create a record in the database.
The reason why is that I could create the record in the database and if anything wrong happen at this stage I can return an error message directly at this point. Also it saves me another ajax request to my server.
So I've tried to set this up specifying the success_action_redirect to http:\\localhost\callback.php where I have a script that is waiting for some parameters.
But it looks like this script is never called and the response of the xhr.send() is empty.
I think it's a cross-browser issue and I'm wondering if it would be possible to use jsonp somehow to pass-by this?
Any ideas?
UPDATE
Apparently xhr is following redirect natively so it should work but when I specified the success_action_redirect it returns error Server responded with 0 code.
At first I thought it was because the redirect URL was on my local server so I've changed it to an accessible server but no chance.
Anyone knows why it's returning this error message?
I also run into this problem. It seems like nobody has a solution to this like this
maybe the best workaround i have found is something like this.
It seems thet the only workaround includes a second xhr-request to execute the callback manually. therefore the
success_action_status
should be used. Witht his you will get a 201 response if the upload was successful and you can start a second request for the actual callback. For me it looks like the only possible solution at the moment.
Any other solutions?
I know Amazon S3 added the multi-part upload for huge files. That's great. What I also need is a similar functionality on the client side for customers who get part way through downloading a gigabyte plus file and have errors.
I realize browsers have some level of retry and resume built in, but when you're talking about huge files I'd like to be able to pick up where they left off regardless of the type of error out.
Any ideas?
Thanks,
Brian
S3 supports the standard HTTP "Range" header if you want to build your own solution.
S3 Getting Objects
I use aria2c. For private content, you can use "GetPreSignedUrlRequest" to generate temporary private URLs that you can pass to aria2c
S3 has a feature called byte range fetches. It’s kind of the download compliment to multipart upload:
Using the Range HTTP header in a GET Object request, you can fetch a byte-range from an object, transferring only the specified portion. You can use concurrent connections to Amazon S3 to fetch different byte ranges from within the same object. This helps you achieve higher aggregate throughput versus a single whole-object request. Fetching smaller ranges of a large object also allows your application to improve retry times when requests are interrupted. For more information, see Getting Objects.
Typical sizes for byte-range requests are 8 MB or 16 MB. If objects are PUT using a multipart upload, it’s a good practice to GET them in the same part sizes (or at least aligned to part boundaries) for best performance. GET requests can directly address individual parts; for example, GET ?partNumber=N.
Source: https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html
Just updating for current situation, S3 natively supports multipart GET as well as PUT. https://youtu.be/uXHw0Xae2ww?t=1459.
NOTE: For Ruby user only
Try aws-sdk gem from Ruby, and download
object = AWS::S3::Object.new(...)
object.download_file('path/to/file.rb')
Because it download a large file with multipart by default.
Files larger than 5MB are downloaded using multipart method
http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Object.html#download_file-instance_method