I'm not sure how to word the question but here is what I am looking to do.
I have a site that uses custom map tile overlays on a google map.
The javascript calls a php file on my server that checks to see if an existing map tile exists for the given x, y, and zoom level.
If if exists, it displays that image using file_get_contents.
If it doesn't exist, it creates the new tile then displays it.
I would like to utilize Amazon S3 store and serve the images since there could end being a lot of them and my server is slow. If I have my script check to see if the image exists on amazon and then display it, I am guessing I am not getting the benefits of the speed and Amazons CDN. Is there a way to do this?
Or is there a way to try and pull the file from Amazon first then set up something on Amazon to redirect to my script if the files no there?
Maybe host the script on another of Amazons services? The tile generation is quite slow also in some cases.
Thanks
Ideas:
1 - Use CloudFront, but point it to a cluster of tile generation machines. This way, you can generate the tiles on demand, and any future requests are served right from Cloudfront.
2 - Use CloudFront, but back with with an S3 store of generated tiles. Turn on logging for the S3 bucket, so you can detect failed requests. Consume those logs on a schedule, and generate the missing tiles. This results in a cheaper way of generating tiles, but means that when a tile fails the user get's nothing.
3 - Just pre-generate all the tiles. Throw tasks in an SQS queue, then spin up a collection of EC2 instances to generate the tiles. This will cost the most up front, but all users get a fast experience.
I've written a blog post with a strategy for dealing with this. It's designed to make intelligent and thrifty use of CloudFront, maximize caching and deal with new versions of existing images. You may find the technique described there helpful. The example code shows how to handle different dimensions (i.e. thumbnails) of images. You could modify it to handle different zoom levels.
I need to update that post to support CloudFront custom origins, and I think that for your application you might be better off skipping S3 and using a custom origin. The advantage of a custom origin is simply that it's probably going to be easier to manage all of your images on your local filesystem compared to managing them on S3.
Related
At 11:50 in this video the presenter explains how to configure a Cloudfront behaviour to whitelist the CloudFront-Viewer-Country. While I have followed his instructions as closely as I can, this part of the video is the only one where I cannot be sure because he is using AWS's old console and I'm using the new design, which does not have the same terminology - and hence while his example clearly works, mine does not, leading me to suspect that this is the breaking point.
Old console with the CloudFront-Viewer-Country header whitelisted:
My console with what I hope is the same confguration:
How can I know if these show the same setup of behaviour?
When I say that mine does not work, I have 4 different S3 buckets with different images all named the same. These are served via CloudFront with a Lambda#Edge function derived from the example JS function to direct requests to the appropriate bucket content. What I expect is that viewing the image from the European region (via VPN change) I should see one image and when viewing from the US region I should see a different image. I do not.
For an app (web + phone) there are two options:
Image binaries in database. Server replies to app HTTP request with images as base64
Images in storage service like Amazon S3 or Azure Blob Storage or a self-hosted one. Image links in database. Server handles app HTTP requests by sending back only the links to images. The app fetches the images from storage by their link
Which option above is the standard practice? Which one has less trouble down the road?
To some extent, the answer to this question is always opinion based, and partly depends on the specific use case.
I would think that the second approach is used more often. One reason is that normally, storage within a database is slightly more expensive than file storage in many cases. Also, what is the real use case? Assuming you use HTML pages that reference images via the img element or via CSS as background image, then the base64 return value would not be that useful, and OTOH the more complicated graphic at the bottom of your picture would get a bit more simple from the client view: The resolution of the link would be resolved by the server when generating the HTML and determine the src of the img, and then the browser would simply apply standard HTML logic and request the image data from the storage service via HTTP.
However, if you would want to optimize load times (and your images would be more or less unique per page so that browser caching of images across pages would not help much), then you could use data URLs embedded into the HTML, and then the first approach could potentially be useful. In this case, all the logic including the generation of the data URL within the HTML would be handled on the server, and the browser would have a single http request.
I wonder as one of my personal projects development goes further forward how should i organize the files ( images, videos, audio files ) uploaded by the users onto AWS's S3/GCE Cloud Storage, i'm used to see these kinds of URL below;
Facebook fbcdn-sphotos-g-a.akamaihd.net/hphotos-ak-xft1/v/t1.0-9/11873531_1015...750483_5263546700711467249_n.jpg?oh=b3f06f7e...b7ebf7&oe=56392950&__gda__=1446569890_628...c7765669456
Tumblr 36.media.tumblr.com/686b47...e93fa09c2478/tumblr_nt7lnyP3ld1rqbl96o1_500.png
Twitter pbs.twimg.com/media/CMimixsV...AcZeM.jpg
Does these random characters carry some kind of meaning? or they're just "UUIDs"? Is there a performance/organization issue in using, for instance this kind of URL below?
content.socialnetworkX.com/userY/post/customName_dinosaurs.jpg
EDIT: Let be clear that i'm considering millions of files.
For S3, see the Performance Considerations page where it talks about object naming. Specifically, if you plan to upload objects at a high rate, you should avoid sequentially named objects, as they can be a bottleneck.
Google Cloud Storage does not have this performance bottleneck. See this answer.
Can uploadcare-widget be used without using the upload care service?
The goal:
Use the widget (specifically to allow users to upload files from their google drive/dropbox accounts).
Instead of using upload care's backend, use your own backend, i.e. node.js/aws s3.
Yes, it can. It's open source!
Although you will have to either replicate or get rid of functionality that relies on Uploadcare infrastructure:
uploads (this is the easiest part)
fetching files from social networks and cloud storage services
image preview and cropping that relies on Uploadcare CDN
So unless you're moving enormous amounts of files, most cost efficient way is to use Uploadcare as it is. BTW, you can use your own S3 storage and even upload directly to your S3 buckets.
I'm using Meteor and would like to create a form with an image upload field that saves the uploaded file to an Amazon S3 bucket in its original size as well as multiple thumbnails sizes defined (passed) via the code.
So far I'm using the lepozepo:s3 package which works great but doesn't seem to allow options for generating additional thumbnails.
Given I can upload the original files onto S3 I'm considering looking into a service on Amazon that can generate the desired thumbnails and then notify my Meteor app. But I'm not sure how to achieve that.
Can anyone point me in the right direction or share some insight into the best approach for this?
PS: I want to avoid using Filepicker.io is possible.
Seems I was following the wrong path. CollectionFS has everything I need and more. I now have this working with plenty of scope to do more later. This is one brilliant collection of packages with clear guides on respective Github pages.
Here are the packages I ended up usings:
cfs:standard-packages - base
cfs:gridfs - required for some reason, not sure why
cfs:graphicsmagick - thumbnailing/cropping
cfs:s3 - S3 upload
Code sample →
CollectionFS is now deprecated, but there are other options:
Only upload, without S3 integration*: https://github.com/tomitrescak/meteor-uploads
Use the jQuery-File-Upload (which is great), it generates thumbs, has size and format validation, etc. Using basically these two packages together:
https://atmospherejs.com/tomi/upload-jquery
https://atmospherejs.com/tomi/upload-server
You can use other package for S3 integration.
Like: https://github.com/peerlibrary/meteor-aws-sdk/
Upload + Integration with S3: https://github.com/Lepozepo/S3
Good, but if you need to generate thumbs for example you will need to integrate with other package or do it yourself. I not tested, but I got this suggestion: https://github.com/jamgold/cropuploader
Upload only, but with examples of how to generate thumbs or integrate with S3 / DropBox / GridFS /: https://github.com/VeliovGroup/Meteor-Files/
Rich documentation and does well which proposes: Upload images.
Use that adapt best to your needs.
look at blueimp's "jquery file upload" for client and image server resizing. On client you have a bit limited possibilities quality wise, on server you can use full power of imagemagick. Or look at my blog post on http://doctorllama.wordpress.com for file uploads for meteor in general.
cfs:gridfs - required for some reason, not sure why
Meteor using gridfs to store file chunks inside mongo database. In case of s3 it's for temporary storage.