Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
We're uploading and serving/streaming media (pics,videos) using amazon s3 for storage combined with cloudfront for serving media. The site is used slightly but the Amazon costs come to 3000 $ per month and from the report 90% of the costs originate from the S3 service .
Heard that cloud can be expensive if you don't code the right way ..now my questions :
What is the right way ? and where should I pay more attention, to the way I upload files or to the way I serve them?
Has anyone else had to deal with unexpected high costs , if yes what was the cause?
We have almost similar model. We stream (rtmp ) from S3 and cloudfront. We do have 1000s of files and decent load, but our monthly bill for s3 is around 50$ ( negligible as compared to your figure). Firstly , you should complain about your charges to the technical support of AWS. They always give you a good response and also suggest better ways to utilize resources. Secondly , I think if you do live streaming, where you divide the file into chucks and stream them one by one, instead of streaming or downloading the whole file, it might be effective , in terms of i/o where users are not watching the whole video, but just the part of it. Also, you can try to utilize caching eat application level.
Another opportunity to get better picture on what's going on in your buckets: Qloudstat
Related
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
Is there a way to check that my files are already on the edge servers for my users to load from? Or does amazon s3 take time to spread your files around the world? How Long and can I receive notification about when?
So after I uploaded a file, I immediately tested the load speed by asking users from other far away places(like places in Japan). They said that it was rather slower than my current hosting in the US. That's odd because Amazon does have an edge server in Tokyo so Amazon s3 should be faster?
Before I created my bucket, I did set the region to be in the standard US. Is that why? If so, is there a way to set your files to work around the world?
Thank you for your time.
As you already said, your S3 buckets are situated in a specific location, for example us-east, europe, us-west etc. This is the place where your files are physically stored. They are not distributed geographically. Users from other places in the world will experience delay when requesting data from these buckets.
What you are looking for is the Cloudfront CDN from Amazon. You can specify an origin (that would be your S3 bucket in your case) and then your files will be distributed to all the Amazon Cloudfront edge locations worldwide. Check out their FAQ and the list of edge locations.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
We are gathering information about different systems. At the moment we are looking for a storage solution. We will have a high amount of outgoing traffic with large files.
I want to compare s3 with google cloud storage.
google cloud storage costs around $0.08/GB at 90TB. S3 is around $0.06. But google cloud storage has already a cdn, which makes it way cheaper than amazon s3 with cloudfront.
Now I read somewhere that google cloud stroage is much slower than s3 with very large files. Is this true ?
I can not find any information.
What alternatives do I have if I have a high amount of outgoing traffic and large files ?
Edit:
benchmarks:
http://blog.zencoder.com/2012/07/23/first-look-at-google-compute-engine-for-video-transcoding/
2016 update - Look at this benchmark comparing AWS, Azure, and Google Cloud storage:
http://blog.zachbjornson.com/2015/12/29/cloud-storage-performance.html
As a reddit user commented:
“Not for network throughput on GCP. It consistently beats out all of
the competition by a such huge margin that if you’re writing an App
that’s latency/bandwidth sensitive on the network side just ignore the
competition.”
For current pricing schemes, compare:
https://developers.google.com/storage/pricing#pricing
https://aws.amazon.com/s3/pricing/
Performance is trickier to evaluate: You can get numbers, but those numbers will depend a lot on current conditions, network, proximity to datacenters, ratio of read/writes, etc.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I would like to serve user uploaded content (pictures, videos, and other files) from a CDN. using Amazon S3 with cloudfront seems like a reasonable way to go. My only question is about the speed of the file system. My plan was to host user media with the following uri. cdn.mycompany.com/u/u/i/d/uuid.jpg.
I don't haven any prior experience with S3 or CDN's and I was just wondering if this strategy would scale well to handle a large amount of user uploaded content. And if there might be another conventional way to accomplish this.
You will never have problems dealing with scale on CloudFront. It's an enterprise-grade beast.
Disclaimer: Not if you're Google.
It is an excellent choice. Especially for streaming video and audio, CloudFront is priceless.
My customers use my plugin to display private streaming video and audio, one of them even has 8,000 videos in one bucket without problems.
My question stemmed from a misunderstanding of S3 buckets as a conventional file system. I was concerned that hacking too many files in the same directory would create overhead in finding the file. However, it turns out that S3 buckets are implemented more something like a hashmap so this overhead doesn't actually exist. See here for details: Max files per directory in S3
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I'm looking for examples of large websites that are hosted on Amazon EC2/S3/Cloudfront/etc.
I worked at a company where we ran a site that did 500k unique visitors per month on EC2 running 10 instances but that's still relatively small potatoes compared to some larger sites. I know smug mug and foursquare are also hosted on EC2. What other large websites are on EC2?
Reddit uses EC2
How large is 'large'? I know that heroku runs on EC2, and they do a pretty large amount of traffic.
Netflix, DropBox, Zynga (maker of Farmville) are pretty big users of EC2
Instagram uses Amazon... at least their images are hosted on S3. Check out this link:
http://distillery.s3.amazonaws.com/media/2011/09/17/42923def0eb141fc8bb4ab0c963d0dbc_6.jpg
Also, I'm pretty sure Reddit moved away from Amazon Cloud after the most recent outage.
Here's a link to Amazon case studies that details some of their larger clients:
http://aws.amazon.com/solutions/case-studies/
In the financial world Nasdaq OMX is a good example but virtually every bank uses EC2 for computational needs (not as a main system but highly used by both banks and hedge funds to run simulations).
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I am trying to communicate to a client the likely-hood of losing files in S3. I would also like to know if it is possible to lose an entire bucket from S3. So, I would like to know the following:
Is there a documented expected file loss percentage in S3?
Is there a documented expected bucket loss percentage in S3?
When I say "lose" a file. I mean a file that is lost, damaged or otherwise unable to be pulled from S3. This "loss" is caused by a failure on S3. It is not caused by a tool or other user error.
Amazon doesn't give any kind of SLA or data loss guarantees for data stored on S3, but as far as I know nobody has ever lost any data on S3 aside from user/tool errors.
I would say the probability of user / coder error causing data loss is substantially greater than data loss through some kind of failure on S3. So you may wish to consider some kind of backup strategy to mitigate that.