Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I'm working on a website with a primary function of playing videos, typically one right after another.
Would it be appropriate to store the MP4 & WebW files on Amazon S3, then accomplish playback using HTML5/Flash?
Are there any speed repercussions with serving videos via Amazon S3? Or would I be better off serving the videos from the same Amazon EC2 server I'm using to run the site?
Really I'm looking for Pros/Cons. Thank you.
I cant imagine using Amazon for streaming. Honestly, their traffic costs are way too high for this kind of application.
Anyway, if you still want to use it, S3 doesnt seem to be good option, because it's cluster storage designed for e.g. archiving and not streaming, it has limitations of number of requests per second as well it's concurrency.
For streaming, you need the fastest possible storage, and any of the Amazon services is far away from that, definetely S3 and for EBS, it's not too fast either.
You can consider servers with SSD drives and normal bandwidth prices.
I have myself 10 streaming servers doing 100TB of traffic per day, each with 8x SSD drives and 10Gbps interface plus 64GB of RAM and 16 cores.
I've used Amazons CloudFront to stream content in the past without too much issue (http://aws.amazon.com/cloudfront/), but there are certainly faster methods out there.
However, I do believe it's a good place to start.
Amazon CloudFront supported streaming since December 2009:
We’ve designed Amazon CloudFront to make streaming accessible for
anyone with media content. Streaming with Amazon CloudFront is
exceptionally easy: with only a few clicks on the AWS Management
Console or a simple API call, you’ll be able to stream your content
using a world-wide network of edge locations running Adobe’s Flash®
Media Server. And, like all AWS services, Amazon CloudFront streaming
requires no up-front commitments or long-term contracts. There are no
additional charges for streaming with Amazon CloudFront; you simply
pay normal rates for the data that you transfer using the service.
Recently Amazon CloudFront introduced Live Smooth Streaming:
We are excited to announce the launch of Live Smooth Streaming for
Amazon CloudFront. Smooth Streaming is a feature of Internet
Information Services (IIS) Media Services that enables adaptive
streaming of live media to Microsoft Silverlight clients. You can also
use this solution to deliver your live stream to Apple’s iOS devices
using the Apple HTTP Live Streaming (HLS) format. And you can benefit
from the scale and low-latency offered by Amazon CloudFront when
delivering your live Smooth Streams.
Related
I've been backing up my Mac to the Amazon S3 cloud using Jungle Disk. Now that Mac is dead. Fine, my backups are on the cloud. So, I go to my other Mac and download Jungle Disk. It is a workgroup version of the software. When I run it it wants me to verify that I purchased the software. Well, when I first set up the Jungle Disk client some years ago there was a free client. I'd rather not pay for this unless there's no good alternative.
Next I login to my Amazon S3 Console. I have a bunch of buckets there which are impossible to navigate.
So, I google around for S3 browsers and find Cyberduck. I download and install that. When I run it it wants a server URL. At this point I'm stuck.
Is there a client that knows about the structure of backups in S3 that I can install on this other Mac to get to my backed up data?
After a couple of conversations with Jungle Disk support I was given this (undocumented) url:
https://downloads.jungledisk.com/jungledisk/JungleDiskDesktop3160.dmg
I've downloaded and installed the client, didn't have to pay anything, and I've gotten to my backed up data. Whew!
Sol got his stuff fixed. Sharing additional background for future readers. Jungle Disk uses the WebDAV standard to allow access through our web service layer. Depending on the version of Jungle Disk you're running we have a few different URLs you'll authenticate to. Ping our team at support.jungledisk.com and we'll get you setup.
What exactly the SDK can be used for ? Only for storage like it's done on google drive, box or dropbox etc ? Or can i use the stored scripts to run a complete website ?
What exactly the SDK can be used for?
The Software Development Kit (SDK) can be used to programmatically control nearly every single aspect across all 40± AWS services.
Only for storage like it's done on google drive, box or dropbox etc?
Amazon S3 is a storage-only service. It complements the plethora of other AWS services.
Or can i use the stored scripts to run a complete website?
For that, you'd need something with a server. I recommend taking a look at AWS Elastic Beanstalk first because that's arguably the quickest way to get something running. If you're looking for something with more control, you can check out AWS OpsWorks.
If you want a raw virtual server, take a look at Amazon EC2. If you want to build a template that can automate and configure nearly your entire cloud infrastructure (storage, compute, databases, etc.), take a look at Amazon CloudFormation.
We are starting a video sharing website where users will be able to upload videos in their native formats. However, since video streaming on the web generally is in the FLV format, we need to convert the videos to FLV.
Also, the site will be hosted on Amazon EC2 and storage using S3.
Can i run FFMpeg on amazon EC2 ? Is this the best way to go ? Are there other alternatives to video encoding rather than doing conversion on our own server ? I also came across www.transloadit.com which seems to do the same but they are charging a bomb. Are there cheaper and more intelligent alternatives ?
We are planning to make this website as one of top 10 biggest niche video streaming websites on the internet.
EC2 instances are just virtual machines so you can do whatever you like on them, including running ffmpeg.
Only you can work out the costs/benefits of doing the conversion on EC2, another server or an encoding service like encoding.com(a google search will turn up more services).
Some thoughts:
EC2
pay by the hour and can easily add new servers (although you need to design your process to support multiple servers)
Fast (and free) transfer between EC2 and S3
Your own servers
You pay for hardware upfront
Not as easy to scale quickly if needed
You need to maintain the hardware
Bandwidth charges between EC2/S3 and your servers
In both DIY solutions you need to deal with the notoriously error prone process and converting videos of different formats.
Video Encoding Service
Probably more expensive (debatable if you factor in development time and support costs)
Easiest way to scale quickly
Up and running quickly
Let them deal with the difficult conversions
I'm going to talk about a slightly different part of your comment. You said:
However, since video streaming on the web generally is in the FLV format...
This is false. You'll get far more portability and bang for your buck if you encode in MPEG-4/H.264.
Flash Player can playback H.264 content, so you can still use a Flash-based player for your website if you want to. However, if you ever decide to open up to mobile devices (iPhone, iPad, Android, webOS, Blackberry 6), HTML5-compatible web browsers (Safari, Chrome, Firefox, Opera, IE9), or pretty much anything newer than 5 years ago, H.264 is definitely the way to go.
The site for Miro Video Converter even documents the FFmpeg setting they use, which may save you some time.
Converting video is a relatively processor-intensive process. Amazon charges for CPU time, and they also charge for data transfer. So it's more of a business tradeoff. Can EC2 run ffmpeg and do the video conversion? Yes, it can. But is it more cost-effective to pay for the CPU time on the EC2 instance or to convert on a local server and then transfer the data to the EC2? I don't know. The answer depends on the sizes of videos that you're working with, the cost of connectivity on your local server, and the pricing scheme on your EC2 virtual server instance.
I've currently got a base Windows 2008 Server AMI that I created on Amazon EC2. I use it to create 20-30 EBS-based EC2 instances at a time for processing large amounts of data into PDFs for a client. However, once the data processing is complete, I have to manually connect to each machine and copy off the files. This takes a lot of time and effort, and so I'm trying to figure out the best way to use S3 as a centralised storage for the outputted PDF files.
I've seen a number of third party (commercial) utilities that can map S3 buckets to drives within Windows, but is there a better, more sensible way to achieve what I want? Having not used S3 before, only EC2, I'm not sure of what options are available, and I've not been able to find anything online addressing the issue of using S3 as centralised storage for multiple EC2 Windows instances.
Update: Thanks for suggestions of command line tools for using S3. Was hoping for something a little more integrated and less ad-hoc. Seeing as EC2 is closely related to S3 (S3 used to be the default storage mechanism for AMIs, etc), that there might be something neater/easier I could do. Perhaps even around Private Cloud Networks and EC2 backed S3 servers, etc, or something (an area I know nothing about). No other ideas?
I'd probably look for a command line tool. A quick search on Google lead me to a .Net tool:
http://s3.codeplex.com/
And a Java one:
http://www.beaconhill.com/opensource/s3cp.html
I'm sure there are others out there as well.
You could use an EC2 instance with EBS exported through samba which can act as a centralized storage that windows instances can map?
this sounds very much like a hadoop/Amazon MapReduce job to me. Unfortunately, hadoop is best deployed on Linux:
Hadoop on windows server
I assume the software you use for pdf-processing is Windows only?
If this is not the case, I'd seriously consider porting your solution to Linux.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I just started using the Amazon S3 and Amazon CloudFront. What proper client tools are out there that I can use to manage my account? Like uploading files etc. Yes I am a developer, but, I am pressed for time, I just want to deploy my apps.
I came across this one S3 Browser. It's almost what I am looking for.
EDIT ~ Is it possible to map a bucket as a windows Drive?
I use CloudBerry FreeWare. Easy to use, just like FTP software.
https://www.cloudberrylab.com/explorer/amazon-s3.aspx
Jeff Atwood mentioned S3Fox Organizer on his CodingHorror blog.
http://www.s3fox.net/
Using Amazon S3 as an Image Hosting Service
Cyberduck for Mac & Windows supports the full feature set of both S3 and Cloudfront distributions including custom origins. See http://trac.cyberduck.ch/wiki/help/en/howto/s3.
I use CrossFTP Pro to handle Amazon S3 and CloudFront on the Mac, Linux, and Windows. It can handle the HTTP header metadata and log settings as well.
While several of the mentioned client tools offer useful advanced features and/or a different approach to usability still, you can nowadays achieve most goals immediately and conveniently by means of Amazon's very own AWS Management Console, which allows you to Access and manage Amazon’s growing suite of infrastructure web services through a simple and intuitive, web-based user interface.
Obviously the best way to check it out is to simply explore it yourself, but the following resources allow a sneak peek:
Amazon S3 Features and the introduction AWS Management Console Now Supports Amazon S3 as well as New S3 Features for The AWS Management Console
Amazon CloudFront Features (includes a screencast) and the introduction AWS Management Console Support for CloudFront as well as Improved CloudFront Support in the AWS Management Console
I use the S3 Organizer plugin for FireFox to manage S3 as well as ElasticFox to manage my AWS instances.
I like Cloudberry Explorer too, but had to use Bucket Explorer to create multiple distributions per bucket. I probably missed how, but couldn't figure out how to do that with the other tools.
In fact you don't need anthing to install at all. You can manage your files online from your browser wherever you are using S3fm ( http://www.s3fm.com/ ), free web based Amazon S3 file manager.
It's an Ajax app, not a hosted solution so you don't actually share your credentials with anyone.