I would like to implement a cloud storage service with the same interface of OpenStack Swift or Amazon S3. In other words, my cloud storage service should expose the same API of the above-mentioned services but with a custom implementation. This way, a client will be able to interoperate with my service without changing its implementation.
I was wondering if there is an easier approach than manually implementing such interfaces starting from the documentation: http://docs.openstack.org/api/openstack-object-storage/1.0/content/ http://docs.aws.amazon.com/AmazonS3/latest/API/APIRest.html
For instance, it would be nice if there was a "skeleton" of OpenStack Swift or Amazon S3 APIs from which I can start implementing my service.
Thanks
I found exactly what I was looking for:
https://github.com/jubos/fake-s3
https://github.com/scireum/s3ninja
These tools emulate most of Amazon S3 API. They are meant for development and test purposes but in my case I can use them as a starting point for implementing my cloud storage service.
Someone has done this for you, try jcloud, it supports AWS S3 and swift: Apache jclouds®.
I would recommend using Swift (Openstack object store ) which also supports S3 API
Take a look at the following link:
http://docs.openstack.org/grizzly/openstack-object-storage/admin/content/configuring-openstack-object-storage-with-s3_api.html
This way you can work with openstack swift or Amazon S3
Another option is libcloud, it is a python abstraction that supports a number of providers (including S3 and Swift):
https://libcloud.readthedocs.org/en/latest/storage/index.html
http://libcloud.apache.org/
If you are looking for an enterprise / carrier grade object storage software solution, look at Cloudian http://www.cloudian.com.
Cloudian's software delivers a fully Amazon S3 compliant API, meaning that it delivers the broadest range of S3 feature coverage and 100% fidelity with the AWS S3 API.
The software comes with a Free 10TB license, so pretty much it is free up to 10TB of managed storage, after that it is reasonably priced. You can install the software in any x86 hardware running Linux.
Cloudian does not support the Swift API though.
[Disclaimer: I work for Cloudian]
Related
I would like to access my Amazon S3 buckets without third-party software, but simply through the WebDAV functionality available in most operating systems. Is there a way to do that ? It is important to me that no third-party software is required.
There's a number of ways to do this. I'm not sure about your situation, so here they are:
Option 1: Easiest: You can use a 3rd party "cloud gateway" provider, like http://storagemadeeasy.com/CloudDav/
Option 2: Set up your own "cloud gateway" server
Set up a dedicated server or virtual server to act as a gateway. Using Amazon's own EC2 would be a good choice.
Set up software that mounts S3 as a drive. Two I know of on Windows: (1) CloudBerry Drive http://www.cloudberrylab.com/ and (2) WebDrive (http://webdrive.com). For Linux, I have never done it, but you can try: https://github.com/s3fs-fuse/s3fs-fuse
Set up a webdav server like CrushFTP. (It comes to mind because it's stable and cheap and works on any OS.) Another option is IIS but I personally find it's harder to set up securely for webdav.
Set up a user in your WebDav server (ie CrushFTP or IIS) with access to the mapped S3 drive.
Possible snag: Assuming you're using Windows, to start your services automatically and have this work, you may need to set up both services to use the same Windows user account (Services->(Your Service)->[right-click]Properties->Log On tab). This is because the S3 mapping software might not map the S3 drive for all Windows users. Alternatively, you can use FireDaemon if you get stuck on this step to start the programs as a service all under the same username.
Other notes: I have experience using WebDrive under pretty heavy loads, and it seems to work well. Under tons of pounding (I'm talking thousands of files per hour being added to a 5 TB WebDrive) it started to crash Windows. But I'm not sure if you are going that far with it. Also, if you're using EC2, you may not have that issue since it was likely caused by a huge transfer queue in memory and EC2 will have faster transit to S3 and keep the queue smaller.
I finally gave up on this idea and today I use Rclone (https://rclone.org) to synchronize my files between AWS S3 and different computers. Rclone has the ability to mount remote storage on a local computer, but I don't use this feature. I simply use the copy and sync commands.
S3 does not support webdav, so you're out of luck!
Also, S3 does not support hierarchial name spaces, so you cant directly map a filesystem onto it
There is an example java project here for putting a webdav server over Amazon S3 - https://github.com/miltonio/milton-aws
What exactly the SDK can be used for ? Only for storage like it's done on google drive, box or dropbox etc ? Or can i use the stored scripts to run a complete website ?
What exactly the SDK can be used for?
The Software Development Kit (SDK) can be used to programmatically control nearly every single aspect across all 40± AWS services.
Only for storage like it's done on google drive, box or dropbox etc?
Amazon S3 is a storage-only service. It complements the plethora of other AWS services.
Or can i use the stored scripts to run a complete website?
For that, you'd need something with a server. I recommend taking a look at AWS Elastic Beanstalk first because that's arguably the quickest way to get something running. If you're looking for something with more control, you can check out AWS OpsWorks.
If you want a raw virtual server, take a look at Amazon EC2. If you want to build a template that can automate and configure nearly your entire cloud infrastructure (storage, compute, databases, etc.), take a look at Amazon CloudFormation.
I know amazon has several api's but I was wondering what language and tools does amazon use to build their api, is it java and soap? Im not sure if all the api's are created the same way, but if not then how is the one for mobile apps created? Thanks!
Amazon are notoriously secretive about the internals of their systems so I don't think you'll find a satisfactory answer.
The AWS APIs, as far as I know, use Java internally & they use plenty of C++ to run their shopping website
I've currently got a base Windows 2008 Server AMI that I created on Amazon EC2. I use it to create 20-30 EBS-based EC2 instances at a time for processing large amounts of data into PDFs for a client. However, once the data processing is complete, I have to manually connect to each machine and copy off the files. This takes a lot of time and effort, and so I'm trying to figure out the best way to use S3 as a centralised storage for the outputted PDF files.
I've seen a number of third party (commercial) utilities that can map S3 buckets to drives within Windows, but is there a better, more sensible way to achieve what I want? Having not used S3 before, only EC2, I'm not sure of what options are available, and I've not been able to find anything online addressing the issue of using S3 as centralised storage for multiple EC2 Windows instances.
Update: Thanks for suggestions of command line tools for using S3. Was hoping for something a little more integrated and less ad-hoc. Seeing as EC2 is closely related to S3 (S3 used to be the default storage mechanism for AMIs, etc), that there might be something neater/easier I could do. Perhaps even around Private Cloud Networks and EC2 backed S3 servers, etc, or something (an area I know nothing about). No other ideas?
I'd probably look for a command line tool. A quick search on Google lead me to a .Net tool:
http://s3.codeplex.com/
And a Java one:
http://www.beaconhill.com/opensource/s3cp.html
I'm sure there are others out there as well.
You could use an EC2 instance with EBS exported through samba which can act as a centralized storage that windows instances can map?
this sounds very much like a hadoop/Amazon MapReduce job to me. Unfortunately, hadoop is best deployed on Linux:
Hadoop on windows server
I assume the software you use for pdf-processing is Windows only?
If this is not the case, I'd seriously consider porting your solution to Linux.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I just started using the Amazon S3 and Amazon CloudFront. What proper client tools are out there that I can use to manage my account? Like uploading files etc. Yes I am a developer, but, I am pressed for time, I just want to deploy my apps.
I came across this one S3 Browser. It's almost what I am looking for.
EDIT ~ Is it possible to map a bucket as a windows Drive?
I use CloudBerry FreeWare. Easy to use, just like FTP software.
https://www.cloudberrylab.com/explorer/amazon-s3.aspx
Jeff Atwood mentioned S3Fox Organizer on his CodingHorror blog.
http://www.s3fox.net/
Using Amazon S3 as an Image Hosting Service
Cyberduck for Mac & Windows supports the full feature set of both S3 and Cloudfront distributions including custom origins. See http://trac.cyberduck.ch/wiki/help/en/howto/s3.
I use CrossFTP Pro to handle Amazon S3 and CloudFront on the Mac, Linux, and Windows. It can handle the HTTP header metadata and log settings as well.
While several of the mentioned client tools offer useful advanced features and/or a different approach to usability still, you can nowadays achieve most goals immediately and conveniently by means of Amazon's very own AWS Management Console, which allows you to Access and manage Amazon’s growing suite of infrastructure web services through a simple and intuitive, web-based user interface.
Obviously the best way to check it out is to simply explore it yourself, but the following resources allow a sneak peek:
Amazon S3 Features and the introduction AWS Management Console Now Supports Amazon S3 as well as New S3 Features for The AWS Management Console
Amazon CloudFront Features (includes a screencast) and the introduction AWS Management Console Support for CloudFront as well as Improved CloudFront Support in the AWS Management Console
I use the S3 Organizer plugin for FireFox to manage S3 as well as ElasticFox to manage my AWS instances.
I like Cloudberry Explorer too, but had to use Bucket Explorer to create multiple distributions per bucket. I probably missed how, but couldn't figure out how to do that with the other tools.
In fact you don't need anthing to install at all. You can manage your files online from your browser wherever you are using S3fm ( http://www.s3fm.com/ ), free web based Amazon S3 file manager.
It's an Ajax app, not a hosted solution so you don't actually share your credentials with anyone.