I have created an Amazon Web Services S3 bucket in the past, without specifying the geographic region for it to be stored in. How do I determine which region the bucket is located in?
My reason for asking, is that I am preparing to run some "Amazon Elastic MapReduce" jobs on the data in the bucket, and this service asks me in what region I want the job to run. As it seems reasonable to run the job as close to the data as possible, I would like to know where to data actually is.
The options in Elastic MapReduce are:
US East (Virginia)
US West (N. California)
EU West (Ireland)
I am not if these are the exact same locations available in S3. Anyways, I would like to identify the one that is at least closest to my data.
Unless you specified something else, it'll be North America.
If you log in to Amazon web services, and view your usage report, it'll tell you whether it's EU / US or something other.
From reading the Amazon S3 / MapReduce stuff, the regions are different between the two apps, so you probably want to stick to either EU or US (rather than east or west).
Cheers
You can use the GET Location Bucket request. Most probably your bucket is in US.
For anyone else who comes across this looking for a way to determine the region of a bucket, regardless of whether you're the bucket owner (which is a constraint of GET bucket location), use HEAD Bucket, e.g.:
curl -sI foo.s3.amazonaws.com | awk '/^x-amz-bucket-region:/ { print $2 }'
which returns the value of the header x-amz-bucket-region for the bucket "foo", in this case:
us-east-1
Related
According to the S3 FAQ:
"Amazon S3 Standard, S3 Standard-Infrequent Access, and S3 Glacier
storage classes replicate data across a minimum of three AZs to
protect against the loss of one entire AZ. This remains true in
Regions where fewer than three AZs are publicly available."
I'm not clear on what this means. Suppose you store your data in a region with fewer than three AZs that are "publicly available." Does that mean that Amazon will store your data in an AZ within that region that is not publicly available, if necessary? Or that it will store your data in an AZ in another region to make up the difference?
S3 will store your data in an AZ that is not publicly available. The same is also true for DynamoDB, and possibly other services as well.
Source:
I want to say I heard it at a re:Invent session. I’ll try to find a link to some documentation.
This says even if you have mentioned AZ where publicly available AZs are < 3, Amazon S3 makes sure to replicate your data in a total of at least 3 AZs(including public & non-public).
When I run the following:
aws s3 mb s3://toto-pillar-itg-test-export-8 --region eu-west-1
I get:
make_bucket failed: s3://toto-pillar-itg-test-export-8 An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
But, after, when I run the following:
aws s3 mb s3://toto-pillar-itg-test-export-8 --region us-east-1
It works well.
I don't understand why I can't create bucket in eu-west-1 region.
It's not entirely clear what operations you may have attempted, in what order, but here are some thoughts to consider:
You can't have more than one bucket with the same name, regardless of region.
No two AWS accounts can simultaneously have a bucket with the same name, regardless of region.
After creating a bucket, then deleting the bucket, there is a documented but unspecified period of time that must elapse before you -- or anyone else -- can create another bucket with the same name.
The us-east-1 region is the authoritative keeper of the global list of unique bucket names. The other regions only have a copy, so us-east-1 could be expected to be aware of the deletion of a bucket sooner than any other region, making the wait time there shorter than the wait time elsewhere.
The timing may also vary depending on whether create follows delete in the same region or a different region, or by the same account or a different account, but the contribution to the delay by these factors, if any, is not documented.
Clearly, at one point, the eu-west-1 region believed the bucket existed, as evidenced by BucketAlreadyExists, while us-east-1 did not. It may have been a coincidence of the timing of your requests, but the evidence so far suggests that before you tried any of these commands, this bucket had recently been deleted. If that is the case, this is expected behavior, and would eventually resolve itself.
After a bucket is deleted, the name becomes available to reuse, but the name might not be available for you to reuse for various reasons. For example, some other account could create a bucket with that name. Note, too, that it might take some time before the name can be reused. So if you want to use the same bucket name, don't delete the bucket. (emphasis added)
https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
What is better to use, EC2 instances for EBS or Amazon S3 for subscription based streaming channel like Netflix.
150GB upload per month, 250GB streaming per month, no peak time, with viewers based around Australia, India, North America, Europe, Brazil
and 80TB of storage that needs to migrate to the cloud?
For scalability and worldwide presence, the definite answer (using only AWS services) is:
Store videos on Amazon S3
Serve videos through Amazon CloudFront
Amazon CloudFront has presence in 70+ locations around the world and will handle the video streaming protocols for you. Mark content as private and have your application determine whether users are entitled to view videos. You can then generate pre-signed URLs that permit access to a given video for a limited period of time. See: Serving Private Content through CloudFront
In comparison, using Amazon EC2 + Amazon EBS is a poor choice because:
You would need to scale-out additional instances based upon your load
You would need to run instances in multiple regions to be closer to your users (hence lower latency)
You would need to replicate all videos to every server rather than storing a single copy of each video
Please note that your largest cost will be Data Transfer (see Amazon CloudFront Pricing. Your quoted figure of "250GB streaming per month" seems extremely low -- my family alone uses that much bandwidth per month!
I have two wordpress blogs and I am planning to use amazon S3 with one blog and (amazon S3+cloudfront) for another blog.
I read that we need to choose a location when we start our AWS account.
However, for one site (One using cloudfront and amazon S3), my target market is US and UK and another site (Using amazon S3 alone), my target market is India.
In this case, should I use two separate accounts? Or can I have one single account with two locations? (US and Asia).
The one I am using cloudfront for will have video streaming and the one which I use S3 alone will be heavy on images.
Thank you in advance
you can have multiple locations within the same account. When creating a bucket you will be given a choice in which region to create it. E.g. you can have different buckets within the same account located in the different regions.
Thanks,
Andy
What is the easiest way to duplicate an entire Amazon S3 bucket to a bucket in a different account?
Ideally, we'd like to duplicate the bucket nightly to a different account in Amazon's European data center for backup purposes.
One thing to consider is that you might want to have whatever is doing this running in an Amazon EC2 VM. If you have your backup running outside of Amazon's cloud then you pay for the data transfer both ways. If you run in an EC2 VM, you pay no bandwidth fees (although I'm not sure if this is true when going between the North American and European stores) - only for the wall time that the EC2 instance is running (and whatever it costs to store the EC2 VM, which should be minimal I think).
Cool, I may look into writing a script to host on Ec2. The main purpose of the backup is to guard against human error on our side -- if a user accidentally deletes a bucket or something like that.
If you're worried about deletion, you should probably look at S3's new Versioning feature.
I suspect there is no "automatic" way to do this. You'll just have to write a simple app that moves the files over. Depending on how you track the files in S3 you could move just the "changes" as well.
On a related note, I'm pretty sure Amazon does a darn good job backup up the data so I don't think you necessarily need to worry about data loss, unless your back up for archival purposes, or you want to safeguard against accidentally deleting files.
You can make an application or service that responsible to create two instances of AmazonS3Client one for the source and the other for the destination, then the source AmazonS3Client start looping in the source bucket and streaming objects in, and the destination AmazonS3Client streaming them out to the destination bucket.
Note: this doesn't work for cross-account syncing, but this works for cross-region on the same account.
For simply copying everything from one bucket to another, you can use the AWS CLI (https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/): aws s3 sync s3://SOURCE_BUCKET_NAME s3://NEW_BUCKET_NAME
In your case, you'll need the --source-region flag: https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
If you are moving an enormous amount of data, you can optimize how quickly it happens by finding ways to split the transfers into different groups: https://aws.amazon.com/premiumsupport/knowledge-center/s3-large-transfer-between-buckets/
There are a variety of ways to run this nightly. One is example is the AWS instance-schedule (personally unverified) https://docs.aws.amazon.com/solutions/latest/instance-scheduler/appendix-a.html