When I run the following:
aws s3 mb s3://toto-pillar-itg-test-export-8 --region eu-west-1
I get:
make_bucket failed: s3://toto-pillar-itg-test-export-8 An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
But, after, when I run the following:
aws s3 mb s3://toto-pillar-itg-test-export-8 --region us-east-1
It works well.
I don't understand why I can't create bucket in eu-west-1 region.
It's not entirely clear what operations you may have attempted, in what order, but here are some thoughts to consider:
You can't have more than one bucket with the same name, regardless of region.
No two AWS accounts can simultaneously have a bucket with the same name, regardless of region.
After creating a bucket, then deleting the bucket, there is a documented but unspecified period of time that must elapse before you -- or anyone else -- can create another bucket with the same name.
The us-east-1 region is the authoritative keeper of the global list of unique bucket names. The other regions only have a copy, so us-east-1 could be expected to be aware of the deletion of a bucket sooner than any other region, making the wait time there shorter than the wait time elsewhere.
The timing may also vary depending on whether create follows delete in the same region or a different region, or by the same account or a different account, but the contribution to the delay by these factors, if any, is not documented.
Clearly, at one point, the eu-west-1 region believed the bucket existed, as evidenced by BucketAlreadyExists, while us-east-1 did not. It may have been a coincidence of the timing of your requests, but the evidence so far suggests that before you tried any of these commands, this bucket had recently been deleted. If that is the case, this is expected behavior, and would eventually resolve itself.
After a bucket is deleted, the name becomes available to reuse, but the name might not be available for you to reuse for various reasons. For example, some other account could create a bucket with that name. Note, too, that it might take some time before the name can be reused. So if you want to use the same bucket name, don't delete the bucket. (emphasis added)
https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
Related
I have two buckets. In the AWS Console they have the following "Date created"
2017-11-22 14:07:03 i-repo
2018-01-12 17:16:31 l-repo
Using the AWS CLI (aws-cli/1.16.90 Python/3.7.2 Darwin/17.7.0 botocore/1.12.80) command aws s3 ls, I get the following:
2018-02-08 12:49:03 i-repo
2018-12-19 15:55:29 l-repo
Using the AWS CLI command aws s3api list-buckets, I get the same incorrect dates. I have confirmed that the dates that the AWS CLI reports relate to the date of the most recent bucket policy change, NOT the bucket create date. Am I missing something, or is this a bug?
Looks like this is a known issue/intended. See below:
After further investigation and discussion with the S3 team, I have found that this is expected behavior due to the design of the service. The GET Service call in S3 (s3api list-buckets or s3 ls with no further arguments in the CLI) works differently when being run against different regions. All bucket creations are mastered in us-east-1, then replicated on a global scale - the resulting difference is that there are no "replication" events to the us-east-1 region. The Date Created field displayed in the web console is according to the actual creation date registered in us-east-1, while the AWS CLI and SDKs will display the creation date depending on the specified region (or the default region set in your configuration).
When using an endpoint other than us-east-1, the CreationDate you receive is actually the last modified time according to the bucket's last replication time in this region. This date can change when making changes to your bucket, such as editing its bucket policy. This experienced behavior is result of how S3's architecture has been designed and implemented, making it difficult to change without affecting customers that already expect this behavior.
S3 does intend to change this behavior so that the actual bucket creation date is shown regardless of the region in which the GET Service call is issued, however to answer your question we do not yet have an ETA for the implementation of this change. This change would most likely be announced in the AWS Forums for S3 if you'd like to know when it takes place.
Source
I have the following requirement. I'm however unsure of how to go about it
Bucket 1 contains data.
Bucket 2 should have duplicate data of Bucket 1. Whenever any file is changed in bucket 1, it is also be changed in bucket 2.
Data in bucket 2 can be independently changed. However, this data change should not be reflected in bucket 1.
This entire process must be automated and run in real time.
Depending on your needs, you might find Cross Region Replication works for you. This would require the buckets to be in separate regions. It also wouldn't copy items that were replicated from another bucket.
Essentially you just create two buckets in separate regions, create an IAM role allowing the replication, then create a Replication Configuration.
If you already have data in the source bucket that you want to appear in the target bucket, then you will also need to run a sync (you can do this as a one-off via the cli).
Another option is using AWS Lambda, which allows the buckets to be in the same region, and gives you more control should you need it. You can also replicate to multiple buckets if you want to.
Hi is there any version of amazon web services provides unlimited buckets?
No. From the S3 documentation:
Each AWS account can own up to 100 buckets at a time.
S3 buckets are expensive (in terms of resources) to create and destroy:
The high availability engineering of Amazon S3 is focused on get, put, list, and delete operations. Because bucket operations work against a centralized, global resource space, it is not appropriate to make bucket create or delete calls on the high availability code path of your application. It is better to create or delete buckets in a separate initialization or setup routine that you run less often.
There's also no good reason to use lots of buckets:
There is no limit to the number of objects that can be stored in a bucket and no variation in performance whether you use many buckets or just a few. You can store all of your objects in a single bucket, or you can organize them across several buckets.
You want a separate space for each of your users to put things. Fine: create a single bucket and give your user-specific information a <user_id>/ prefix. Better yet, put it in users/<user_id>/, so you can use the same bucket for other non-user-specific things later, or change naming schemes, or anything else you might want.
ListObjects accepts a prefix parameter (users/<user_id>/), and has special provisions for hierarchical keys that might be relevant.
Will is correct. For cases where you can really prove a decent use-case, I'm would imagine that AWS would consider bumping your quota (as they will do for most any service). I wouldn't be surprised if particular users have 200-300 buckets or more, but not without justifying it on good grounds to AWS.
With that said, I cannot find any S3 Quota Increase form alongside the other quota increase forms.
I'm setting up my client with a system that allows users to upload a video or two. These videos will be stored on Amazon S3, which I've not used before. I'm unsure about buckets, and what they represent. Do you think I would have a single bucket for my application, a bucket per user or a bucket per file?
If I were to just have the one bucket, presumably I'd have to have really long, illogical file names to prevent a file name clash.
There is no limit to the amount of objects you can store in a bucket, so generally you would have a single bucket per application, or even across multiple applications. Bucket names have to be globally unique across S3 so it would certainly be impossible to manage a bucket per object. A bucket per user would also be difficult if you had more than a handful of users.
For more background on buckets you can try reading Working with Amazon S3 Buckets
Your application should generate unique keys for objects you are adding to the bucket. Try and avoid numeric ascending ids, as these are considered inefficient. Simply reversing a numeric id can usually make an effective object key. See Amazon S3 Performance Tips & Tricks for a more detailed explanation.
I have created an Amazon Web Services S3 bucket in the past, without specifying the geographic region for it to be stored in. How do I determine which region the bucket is located in?
My reason for asking, is that I am preparing to run some "Amazon Elastic MapReduce" jobs on the data in the bucket, and this service asks me in what region I want the job to run. As it seems reasonable to run the job as close to the data as possible, I would like to know where to data actually is.
The options in Elastic MapReduce are:
US East (Virginia)
US West (N. California)
EU West (Ireland)
I am not if these are the exact same locations available in S3. Anyways, I would like to identify the one that is at least closest to my data.
Unless you specified something else, it'll be North America.
If you log in to Amazon web services, and view your usage report, it'll tell you whether it's EU / US or something other.
From reading the Amazon S3 / MapReduce stuff, the regions are different between the two apps, so you probably want to stick to either EU or US (rather than east or west).
Cheers
You can use the GET Location Bucket request. Most probably your bucket is in US.
For anyone else who comes across this looking for a way to determine the region of a bucket, regardless of whether you're the bucket owner (which is a constraint of GET bucket location), use HEAD Bucket, e.g.:
curl -sI foo.s3.amazonaws.com | awk '/^x-amz-bucket-region:/ { print $2 }'
which returns the value of the header x-amz-bucket-region for the bucket "foo", in this case:
us-east-1