I have two buckets. In the AWS Console they have the following "Date created"
2017-11-22 14:07:03 i-repo
2018-01-12 17:16:31 l-repo
Using the AWS CLI (aws-cli/1.16.90 Python/3.7.2 Darwin/17.7.0 botocore/1.12.80) command aws s3 ls, I get the following:
2018-02-08 12:49:03 i-repo
2018-12-19 15:55:29 l-repo
Using the AWS CLI command aws s3api list-buckets, I get the same incorrect dates. I have confirmed that the dates that the AWS CLI reports relate to the date of the most recent bucket policy change, NOT the bucket create date. Am I missing something, or is this a bug?
Looks like this is a known issue/intended. See below:
After further investigation and discussion with the S3 team, I have found that this is expected behavior due to the design of the service. The GET Service call in S3 (s3api list-buckets or s3 ls with no further arguments in the CLI) works differently when being run against different regions. All bucket creations are mastered in us-east-1, then replicated on a global scale - the resulting difference is that there are no "replication" events to the us-east-1 region. The Date Created field displayed in the web console is according to the actual creation date registered in us-east-1, while the AWS CLI and SDKs will display the creation date depending on the specified region (or the default region set in your configuration).
When using an endpoint other than us-east-1, the CreationDate you receive is actually the last modified time according to the bucket's last replication time in this region. This date can change when making changes to your bucket, such as editing its bucket policy. This experienced behavior is result of how S3's architecture has been designed and implemented, making it difficult to change without affecting customers that already expect this behavior.
S3 does intend to change this behavior so that the actual bucket creation date is shown regardless of the region in which the GET Service call is issued, however to answer your question we do not yet have an ETA for the implementation of this change. This change would most likely be announced in the AWS Forums for S3 if you'd like to know when it takes place.
Source
Related
I am using s3 bucket to store my data. And I keep pushing data to this bucket every single day. I wonder whether there is feature I can compare the files different in my bucket between two date. I not, is there a way for me to build one via aws cli or sdk?
The reason I want to check this is that I have a s3 bucket and my clients keep pushing data to this bucket. I want to have a look how much data they pushed since the last time I load them. Is there a pattern in aws support this query? Or do I have to create any rules in s3 bucket to analyse it?
Listing from Amazon S3
You can activate Amazon S3 Inventory, which can provide a daily file listing the contents of an Amazon S3 bucket. You could then compare differences between two inventory files.
List it yourself and store it
Alternatively, you could list the contents of a bucket and look for objects dated since the last listing. However, if objects are deleted, you will only know this if you keep a list of objects that were previously in the bucket. It's probably easier to use S3 inventory.
Process it in real-time
Instead of thinking about files in batches, you could configure Amazon S3 Events to trigger something whenever a new file is uploaded to the Amazon S3 bucket. The event can:
Trigger a notification via Amazon Simple Notification Service (SNS), such as an email
Invoke an AWS Lambda function to run some code you provide. For example, the code could process the file and send it somewhere.
When I run the following:
aws s3 mb s3://toto-pillar-itg-test-export-8 --region eu-west-1
I get:
make_bucket failed: s3://toto-pillar-itg-test-export-8 An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
But, after, when I run the following:
aws s3 mb s3://toto-pillar-itg-test-export-8 --region us-east-1
It works well.
I don't understand why I can't create bucket in eu-west-1 region.
It's not entirely clear what operations you may have attempted, in what order, but here are some thoughts to consider:
You can't have more than one bucket with the same name, regardless of region.
No two AWS accounts can simultaneously have a bucket with the same name, regardless of region.
After creating a bucket, then deleting the bucket, there is a documented but unspecified period of time that must elapse before you -- or anyone else -- can create another bucket with the same name.
The us-east-1 region is the authoritative keeper of the global list of unique bucket names. The other regions only have a copy, so us-east-1 could be expected to be aware of the deletion of a bucket sooner than any other region, making the wait time there shorter than the wait time elsewhere.
The timing may also vary depending on whether create follows delete in the same region or a different region, or by the same account or a different account, but the contribution to the delay by these factors, if any, is not documented.
Clearly, at one point, the eu-west-1 region believed the bucket existed, as evidenced by BucketAlreadyExists, while us-east-1 did not. It may have been a coincidence of the timing of your requests, but the evidence so far suggests that before you tried any of these commands, this bucket had recently been deleted. If that is the case, this is expected behavior, and would eventually resolve itself.
After a bucket is deleted, the name becomes available to reuse, but the name might not be available for you to reuse for various reasons. For example, some other account could create a bucket with that name. Note, too, that it might take some time before the name can be reused. So if you want to use the same bucket name, don't delete the bucket. (emphasis added)
https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html
I have the following requirement. I'm however unsure of how to go about it
Bucket 1 contains data.
Bucket 2 should have duplicate data of Bucket 1. Whenever any file is changed in bucket 1, it is also be changed in bucket 2.
Data in bucket 2 can be independently changed. However, this data change should not be reflected in bucket 1.
This entire process must be automated and run in real time.
Depending on your needs, you might find Cross Region Replication works for you. This would require the buckets to be in separate regions. It also wouldn't copy items that were replicated from another bucket.
Essentially you just create two buckets in separate regions, create an IAM role allowing the replication, then create a Replication Configuration.
If you already have data in the source bucket that you want to appear in the target bucket, then you will also need to run a sync (you can do this as a one-off via the cli).
Another option is using AWS Lambda, which allows the buckets to be in the same region, and gives you more control should you need it. You can also replicate to multiple buckets if you want to.
On application level , app-user marks S3 objects as deleted [sets delete marker].
The bucket is versioning-enabled.
I want to retain the marked as deleted object and all it's versions for 30 days and then delete all of them.
Is the following lifecycle policy correct?
Yes, you have the correct settings to only delete versions after 30 days. On the previous page in the S3 console you can choose the prefix under which to delete the previous versions, or the whole bucket.
I use http://www.insight4storage.com/ from AWS Marketplace to lower my costs by tracking my AWS S3 storage usage trends by prefix, bucket or storage class over time; plus it shows me the previous versions storage by prefix and per bucket.
I am aware that Standard per month charge is more than Standard-IA.
I wasn't able to see in the docs if I upload to Standard and then set a Lifecycle rule to transition to Standard-IA is there a cost involved (for example to copy from Standard > Standard-IA) or is that part free?
Asked another way if I know my files are always going to be 'IA' should I just upload them that way or set the bucket that way from the offset?
The Amazon S3 pricing page shows:
Lifecycle Transition Requests into Standard – Infrequent Access: $0.01 per 1,000 requests
It is better to upload directly into STANDARD_IA. This can be specified as part of the upload (eg see --storage-class option in the AWS Command-Line Interface (CLI) cp command ).
It is not possible to "set the bucket that way" -- the storage class is stored per-object.