Just a quick one - how do you identify which IAM user uploaded a file to an S3 bucket? I can see properties like 'last modified', but not the IAM user.
For my use case, I can't add random metadata because the file is being uploaded by Cyberduck.
Thanks!
John
You can try hit the Get Bucket REST API programmatically or with something like curl. The Contents > Owner > DisplayName key might be what you're looking for.
Sample response:
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>bucket</Name>
<Prefix/>
<Marker/>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>my-image.jpg</Key>
<LastModified>2009-10-12T17:50:30.000Z</LastModified>
<ETag>"fba9dede5f27731c9771645a39863328"</ETag>
<Size>434234</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID>
<DisplayName>mtd#amazon.com</DisplayName>
</Owner>
</Contents>
</ListBucketResult>
Have you looked at S3 server access logging? I believe the logs will include the canonical user ID of the uploader, unless anonymous. Not quite sure how you turn that into an IAM access key, but perhaps there's a way.
Or you could look at CloudTrail logs (assuming that you have CloudTrail enabled). They should show you the access key used to perform the upload.
Or I guess you could set up different upload locations, one per authorized IAM user, and then add appropriate policies so that only user X could upload to his specific location.
[Added] You might also want to specify a bucket policy that requires uploaders to give you, the bucket owner, full control of the object. Then you can query the ACLs of the object and determine the owner (which will be the original uploader).
If the file to the S3 bucket is uploaded by POST operation and you can grab the details information using Amazon Athena > Query editor. The query would look like this:
SELECT bucketowner, Requester, RemoteIP, Operation, Key, HTTPStatus, ErrorCode, RequestDateTime
FROM "databasename"."tablename"
WHERE (Operation='REST.PUT.OBJECT' OR Operation = 'REST.POST.UPLOAD')
AND parse_datetime(RequestDateTime,'dd/MMM/yyyy:HH:mm:ss Z')
BETWEEN parse_datetime('2021-11-11:00:42:42','yyyy-MM-dd:HH:mm:ss')
AND parse_datetime('2021-12-31:00:42:42','yyyy-MM-dd:HH:mm:ss')
AND Requester='arn:aws:sts::accoint-number::assumed-role/ROLE-NAME/email'
For more information on athena
Related
I created S3 bucket and I have added one user in IAM. Suppose my bucuket name sample123. When in bucket ploicy I mentioned resource like below statement user is not able to upload document.
Resource": "arn:aws:s3:::sample123"
But when resource is mentioned in policy as below , the user is able to upload document.
Resource": [ "arn:aws:s3:::sample123","arn:aws:s3:::sample123/*"]
what adding /* to ARN will do in policy. Note : I gave full bucket permissions to the user.
sample123/* means the all objects in sample123 bucket.
doc of S3 ARN examples says:
The ARN format for Amazon S3 resources reduces to the following:
arn:aws:s3:::bucket_name/key_name
...
The following ARN uses the wildcard * in the relative-ID part of the
ARN to identify all objects in the examplebucket bucket.
arn:aws:s3:::examplebucket/*
Also refer Example of S3 Actions with policy
I'm really struggling with how to do data transfer from Amazon S3 bucket to Redshift with COPY command.
So far, I created an IAM User and 'AmazonS3ReadOnlyAccess' policy is assigned. But when I call COPY command likes following, Access Denied Error is always returned.
copy my_table from 's3://s3.ap-northeast-2.amazonaws.com/mybucket/myobject' credentials 'aws_access_key_id=<...>;aws_secret_access_key=<...>' REGION'ap-northeast-2' delimiter '|';
Error:
Amazon Invalid operation: S3ServiceException:Access Denied,Status 403,Error AccessDenied,Rid EB18FDE35E1E0CAB,ExtRid ,CanRetry 1
Details: -----------------------------------------------
error: S3ServiceException:Access Denied,Status 403,Error AccessDenied,Rid EB18FDE35E1E0CAB,ExtRid ,CanRetry 1
code: 8001
context: Listing bucket=s3.ap-northeast-2.amazonaws.com prefix=mybucket/myobject
query: 1311463
location: s3_utility.cpp:542
process: padbmaster [pid=4527]
-----------------------------------------------;
Is there anyone can give me some clues or advice?
Thanks a lot!
Remove the endpoint s3.ap-northeast-2.amazonaws.com from the S3 path:
COPY my_table
FROM 's3://mybucket/myobject'
CREDENTIALS ''
REGION 'ap-northeast-2'
DELIMITER '|'
;
(See the examples in the documentation.) While the Access Denied error is definitely misleading, the returned message gives some hint as to what went wrong:
bucket=s3.ap-northeast-2.amazonaws.com
prefix=mybucket/myobject
We'd expect to see bucket=mybucket and prefix=myobject, though.
Check encription of bucket.
According doc : https://docs.aws.amazon.com/en_us/redshift/latest/dg/c_loading-encrypted-files.html
The COPY command automatically recognizes and loads files encrypted using SSE-S3 and SSE-KMS.
Check kms: rules on you key|role
If files from EMR, check Security configurations for S3.
your redshift cluster role does not have right to access to the S3 bucket. make sure the role you use for redshift has access to the bucket and bucket does not have policy that blocks the access
My folder configuration in Amazon S3 looks like BucketName/R/A/123 now i want to add another folder under my Bucket and want to save data as BucketName/**I**/A/123. When i try to save my data, i get an error:
<Error><Code>AccessDenied</Code><Message>Invalid according to Policy: Policy Condition failed: ["starts-with", "$key", "R/"]</Message></Error>
I understand i need to give some permission for folder I to be created under this Bucket but i am struggling to find where.
Can anyone point me where i need to make a change.
Thanks
I understand i need to give some permission for folder I to be created under this Bucket but i am struggling to find where.
No, not according to this error message.
You appear to be supplying this policy in your code...
["starts-with", "$key", "R/"]
...and this policy -- part of a form post upload policy document your code is generating -- is telling S3 that you want it to deny an upload with a key that doesn't stat with "R/" ...
If this isn't what you want, you need to change the policy your code is supplying, so that it allows you to name the key the way you want to name it.
We have a service where customers of ours give us access to their S3 buckets and we push items into those S3 buckets. We need to be able to do 2 things:
Set the permissions on the item to be publicly readable
Set the owner of the bucket to have full permissions to the item
Here is what I already know:
I cannot have 2 canned-ACLs with the PUT
Problem:
I "could" set ACL headers, but AFAIK there is no way to set the "owner-has-full-permissions" via header without knowing information about the owner (Like cannonical_id or email), correct? Is there a "uri" version of "owner-has-full-permissions" like there is for "public-read" (e.g. "http://acs.amazonaws.com/groups/global/AllUsers")?
I don't want to have to make 2 separate calls (one to get the buckets owner info) and one to put the item with both permissions.
I had the same problem, the following code can get the permissions you require.
AccessControlList accessControlList = new AccessControlList();
accessControlList.grantPermission(GroupGrantee.AllUsers, Permission.Read);
accessControlList.grantPermission(new CanonicalGrantee(s3Client.getS3AccountOwner()
.getId()), Permission.FullControl);
putReq.setAccessControlList(accessControlList);
Is there any way I can ask the CloudFront API for the name of the bucket it uses on Amazon S3?
This is possible via the GET Distribution action:
To get the information about a distribution, you do a GET on the
2012-03-15/distribution/ resource.
Have a look at the sample syntax in the Responses section, which specifically includes fragments for either S3Origin or CustomOrigin, e.g. abbreviated:
<Distribution xmlns="http://cloudfront.amazonaws.com/doc/2012-03-15/">
<!-- ... -->
<DistributionConfig>
<S3Origin>
<DNSName>myawsbucket.s3.amazonaws.com</DNSName>
<OriginAccessIdentity>origin-access-identity/cloudfront/E127EXAMPLE51Z</OriginAccessIdentity>
</S3Origin>
<!-- ... -->
</DistributionConfig>
</Distribution>
Please note that The S3Origin element is returned only if you use an Amazon S3 origin for your distribution, whereas The CustomOrigin element is returned only if you use a custom origin for your distribution. Furthermore, for more information about the CustomOrigin element and the S3Origin element, see DistributionConfig Complex Type.