Is there any way I can ask the CloudFront API for the name of the bucket it uses on Amazon S3?
This is possible via the GET Distribution action:
To get the information about a distribution, you do a GET on the
2012-03-15/distribution/ resource.
Have a look at the sample syntax in the Responses section, which specifically includes fragments for either S3Origin or CustomOrigin, e.g. abbreviated:
<Distribution xmlns="http://cloudfront.amazonaws.com/doc/2012-03-15/">
<!-- ... -->
<DistributionConfig>
<S3Origin>
<DNSName>myawsbucket.s3.amazonaws.com</DNSName>
<OriginAccessIdentity>origin-access-identity/cloudfront/E127EXAMPLE51Z</OriginAccessIdentity>
</S3Origin>
<!-- ... -->
</DistributionConfig>
</Distribution>
Please note that The S3Origin element is returned only if you use an Amazon S3 origin for your distribution, whereas The CustomOrigin element is returned only if you use a custom origin for your distribution. Furthermore, for more information about the CustomOrigin element and the S3Origin element, see DistributionConfig Complex Type.
Related
[nodejs sdk, s3.putBucketPolicy, error handling]
Is there a way to determine (one or more) invalid arn's (invalid account numbers) from error object returned by S3 putBucketPolicy call? Error statusCode is 400 however, trying to figure out which set of principals are invalid.
To clarify further I am not looking for validating role, root ARN patterns. More like, one or more account number(s) thats not correct. Can we extract that from error object or else where?
There are couple of ways to validate an arn:
Using organizations service
Using ARN as principal and apply a bucket policy for a dummy bucket. Let AWS SDK validate it for you, clean up afterwards.
I created S3 bucket and I have added one user in IAM. Suppose my bucuket name sample123. When in bucket ploicy I mentioned resource like below statement user is not able to upload document.
Resource": "arn:aws:s3:::sample123"
But when resource is mentioned in policy as below , the user is able to upload document.
Resource": [ "arn:aws:s3:::sample123","arn:aws:s3:::sample123/*"]
what adding /* to ARN will do in policy. Note : I gave full bucket permissions to the user.
sample123/* means the all objects in sample123 bucket.
doc of S3 ARN examples says:
The ARN format for Amazon S3 resources reduces to the following:
arn:aws:s3:::bucket_name/key_name
...
The following ARN uses the wildcard * in the relative-ID part of the
ARN to identify all objects in the examplebucket bucket.
arn:aws:s3:::examplebucket/*
Also refer Example of S3 Actions with policy
I am new to NIFI and want to push data from Kafka to an S3 bucket. I am using the PutS3Object processor and can push data to S3 if I hard code the Bucket value as mphdf/orderEvent, but I want to specify the buckets based on a field in the content of the FlowFile, which is in Json. So, if the Json content is this {"menu": {"type": "file","value": "File"}}, can I have the value for the Bucket property as as mphdf/$.menu.type? I have tried to do this and get the error below. I want to know if there is a way to access the FlowFile content with the PutS3Object processor and make Bucket names configurable or will I have to build my own processor?
ERROR [Timer-Driven Process Thread-10]
o.a.nifi.processors.aws.s3.PutS3Object
com.amazonaws.services.s3.model.AmazonS3Exception: The XML you
provided was not well-formed or did not validate against our
published schema (Service: Amazon S3; Status Code: 400; Error Code:
MalformedXML; Request ID: 77DF07828CBA0E5F)
I believe what you want to do is use an EvaluateJSONPath processor, which evaluates arbitrary JSONPath expressions against the JSON content and extracts the results to flowfile attributes. You can then reference the flowfile attribute using NiFi Expression Language in the PutS3Object configuration (see your first property Object Key which references ${filename}). In this way, you would evaluate $.menu.type and store it into an attribute menuType in the EvaluateJSONPath processor, then in PutS3Object you would have Bucket be mphdf/${menuType}.
You might have to play around with it a bit but off the top of my head I think that should work.
My folder configuration in Amazon S3 looks like BucketName/R/A/123 now i want to add another folder under my Bucket and want to save data as BucketName/**I**/A/123. When i try to save my data, i get an error:
<Error><Code>AccessDenied</Code><Message>Invalid according to Policy: Policy Condition failed: ["starts-with", "$key", "R/"]</Message></Error>
I understand i need to give some permission for folder I to be created under this Bucket but i am struggling to find where.
Can anyone point me where i need to make a change.
Thanks
I understand i need to give some permission for folder I to be created under this Bucket but i am struggling to find where.
No, not according to this error message.
You appear to be supplying this policy in your code...
["starts-with", "$key", "R/"]
...and this policy -- part of a form post upload policy document your code is generating -- is telling S3 that you want it to deny an upload with a key that doesn't stat with "R/" ...
If this isn't what you want, you need to change the policy your code is supplying, so that it allows you to name the key the way you want to name it.
Just a quick one - how do you identify which IAM user uploaded a file to an S3 bucket? I can see properties like 'last modified', but not the IAM user.
For my use case, I can't add random metadata because the file is being uploaded by Cyberduck.
Thanks!
John
You can try hit the Get Bucket REST API programmatically or with something like curl. The Contents > Owner > DisplayName key might be what you're looking for.
Sample response:
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>bucket</Name>
<Prefix/>
<Marker/>
<MaxKeys>1000</MaxKeys>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>my-image.jpg</Key>
<LastModified>2009-10-12T17:50:30.000Z</LastModified>
<ETag>"fba9dede5f27731c9771645a39863328"</ETag>
<Size>434234</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID>
<DisplayName>mtd#amazon.com</DisplayName>
</Owner>
</Contents>
</ListBucketResult>
Have you looked at S3 server access logging? I believe the logs will include the canonical user ID of the uploader, unless anonymous. Not quite sure how you turn that into an IAM access key, but perhaps there's a way.
Or you could look at CloudTrail logs (assuming that you have CloudTrail enabled). They should show you the access key used to perform the upload.
Or I guess you could set up different upload locations, one per authorized IAM user, and then add appropriate policies so that only user X could upload to his specific location.
[Added] You might also want to specify a bucket policy that requires uploaders to give you, the bucket owner, full control of the object. Then you can query the ACLs of the object and determine the owner (which will be the original uploader).
If the file to the S3 bucket is uploaded by POST operation and you can grab the details information using Amazon Athena > Query editor. The query would look like this:
SELECT bucketowner, Requester, RemoteIP, Operation, Key, HTTPStatus, ErrorCode, RequestDateTime
FROM "databasename"."tablename"
WHERE (Operation='REST.PUT.OBJECT' OR Operation = 'REST.POST.UPLOAD')
AND parse_datetime(RequestDateTime,'dd/MMM/yyyy:HH:mm:ss Z')
BETWEEN parse_datetime('2021-11-11:00:42:42','yyyy-MM-dd:HH:mm:ss')
AND parse_datetime('2021-12-31:00:42:42','yyyy-MM-dd:HH:mm:ss')
AND Requester='arn:aws:sts::accoint-number::assumed-role/ROLE-NAME/email'
For more information on athena