I tried to read keys which are encrypted by using AWS KMS, I first hit the following error.
S3ResponseError: 400 Bad Request
InvalidArgumentRequests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.Authorizationnull1D584F77747CBB01LQIxPHmUGGDMnnI45xqWHtrB1+96tc7pDIEi6bVEE5i425SRypqeBXzvsH0CqPzwJe4xVv1UjhQ=
After setting os.environ['S3_USE_SIGV4'] = 'True', the above 400 error is gone, but now I hit the 403 error.
S3ResponseError: 403 Forbidden
May I ask if anyone hit the same issue before?
This error is caused by a wrong S3 hostname which was s3-ap-southeast-1.s3.amazonaws.com but should be s3-ap-southeast-1.amazonaws.com
Related
I have set up MinIO behind a reverse proxy in EKS. Everything worked well until MinIO was updated to RELEASE.2021-11-03T03-36-36Z. Now I am getting the following error when trying to access my MinIO bucket using the mc command-line utility: mc: <ERROR> Unable to list folder. The request signature we calculated does not match the signature you provided. Check your key and signing method.
mc version is RELEASE.2021-11-16T20-37-36Z. When I port-forward the MinIO container to localhost and access it in a browser at http://localhost:9001 I can get to it, but I can't log in anymore. I get the error:
Invalid Login, 403 Forbidden`. This is seen in my MinIO container
It also logs the following:
API: SYSTEM()
Time: 03:19:57 UTC 11/23/2021
DeploymentID: 60a8ed7a-7448-4a3d-9220-ff823facd54e
Error: The request signature we calculated does not match the signature you provided. Check your key and signing method. (*errors.errorString)
requestHeaders={"method":"POST","reqURI":"/minio/admin/v3/update?updateURL=","header":{"Authorization":["AWS4-HMAC-SHA256 Credential=<credential-scrubbed>/20211123//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=37850012ca8d27498793c514aa826f1c29b19ceae96057b9d46e24599cc8081b"],"Connection":["keep-alive"],"Content-Length":["0"],"Host":["<host-info-scrubbed>"],"User-Agent":["MinIO (darwin; amd64) madmin-go/0.0.1 mc/RELEASE.2021-11-16T20-37-36Z"],"X-Amz-Content-Sha256":["<scrubbed>"],"X-Amz-Date":["20211123T031957Z"],"X-Forwarded-For":["10.192.57.142"],"X-Forwarded-Host":["<host-info-scurbbed>"],"X-Forwarded-Path":["/minio/admin/v3/update"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Real-Ip":["10.192.57.142"]}}
5: cmd/auth-handler.go:154:cmd.validateAdminSignature()
4: cmd/auth-handler.go:165:cmd.checkAdminRequestAuth()
3: cmd/admin-handler-utils.go:41:cmd.validateAdminReq()
2: cmd/admin-handlers.go:87:cmd.adminAPIHandlers.ServerUpdateHandler()
1: net/http/server.go:2046:http.HandlerFunc.ServeHTTP()
When checking the proxy logs (NGINX), I see:
10.192.57.142 - - [24/Nov/2021:21:18:17 +0000] "GET / HTTP/1.1" 403 334 "-" "MinIO (darwin; amd64) minio-go/v7.0.16 mc/RELEASE.2021-11-16T20-37-36Z"
Any suggestions or advice on what I can do to resolve this would be great! I'm using the mc client on OSX.
I can successfully upload files to my Minio server using mc command line client (logged in as root):
./mc cp roobina.jpg minio/mag
roobina.jpg: 63.50 KiB / 63.50 KiB
But when I try to upload a file to a bucket using minio's own web interface I receive this error:
Unauthorized request.
When using api (in a php application using AmazonS3 libraries), I receive this error:
Error:Error executing "PutObject" on "https://s3.***.net/clbu/public/4d/4b/d1ad580690058a636ad58e5af931541336ec.jpg"; AWS HTTP error: Client error: `PUT https://s3.***.net/clbu/public/4d/4b/d1ad580690058a636ad58e5af931541336ec.jpg` resulted in a `403 Forbidden` response:
Forbidden (truncated...) Unable to parse error information from response - Error parsing XML: String could not be parsed as XML
Could someone please help?
After looking at different possible causes, I found that mod_security of apache (used as reverse proxy of minio:9000) was interfering with uploads causing the problem.
I disabled mod_security on the reverse proxy account and the problem is now solved.
I was simply following the "get started" tutorial here
But I failed at "Step 4 Add a Bucket Policy to Allow Public Reads". It always complains "access denied" with a red error icon.
I am not able to set it via command line either. Here is the command I use:
aws s3api put-bucket-policy --bucket bucket-name --policy
file://bucket-policy.json
Here is the error I got:
An error occurred (AccessDenied) when calling the PutBucketPolicy
operation: Access Denied
The issue was, you have to uncheck the boxes under permissions -> public access settings. Amazon failed to mention this in their tutorial. Bad tutorial.
I recently implemented AWS Signature version 4 using the REST API. This is verified by an extensive regression test working perfectly.
The problem I'm experiencing is that the regression test succeeds when run against a bucket residing in the eu-central-1 region, but consistently fails with the Accessed Denied error message for buckets residing in us-east-1 or us-west-2.
Here are snippets from successful and failed attempts.
eu-central-1 : successful
HTTP request:
GET./
host:s3.eu-central-1.amazonaws.com.x-amz-content-sha256:e3b0...b855.x-amz-date:Wed, 25 May 2016 03:13:21 +0000
host;x-amz-content-sha256;x-amz-date.e3b0...b855
Signed string:
AWS4-HMAC-SHA256
Credential=AKIAJZN7UY6XHIZPWIKQ/20160525/eu-central-1/s3/aws4_request,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=cf5f...4dc8
Server response:
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult
xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Owner>
<ID>100a...a575</ID>
</Owner>
<Buckets>
<Bucket>
. . .
</Bucket>
</Buckets>
</ListAllMyBucketsResult>
us-east-1 : failed
HTTP request:
GET./
host:s3.us-east-1.amazonaws.com.x-amz-content-sha256:e3b0...b855.x-amz-date:Wed, 25 May 2016 03:02:27 +0000
host;x-amz-content-sha256;x-amz-date.e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Signed string:
AWS4-HMAC-SHA256
Credential=AKIAJZN7UY6XHIZPWIKQ/20160525/us-east-1/s3/aws4_request,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=01e97...4d00
Server response:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>92EEF2A86ECA88EF</RequestId>
<HostId>i3wTU6OzBrlX89xR4KnnezBx1Tb2IGN2wtgPJMRtKLjHxF/B6VdCQqPz1279J7e5</HostId>
</Error>
us-west-2 : failed
HTTP request:
GET./
host:s3.us-west-2.amazonaws.com.x-amz-content-sha256:e3b0...b855.x-amz-date:Wed, 25 May 2016 07:04:47 +0000
host;x-amz-content-sha256;x-amz-date.e3b0...b855
Signed string:
AWS4-HMAC-SHA256
Credential=AKIAJZN7UY6XHIZPWIKQ/20160525/us-west-2/s3/aws4_request,
SignedHeaders=host;x-amz-content-sha256;x-amz-date,
Signature=cf70...36b9
Server response:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>DB143DBF0F316EB8</RequestId>
<HostId>5hWJ0AHM466QcT+BK4UaEFpqXFNaJFEuAPlN/ZZPBhL+NDYBoGaySRkXQ3BRdyfy9PBDuSb0oHA=</HostId>
</Error>
Attempts made to date include:
I found references (like here) where when using US Standard (i.e., us-east-1) the REST endpoint should not include "us-east-1". I have not yet found this written officially. I therefore created a us-west-2 bucket, in the hope that the REST endpoint needs to contain "us-west-2", but that also fails.
I searched on Google and StackOverflow for possible reasons for "Access Denied", which led me to adding a bucket policy that gives permissions to all -- to no avail.
The permissions of the EU and US accounts in the AWS console look the same, so no hint there, yet.
I added logging to the buckets in the hope of seeing a failure entry, but nothing is logged until authentication is completed.
Does anyone have an idea why AWS v4 authentication will consistently succeed for an eu-central-1 bucket, but equally fail for us-east-1 and us-east-2 buckets?
Here's your issue.
For unknown reasons,¹ eu-central-1 is an oddball in S3. The REST endpoint works with two variations in hostname: bucket.s3.eu-central-1.amazonaws.com or bucket.s3-eu-central-1.amazonaws.com.
The difference is the dot or dash after s3.
All other regions (as of now) except us-east-1 and ap-northeast-2 (which is just like eu-central-1) work only with the dash after s3, e.g. bucket.s3-us-west-2.amazonaws.com... not with a dot.
And us-east-1 expects either bucket.s3.amazonaws.com or bucket.s3-external-1.amazonaws.com.
And finally, any region will work with just bucket.s3.amazonaws.com within a few minutes after the original creation of a bucket, because the DNS is integrated with the bucket location database and automatically routes requests to the right place, for each bucket.
But note that when you sign the requests, you always use the actual region name in the signing algorithm itself -- not the endpoint -- as you appear to already be doing.
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
¹I'll speculate that this convention is actually the "new normal" for new regions -- it's more consistent with other AWS services. S3 is one of the oldest, so it makes sense that legacy design decisions are more likely to exist, as seems to be the case, here.
The setting
I have an Amazon Cloudfront distribution that was originally set as secured. Objects in this distribution required a URL signing. For example, a valid URL used to be of the following format:
https://d1stsppuecoabc.cloudfront.net/images/TheImage.jpg?Expires=1413119282&Signature=NLLRTVVmzyTEzhm-ugpRymi~nM2v97vxoZV5K9sCd4d7~PhgWINoTUVBElkWehIWqLMIAq0S2HWU9ak5XIwNN9B57mwWlsuOleB~XBN1A-5kzwLr7pSM5UzGn4zn6GRiH-qb2zEoE2Fz9MnD9Zc5nMoh2XXwawMvWG7EYInK1m~X9LXfDvNaOO5iY7xY4HyIS-Q~xYHWUnt0TgcHJ8cE9xrSiwP1qX3B8lEUtMkvVbyLw__&Key-Pair-Id=APKAI7F5R77FFNFWGABC
The distribution points to an S3 bucket that also used to be secured (it only allowed access through the cloudfront).
What happened
At some point, the URL singing expired and would return a 403.
Since we no longer need to keep the same security level, I recently changed the setting of the cloudfront distribution and of the S3 bucket it is pointing to, both to be public.
I then tried to invalidate objects in this distribution. Invalidation did not throw any errors, however the invalidation did not seem to succeed. Requests to the same cloudfront URL (with or without the query string) still return 403.
The response header looks like:
HTTP/1.1 403 Forbidden
Server: CloudFront
Date: Mon, 18 Aug 2014 15:16:08 GMT
Content-Type: text/xml
Content-Length: 110
Connection: keep-alive
X-Cache: Error from cloudfront
Via: 1.1 3abf650c7bf73e47515000bddf3f04a0.cloudfront.net (CloudFront)
X-Amz-Cf-Id: j1CszSXz0DO-IxFvHWyqkDSdO462LwkfLY0muRDrULU7zT_W4HuZ2B==
Things I tried
I tried to set another cloudfront distribution that points to the same S3 as origin server. Requests to the same object in the new distribution were successful.
The question
Did anyone encounter the same situation where a cloudfront URL that returns 403 cannot be invalidated? Is there any reason why wouldn't the object get invalidated?
Thanks for your help!
First check that Invalidation is not in progress. If it is then wait till it is completed.
If you are accessing S3 Object through CloudFront using Public URL then you need to have public read permission on that S3 Object.
If you are trying to access S3 Object through CloudFront using Signed URL then make sure that time that are mention while generating sign url, must be greater then current time.