For some reason I don't get Accept-Ranges in s3 response parameters. Example curl headers response:
HTTP/1.1 200 OK
x-amz-id-2: ***
x-amz-request-id: ***
Date: Thu, 24 Oct 2019 12:40:51 GMT
Last-Modified: Thu, 24 Oct 2019 10:35:55 GMT
ETag: "770dc36e888906bfc63641cbaf7775ce"
Content-Type: application/octet-stream
Content-Length: 1846218
Server: AmazonS3
Any ideas why? Also I have CORS setuped as that:
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<ExposeHeader>Accept-Ranges</ExposeHeader>
<ExposeHeader>Content-Range</ExposeHeader>
<ExposeHeader>Content-Encoding</ExposeHeader>
<ExposeHeader>Content-Length</ExposeHeader>
<ExposeHeader>Access-Control-Expose-Headers</ExposeHeader>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
I can't see the url of the request you are making but ..
AWS S3 does not send the HTTP header Accept-Ranges: bytes if you are using the default endpoint URL. Use the following format as the origin URL instead which serves the required Accept-Ranges header field: .s3.amazonaws.com
From KeyCDN
Hope that helps
Related
Just wanted to know if the GET Bucket op response ever skips the Content-Length header. I tested this and i saw that there was no Content-Length header in the response for GET Bucket op.
How does an application reading the response understand where the body of the response ends if the response doesn't contain Content-Length header?
Request-Response Snippet:
GET /?max-keys=1000&prefix&delimiter=%2F HTTP/1.1
Date: Sat, 09 Apr 2016 18:27:23 GMT
x-amz-request-payer: requester
Authorization: AWS AKIAIP3KAUILC4GG7A2A:UG3bGvIjayrxrkxEX1mfrvETy/M=
Connection: Keep-Alive
User-Agent: Cyberduck/4.9.19632 (Mac OS X/10.10.5) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: yg76HSq5j0mi0oR6dXF8ZfGq722kHBWiMQmNvXPqiLxr1S4nGj5GVn1RVrPQrOUfNynxxaMSYEY=
x-amz-request-id: B4468E68E10B6AEF
Date: Sat, 09 Apr 2016 18:27:25 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Server: AmazonS3
Connection: close
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">......</ListBucketResult>
Thanks!
The Content-Length header is optional in response. And it may not reflect the real content-length even if it presents. Think about gzipped response. So to answer the question: When no Content-Length is received, the client keeps reading until the server closes the connection.
In Java, keep calling InputStream.read() until it returns -1.
Is the Content-Length header required for a HTTP/1.0 response?
I followed this tutorial http://blog.fineuploader.com/2014/01/15/uploads-without-any-server-code/ and am making good progress, but i'm stumped again.
Here is the CORS policy for my bucket (i'm assuming this is where the error is):
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>https://www.xxxdomainxxx.fr</AllowedOrigin>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>DELETE</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<ExposeHeader>ETag</ExposeHeader>
<AllowedHeader>content-type</AllowedHeader>
<AllowedHeader>origin</AllowedHeader>
<AllowedHeader>x-amz-acl</AllowedHeader>
<AllowedHeader>x-amz-meta-qqfilename</AllowedHeader>
<AllowedHeader>x-amz-date</AllowedHeader>
<AllowedHeader>authorization</AllowedHeader>
<AllowedHeader>x-amz-security-token</AllowedHeader>
</CORSRule>
</CORSConfiguration>
The upload goes fine (I see the progress bar) and it goes through the entire file, but at the end of the upload, the UI switches to "processing" and then i get an "access denied" message and no file is in the bucket. The console printed:
Failed to load resource: the server responded with a status of 403 (Forbidden) (xxxxxxx.s3.amazonaws.com, line 0)
I'm getting a 403 error back from AWS and i'm not sure how to debug this. See anything missing?
Thanks for any pointers.
Update 1
I figured i'd try with a dumb down version of the CORS file to start with
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
But i'm getting the same error.
Here is the failing POST request:
key test test/c8491b98-284a-4d5d-90d4-f6ec7151bc1d.diff
AWSAccessKeyId XXXXXXXXXXX
success_action_status 200
x-amz-security-token XXXXXXXX
acl public-read
x-amz-meta-qqfilename opentok.diff
policy XXXXXX
signature XXXXXXX
file opentok.diff
And the response
HTTP/1.1 403 Forbidden
Access-Control-Allow-Origin *
Access-Control-Allow-Methods GET, POST, PUT
Access-Control-Max-Age 3000
Vary Origin, Access-Control-Request-Headers, Access-Control-Request-Method
x-amz-request-id 8B619A5A96A954F6
x-amz-id-2 ZUPdtFRIdSKDK0ealKUKUCtHDW3GkNU5ZVZPDxlXPi/9J2oZiNcV3TltougJuhXnzY/BlbZrc1c=
Content-Type application/xml
Transfer-Encoding chunked
Date Wed, 07 Oct 2015 08:11:24 GMT
Server AmazonS3
The issue was not with the CORS configuration, but with the policy used for the bucket. The resource string was
arn:was:s3:::bucketName
and needed to be
arn:was:s3:::bucketName/*
The devil is in the details as usual...
I can get Amazon S3 to pass CORS headers with http, but not with https. How do I get it to work with both? What if we're using Akamai as a CDN?
Here's my bucket configuration:
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>https://*</AllowedOrigin>
<AllowedOrigin>http://*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
Here's my test. The only difference between these is that one uses http, the other uses https. Both resources load just fine in the browser, but I would like to use them in a CORS setting where it could be https.
pnore#mbp> curl -i -H "Origin: http://example.com" -H "Access-Control-Request-Method: GET" -H 'Pragma: no-cache' --verbose http://my.custom.domain/path/to/file/in/bucket | head -n 15
* Adding handle: conn: 0x7fee83803a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fee83803a00) send_pipe: 1, recv_pipe: 0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to my.custom.domain port 80 (#0)
* Trying 23.23.23.23...
* Connected to my.custom.domain (23.23.23.23) port 80 (#0)
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0> GET /path/to/file/in/bucket HTTP/1.1
> User-Agent: curl/7.30.0
> Host: my.custom.domain
> Accept: */*
> Origin: http://example.com
> Access-Control-Request-Method: GET
> Pragma: no-cache
>
< HTTP/1.1 200 OK
< x-amz-id-2: random
< x-amz-request-id: random
< Access-Control-Allow-Origin: http://example.com
< Access-Control-Allow-Methods: GET
< Access-Control-Max-Age: 3000
< Access-Control-Allow-Credentials: true
< Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
< Last-Modified: Tue, 10 Jun 2014 15:34:38 GMT
< ETag: "random"
< Accept-Ranges: bytes
< Content-Type: video/webm
< Content-Length: 8981905
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
< Date: Fri, 19 Jun 2015 21:31:22 GMT
< Connection: keep-alive
<
{ [data not shown]
HTTP/1.1 200 OK
x-amz-id-2: random
x-amz-request-id: random
Access-Control-Allow-Origin: http://example.com
Access-Control-Allow-Methods: GET
Access-Control-Max-Age: 3000
Access-Control-Allow-Credentials: true
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Last-Modified: Tue, 10 Jun 2014 15:34:38 GMT
ETag: "random"
Accept-Ranges: bytes
Content-Type: video/webm
Content-Length: 8981905
Server: AmazonS3
Date: Fri, 19 Jun 2015 21:31:22 GMT
...
pnore#mbp> curl -i -H "Origin: http://example.com" -H "Access-Control-Request-Method: GET" -H 'Pragma: no-cache' --verbose https://my.custom.comain/path/to/file/in/bucket | head -n 15
* Adding handle: conn: 0x7fd24380c000
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fd24380c000) send_pipe: 1, recv_pipe: 0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to my.custom.domain port 443 (#0)
* Trying 23.23.23.23...
* Connected to my.custom.domain (23.23.23.23) port 443 (#0)
* TLS 1.2 connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate: my.custom.domain
* Server certificate: GeoTrust SSL CA - G4
* Server certificate: GeoTrust Global CA
> GET /path/to/file/in/bucket HTTP/1.1
> User-Agent: curl/7.30.0
> Host: my.custom.domain
> Accept: */*
> Origin: http://example.com
> Access-Control-Request-Method: GET
> Pragma: no-cache
>
< HTTP/1.1 200 OK
< x-amz-id-2:
< x-amz-request-id:
< Last-Modified: Tue, 10 Jun 2014 15:34:38 GMT
< ETag: "random"
< Accept-Ranges: bytes
< Content-Type: video/webm
< Content-Length: 8981905
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
< Date: Fri, 19 Jun 2015 21:31:29 GMT
< Connection: keep-alive
<
{ [data not shown]
HTTP/1.1 200 OK
x-amz-id-2:
x-amz-request-id:
Last-Modified: Tue, 10 Jun 2014 15:34:38 GMT
ETag: "random"
Accept-Ranges: bytes
Content-Type: video/webm
Content-Length: 8981905
Server: AmazonS3
Date: Fri, 19 Jun 2015 21:31:29 GMT
Connection: keep-alive
...
Note that the first request contains the desired Access-Control-Allow-Origin header, and the second does not.
I've also tried <AllowedOrigin>*</AllowedOrigin> and using different <CORSRule> blocks for each <AllowedOrigin>.
References I've checked:
Getting S3 CORS Access-Control-Allow-Origin to dynamically echo requesting domain 1
Amazon S3 CORS (Cross-Origin Resource Sharing) and Firefox cross-domain font loading 1
Getting S3 CORS Access-Control-Allow-Origin to dynamically echo requesting domain
Aws S3 Bucket CORS configuration is not saving properly
http://blog.errorception.com/2014/11/enabling-cors-on-amazon-cloudfront-with.html
Correct S3 + Cloudfront CORS Configuration?
https://forums.aws.amazon.com/thread.jspa?messageID=377513
How to Configure SSL for Amazon S3 bucket
HTTPS for Amazon S3 static website
SSL on Amazon S3 as "static website"
I couldn't find documentation that explicitly mentioned it, but it appears that the CORS configuration for the bucket only allows one <AllowedOrigin> per <CORSRule> element entry. You are allowed up to 100 <CORSRule> entries in the configuration. Therefore, in order to get your configuration to support both http and https you should create two <CORSRule> entries, like so:
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>https://*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
<CORSRule>
<AllowedOrigin>http://*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
FWIW, I have not tried it, but the configuration may also support a protocol agnostic format, e.g. simply <AllowedOrigin>//*</AllowedOrigin>.
As far as I am able to tell, the only configuration which should be necessary to get s3 to return the Access-Control-Allow-Origin header is setting a CORS policy and sending the Origin header.
Here is my CORS policy:
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>HEAD</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>
When I send this request:
GET /dots.png HTTP/1.1
Host: dta-test.s3.amazonaws.com
Origin: https://www.test.com
Cache-Control: no-cache
I get a response with these headers:
Accept-Ranges →bytes
Cache-Control →public
Content-Length →893
Content-Type →image/png
Date →Thu, 09 Jan 2014 22:27:20 GMT
ETag →"5345e98a3abcb1057d1a427551d8d702"
Last-Modified →Thu, 09 Jan 2014 19:27:56 GMT
Server →AmazonS3
x-amz-id-2 →2AFNGowXCkLUJa09ZNERXqzxn5IwnygTKCXut0m0gvpapjxXn/kAPYQNvv4pYvVy
x-amz-request-id →71B183AF938A075D
As far as I am aware, the Access-Control-Allow-Origin header should be included in this response. What am I missing?
Edit:
This is what I'm missing: I was using the Postman Chrome extension. Chrome sanitizes the Origin header when making requests, even from extensions. It throws the error Refused to set unsafe header "Origin", which I missed.
We can see with curl that this is actually working:
curl -H "Origin: https://www.test.com" -I "https://dta-test.s3.amazonaws.com/dots.png" -s
HTTP/1.1 200 OK
x-amz-id-2: NnRAOdCOpMCtZ8Jk1bpnuRASb0K/gzM0Vv/6D28C6grcbEZoe2OC0cuu3SMwDaXe
x-amz-request-id: 70110890812CAC58
Date: Thu, 09 Jan 2014 23:03:41 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, HEAD
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Cache-Control: public
Last-Modified: Thu, 09 Jan 2014 19:27:56 GMT
ETag: "5345e98a3abcb1057d1a427551d8d702"
Accept-Ranges: bytes
Content-Type: image/png
Content-Length: 893
Server: AmazonS3
For some reason I cannot access the response header of "Content-Range" anymore... Therefore it's impossible to determine the file size of a resource using XHR.
I get error Refused to get unsafe header "Content-Range" on Chrome in this line:
var cr = this.getResponseHeader('Content-Range');
Here's the CORS config:
<?xml version="1.0" ?>
<CorsConfig>
<Cors>
<Origins>
<Origin>*</Origin>
</Origins>
<Methods>
<Method>GET</Method>
<Method>HEAD</Method>
<Method>DELETE</Method>
</Methods>
<ResponseHeaders>
<ResponseHeader>x-goog-meta-foo1</ResponseHeader>
<ResponseHeader>origin</ResponseHeader>
<ResponseHeader>range</ResponseHeader>
<ResponseHeader>Content-Range</ResponseHeader>
<ResponseHeader>Content-Length</ResponseHeader>
</ResponseHeaders>
<MaxAgeSec>1800</MaxAgeSec>
</Cors>
CURL output:
$ curl -H "Origin: http://peer5.com" http://commondatastorage.googleapis.com/peer5_vod/wind2_orig.mp4 -s -D - -o /dev/null
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on May 8 2013 16:51:19 (1368057079)
Expires: Mon, 13 May 2013 09:47:40 GMT
Date: Mon, 13 May 2013 08:47:40 GMT
Cache-Control: public, max-age=3600, no-transform
Last-Modified: Fri, 22 Mar 2013 17:09:47 GMT
ETag: "755232ae8fef22bc7b4e9510a68a646e"
x-goog-generation: 1363972188238000
x-goog-metageneration: 2
Content-Type: video/mp4
x-goog-hash: crc32c=pZmS2Q==
x-goog-hash: md5=dVIyro/vIrx7TpUQpopkbg==
Accept-Ranges: bytes
Content-Length: 15535795
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Content-Length, Date, Server, Transfer-Encoding
This appears to be a bug on our side. Only the last header value in the ResponseHeaders list is returned in the Access-Control-Expose-Headers header. We are working on rolling out a fix, but as a workaround, if you only need the Content-Range header (Content-Length is considered a simple header by the CORS spec and is added automatically), please try setting your CORS config to this:
<?xml version="1.0" ?>
<CorsConfig>
<Cors>
<Origins>
<Origin>*</Origin>
</Origins>
<Methods>
<Method>GET</Method>
<Method>HEAD</Method>
<Method>DELETE</Method>
</Methods>
<ResponseHeaders>
<ResponseHeader>Content-Range</ResponseHeader>
</ResponseHeaders>
<MaxAgeSec>1800</MaxAgeSec>
</Cors>
</CorsConfig>
UPDATE: This bug has been fixed.