I have a private bucket in S3 and would like to only allow access only to requests that include a particular (secret) header, sent from a CDN (not CloudFront, as that would of course be simple to allow using its own id).
So that means writing a bucket policy to allow just those secret-header requests.
I've been doing some research (http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html) and I can see that you can test other attributes of the request like aws:Referer to do a comparison on the referer, and aws:SourceIp to do a comparison on the source IP - but how would I go about doing a comparison on a custom header e.g. "X-my-secret-header"?
Do bucket policies support testing header values? If so, how?
AWS recommends using the "Referer" header to store a token and then use an AWS Bucket Policy that restricts requests to aws:Referer headers that match the token. Note that the token is not the actual referer URL, it's just whatever secret you want to use.
You will need to make sure that the requests from your CDN to S3 are over HTTPS so that secret token is encrypted.
This is described in the AWS article How do I use CloudFront to serve a static website hosted on Amazon S3? under the section "Using a website endpoint as the origin, with access restricted by a Referer header" Assuming you can set the "Referer" header from your CDN to the origin, the same concepts should apply.
Related
I have a situation which I am unable to understand easily why and I am not able to find any documentation either.
I have done the following:
Created a S3 bucket
Given public access to it
Enabled it for static website hosting
Created a CloudFront distribution to it
Enabled HTTPS at cloudfront
Now I am trying to restrict the access of S3 bucket only to CloudFront.
I tried the steps presented at
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html
Unfortunately, when I tried to edit the origin I don't see all the options in the UI especially Restrict Bucket Access is missing.
I only see options to edit Origin Domain Name, Origin Path, Origin Id (grayedout), Origin Custom Headers - No option to enter OAI or setting Restrict Bucket Access etc.
Is it because of enabling HTTPS?
S3 masters, please help!
Origin access identities are only applicable when using the S3 REST endpoint (e.g. example-bucket.s3.amazonaws.com) for the bucket -- not when you are using the website hosting endpoint (e.g. example-bucket.s3-website.us-east-2.amazonaws.com), because website hosting endpoints do not support authenticated requests -- they are only for public content... but OAI is an authentication mechanism.
When using the website endpoint, CloudFront does not treat the origin as an S3 Origin -- it is treated as a Custom Origin, and these options are not available, because if they were available, they wouldn't work anyway (for the reason mentioned above).
For those who have since changed some S3 settings to not be public and intend it to be only retrievable via Cloudfront, it is now there but hidden. You just have to cut copy the value from Origin Domain Name in origin tab and then re-paste it in again (if its the name bucket) and the UI will now render with the Restrict Access input options.
I have a static website that is currently hosted in apache servers. I have an akamai server which routes requests to my site to those servers. I want to move my static websites to Amazon S3, to get away from having to host those static files in my servers.
I created a S3 bucket in amazon, gave it appropriate policies. I also set up my bucket for static website hosting. It told me that I can access the site at
http://my-site.s3-website-us-east-1.amazonaws.com
I modified my akamai properties to point to this url as my origin server. When I goto my website, I get Http 504 errors.
What am i missing here?
Thanks
K
S3 buckets don't support HTTPS?
Buckets support HTTPS, but not directly in conjunction with the static web site hosting feature.
See Website Endpoints in the S3 Developer Guide for discussion of the feature set differences between the REST endpoints and the web site hosting endpoints.
Note that if you try to directly connect to your web site hosting endpoint with your browser, you will get a timeout error.
The REST endpoint https://your-bucket.s3.amazonaws.com will work for providing HTTPS between bucket and CDN, as long as there are no dots in the name of your bucket
Or if you need the web site hosting features (index documents and redirects), you can place CloudFront between Akamai and S3, encrypting the traffic inside CloudFront as it left the AWS network on its way to Akamai (it would still be in the clear from S3 to CloudFront, but this is internal traffic on the AWS network). CloudFront automatically provides HTTPS support on the dddexample.cloudfront.net hostname it assigns to each distribution.
I admit, it sounds a bit silly, initially, to put CloudFront behind another CDN but it's really pretty sensible -- CloudFront was designed in part to augment the capabilities of S3. CloudFront also provides Lambda#Edge, which allows injection of logic at 4 trigger points in the request processing cycle (before and after the CloudFront cache, during the request and during the response) where you can modify request and response headers, generate dynamic responses, and make external network requests if needed to implement processing logic.
I faced this problem currently and as mentioned by Michael - sqlbot, putting the CloudFront between Akamai and S3 Bucket could be a workaround, but doing that you're using a CDN behind another CDN. I strongly recommend you to configure the redirects and also customize the response when origin error directly in Akamai (using REST API endpoint in your bucket). You'll need to create three rules, but first, go to CDN > Properties and select your property, Edit New Version based on the last one and click on Add Rule in Property Configuration Settings section. The first rule will be responsible for redirect empty paths to index.html, create it just like the image below:
builtin.AK_PATH is an Akamai's variable. The next step is responsible for redirect paths different from the static ones (html, ico, json, js, css, jpg, png, gif, etc) to \index.html:
The last step is responsible for customize an error response when origin throws an HTTP error code (just like the CloudFront Error Pages). When the origin returns 404 or 403 HTTP status code, the Akamai will call the Failover Hostname Edge Server (which is inside the Akamai network) with the /index.html path. This setup will be triggered when refreshing pages in the browser and when the application has redirection links (which opens new tabs for example). In the Property Hostnames section, add a new hostname that will work as the Failover Hostname Edge Server, the name should has less than 16 characters, then, add the -a.akamaihd.net suffix to it (that's the Akamai pattern). For example: failover-a.akamaihd.net:
Finally, create a new empty rule just like the image below (type the hostname that you just created in the Alternate Hostname in This Property section):
Since you are already using Akamai as a CDN, you could simply use their NetStorage product line to achieve this in a simplified manner.
All you would need to do is to move the content from s3 to Akamai and it would take care of the rest(hosting, distribution, scaling, security, redundancy).
The origin settings on Luna control panel could simply point to the Netstorage FTP location. This will also remove the network latency otherwise present when accessing the S3 bucket from the Akamai Network.
We are considering moving our file delivery to Cloudfront.
Currently, we generate "secure" URLs for our file delivery on a individual basis which look like:
http://downloads.xxxxx.com/1/2005-01-01_2006-01-01.csv?AWSAccessKeyId=012NFZM3D44FSG20CP82&Expires=1495287427&response-cache-control=No-cache&response-content-disposition=attachment%3B%20filename%3D2005-01-01_2006-01-01.csv&Signature=tWAeES3rhAlv2SQoZkqyYJEexH0%3D
Is there an easy way to apply Cloudfront to the URL above, or do we need to configure them from scratch using Signed URLs? Would the S3 authenticated URL "pass-through" using Cloudfront if I create a simple distribution there?
It is theoretically possible to configure CloudFront to pass-through an S3 signed URL, but doing so would defeat all caching, so... no, it's not a viable solution.
CloudFront uses an entirely different algorithm for signed URLs, so there's not a way to simply transform an existing S3 signed URL into a CloudFront signed URL.
Note also that you'll need to embed the existing response-* parameters in the URL before signing it. CloudFront should still pass them through so that S3 can modify its response as indicated.
Unrelated: one feature you might find interesting is that with CloudFront signed URLs, you actually have the optional ability to embed the client's IP address in the URL in such a way that the URL can only be used from a single IP address. This isn't something that can be done in a straightforward way with an S3 signed URL.
Problem:
I am storing number of HLS streams in S3 with given file structure:
Video1
├──hls3
├──hlsv3-master.m3u8
├──media-1
├──media-2
├──media-3
├──media-4
├──media-5
├──hls4
├──hlsv4-master.m3u8
├──media-1
├──media-2
├──media-3
├──media-4
├──media-5
In my user API I know which exactly user has access to which video content
but I also need to ensure that video links are not sharable and only accessible
by users with right permissions.
Solutions:
1) Use signed / temp S3 urls for private S3 content. Whenever client wants to play some specific video it is
sending request to my API. If user has right permissions the API is generating signed url
and returning it back to client which is passing it to player.
The problem I see here is that real video content is stored in dozen of segments files in media-* directories
and I do not really see how can I protect all of them - would I need to sign each of the segment file urls separately?
2) S3 content is private. Video stream requests made by players are going through my API or separate reverse-proxy.
So whenever client decides to play specific video, API / reverse-proxy is getting the request, doing authentication & authorization
and passing the right content (master play list files & segments).
In this case I still need to make S3 content private and accessible only by my API / reverse-proxy. What should be the recommended way here?
S3 rest authentication via tokens?
3) Use encryption with protected key. In this case all of video segments are encrypted and publicly available. The key is also stored in S3
but is not publicly available. Every key request made by player is authenticated & authorized by my API / reverse-proxy.
These are 3 solutions I have in my mind right now. Not convinced on all of them. I am looking for something simple and bullet proof secure. Any recommendations / suggestions?
Used technology:
ffmpeg for video encoding to different bitrates
bento4 for video segmentation
would I need to sign each of the segment file urls separately?
If the player is requesting directly from S3, then yes. So that's probably not going to be the ideal approach.
One option is CloudFront in front of the bucket. CloudFront can be configured with an Origin Access Identity, which allows it to sign requests and send them to S3 so that it can fetch private S3 objects on behalf of an authorized user, and CloudFront supports both signed URLs (using a different algorithm than S3, with two important differences that I will explain below) or with signed cookies. Signed requests and cookies in CloudFront work very similarly to each other, with the important difference being that a cookie can be set once, then automatically used by the browser for each subsequent request, avoiding the need to sign individual URLs. (Aha.)
For both signed URLs and signed cookies in CloudFront, you get two additional features not easily done with S3 if you use a custom policy:
The policy associated with a CloudFront signature can allow a wildcard in the path, so you could authorize access to any file in, say /media/Video1/* until the time the signature expires. S3 signed URLs do not support wildcards in any form -- an S3 URL can only be valid for a single object.
As long as the CloudFront distribution is configured for IPv4 only, you can tie a signature to a specific client IP address, allowing only access with that signature from a single IP address (CloudFront now supports IPv6 as an optional feature, but it isn't currently compatible with this option). This is a bit aggressive and probably not desirable with a mobile user base, which will switch source addresses as they switch from provider network to Wi-Fi and back.
Signed URLs must still all be generated for all of the content links, but you can generate and sign a URL only once and then reuse the signature, just string-rewriting the URL for each file making that option computationally less expensive... but still cumbersome. Signed cookies, on the other hand, should "just work" for any matching object.
Of course, adding CloudFront should also improve performance through caching and Internet path shortening, since the request hops onto the managed AWS network closer to the browser than it typically will for requests direct to S3. When using CloudFront, requests from the browser are sent to whichever of 60+ global "edge locations" is assumed to be nearest the browser making the request. CloudFront can serve the same cached object to different users with different URLs or cookies, as long as the sigs or cookies are valid, of course.
To use CloudFront signed cookies, at least part of your application -- the part that sets the cookie -- needs to be "behind" the same CloudFront distribution that points to the bucket. This is done by declaring your application as an additional Origin for the distribution, and creating a Cache Behavior for a specific path pattern which, when requested, is forwarded by CloudFront to your application, which can then respond with the appropriate Set-Cookie: headers.
I am not affiliated with AWS, so don't mistake the following as a "pitch" -- just anticipating your next question: CloudFront + S3 is priced such that the cost difference compared to using S3 alone is usually negligible -- S3 doesn't charge you for bandwidth when objects are requested through CloudFront, and CloudFront's bandwidth charges are in some cases slightly lower than the charge for using S3 directly. While this seems counterintuitive, it makes sense that AWS would structure pricing in such a way as to encourage distribution of requests across its network rather than to focus them all against a single S3 region.
Note that no mechanism, either the one above or the one below is completely immune to unauthorized "sharing," since the authentication information is necessarily available to the browser, and thus to the user, depending on the user's expertise... but both approaches seem more than sufficient to keep honest users honest, which is all you can ever hope to do. Since signatures on signed URLs and cookies have expiration times, the duration of the share-ability is limited, and you can identify such patterns through CloudFront log analysis, and react accordingly. No matter what approach you take, don't forget the importance of staying on top of your logs.
The reverse proxy is also a good idea, probably easily implemented, and should perform quite acceptably with no additional data transport charges or throughput issues, if the EC2 machines running the proxy are in the same AWS region as the bucket, and the proxy is based on solid, efficient code like that found in Nginx or HAProxy.
You don't need to sign anything in this environment, because you can configure the bucket to allow the reverse proxy to access the private objects because it has a fixed IP address.
In the bucket policy, you do this by granting "anonymous" users the s3:getObject privilege, only if their source IPv4 address matches the IP address of one of the proxies. The proxy requests objects anonymously (no signing needed) from S3 on behalf of authorized users. This requires that you not be using an S3 VPC endpoint, but instead give the proxy an Elastic IP address or put it behind a NAT Gateway or NAT instance and have S3 trust the source IP of the NAT device. If you do use an S3 VPC endpoint, it should be possible to allow S3 to trust the request simply because it traversed the S3 VPC Endpoint, though I have not tested this. (S3 VPC Endpoints are optional; if you didn't explicitly configure one, then you don't have one, and probably don't need one).
Your third option seems weakest, if I understand it correctly. An authorized but malicious user gets the key an can share it all day long.
Having a heck of a time setting up a referer policy from my url to have access to an Amazon S3 bucket.
I can get it to work with my .myurl.com/ but anytime the request from a secure https request, my access is denied even with the wild card.
Thanks
Edit: Rushed my first initial post, here's more detail.
I have a bucket policy on amazon s3 that only allows access if it comes from my url(s).
"aws:Referer": [
"*.myurl.com/*",
"*.app.dev:3000/*" ]
This referer policy correctly only allows connections from my dev environment and also my staging url if accessed via http. However, if the user is located at https://www.myurl.com/* they are denied access from Amazon.
Is there a way to allow https connections to Amazon S3? Is it my Bucket Policy? I've tried hard coding the https url into the bucket policy, but this did not do the trick.
Sorry about being overly brief.
From the HTTP/1.1 RFC:
Clients SHOULD NOT include a
Referer header field in a (non-secure)
HTTP request if the referring page was
transferred with a secure protocol.