Point subfolder in S3 bucket to *.buildmywebsite domain [duplicate] - amazon-s3

I've got a pretty specific problem here, we've got a system that we already have and maintain, the system involves using subdomains to route people to specific apps.
on a traditional server that goes like follows; we have a wildcard subdomain, *.domain.com that routes to nginx and serves up a folder
so myapp.domain.com > nginx > serves up myapp app folder > myapp folder contains a static site
I'm trying to migrate this in some way to AWS, I basically need to do a similar thing in AWS, I toyed with the idea of putting each static app into an s3 bucket and then the wildcard domain in route 53 but i'm unsure how s3 would know which folder to serve up as that functionality isn't part of route 53
Anyone have any suggestions?
Thanks for all your help

CloudFront + Lambda#Edge + S3 can do this "serverless."
Lambda#Edge is a CloudFront enhancement that allows attributes of requests and responses to be represented and manipulated as simple JavaScript objects. Triggers can be provisioned to fire during request processing, either before the cache is checked ("viewer request" trigger) or before the request proceeds to the back-end ("origin server", an S3 web site hosting endpoint, in this case) following a cache miss ("origin request" trigger)... or during response processing, after the response is received from the origin but before it is considered for storing in the CloudFront cache ("origin response" trigger), or when finalizing the response to the browser ("viewer response" trigger). Response triggers can also examine the original request object.
The following snippet is something I originally posted at the AWS Forums. It is an Origin Request trigger which compares the original hostname to your pattern (e.g. the domain must match *.example.com) and if it does, the hostname prefix subdomain-here.example.com is request is served from a folder named for the subdomain.
lol.example.com/cat.jpg -> my-bucket/lol/cat.jpg
funny-pics.example.com/cat.jpg -> my-bucket/funny-pics/cat.jpg
In this way, static content from as many subdomains as you like can all be served from a single bucket.
In order to access the original incoming Host header, CloudFront needs to be configured to whitelist the Host header for forwarding to the origin even though the net result of the Lambda function's execution will be to modify that value before the origin acually sees it.
The code is actually very simple -- most of the following is explanatory comments.
'use strict';
// if the end of incoming Host header matches this string,
// strip this part and prepend the remaining characters onto the request path,
// along with a new leading slash (otherwise, the request will be handled
// with an unmodified path, at the root of the bucket)
const remove_suffix = '.example.com';
// provide the correct origin hostname here so that we send the correct
// Host header to the S3 website endpoint
const origin_hostname = 'example-bucket.s3-website.us-east-2.amazonaws.com'; // see comments, below
exports.handler = (event, context, callback) => {
const request = event.Records[0].cf.request;
const headers = request.headers;
const host_header = headers.host[0].value;
if(host_header.endsWith(remove_suffix))
{
// prepend '/' + the subdomain onto the existing request path ("uri")
request.uri = '/' + host_header.substring(0,host_header.length - remove_suffix.length) + request.uri;
}
// fix the host header so that S3 understands the request
headers.host[0].value = origin_hostname;
// return control to CloudFront with the modified request
return callback(null,request);
};
Note that index documents and redirects from S3 may also require an Origin Response trigger to normalize the Location header against the original request. This will depend on exactly which S3 website features you use. But the above is a working example that illustrates the general idea.
Note that const origin_hostname needs to be set to the bucket's endpoint hostname as configured in the CloudFront origin settings. In this example, the bucket is in us-east-2 with the web site hosting feature active.

Create a Cloudfront distribution
Add all the Alternate CNAMEs records in the cloudfront distribution
Add a custom origin as the EC2 server.
Set behaviours as per your requirements.
Configure nginx virtualhosts in the server to route to specific folders.

Related

Modifying CloudFront Origin Request for S3

I have two S3 buckets behind a CloudFront distribution.
One bucket is the default behavior origin; one is an additional behavior under a specific path pattern.
The problem is that CloudFront uses the whole path to request the bucket object, which requires the exact path to exist as part of the object name in the bucket.
Path pattern auth/* requires an auth/index.html object if I request .../auth/index.html.
If I change the path, I would have to change all the object names in the bucket.
Is there a way around this?
Thanks to a comment, I found a solution with Lambda#Edge.
If you used the path pattern auth/* in a CloudFront behavior so that all URLs starting with auth/ are routed to a specific S3 bucket, and you want to save all your files in that bucket without an auth/ prefix. Then you need to add this edge Lambda to your CloudFront behavior:
const prefix = "auth/"
exports.handler = async (e) => {
const { request } = e.Records.pop().cf;
if (request.uri.contains(prefix))
request.uri = request.uri.replace(prefix, "");
return request;
});
The event type is origin request.
On a request to auth/index.html CloudFront will fetch index.html from your associated bucket and not auth/index.html
Since origin requests are only sent to the S3 bucket when a cache miss happens, this should be a quite cheap solution.

Using a S3 Static Site with an Application Load Balancer

I have an ALB that currently routes traffic to multiple urls. I'd like to be able to route traffic to a Static S3 site in the event that we need to perform maintenance. We would then display a static "Maintenance" page instead of our login page.
I have created a CloudFront Distribution that allows a S3 site to be loaded with an SSL cert but I am not sure how to connect that distribution to send all of the traffic to the S3 maintenance site.
This is the Terraform ALB listener I'm using. Can I specify my CloudFront distribution arn at the target_group and have it route all traffic to the static site?
Or could I simply link my S3 arn here with an S3 policy allowing the ALB access to get the bucket objects?
resource "aws_alb_listener" "ssl_alb_httpslistener" {
load_balancer_arn = "${aws_alb.alb_lis.arn}"
port = "443"
protocol = "HTTPS"
ssl_policy = "Sec-TLS"
certificate_arn = "${var.ssl_cert_arn}"
default_action {
target_group_arn = "${data.terraform_remote_state.php.target_arn}"
type = "forward"
}
}
I would I expect that I could route traffic that passes through an ALB to a Static S3 site from the target_group. Curious if this is the best way to go about this.
The simple answer is to use a redirect option on the ALB to forward traffic to a new url. My Route53 url is connected to a CloudFront Distribution linked to the S3 bucket. Here I was able to specify a single redirect url and keep my HTTPS traffic options with minimal infrastructure modifications.
You can now have Lambda function as target group and with Lambda, you can trigger S3 , make cloudfront(http) GET request etc.
I faced the same issue of 1MBytes limit with ALB and lambdas and used compression as a workaround.
Set in response headers Content-Encoding= and compress your file(s) accordingly.

Routing requests using cloudflare to different web applications

I currently have two web apps that are set up in cloudflare with the following CNAMEs. Both are keystonejs applications.
app1.example.com ===pointing to ===> AWS ALB 1
app2.example.com ===pointing to ===> AWS ALB 2
I have Cloudflare Enterprise set up, so i'm able to use the "Render Override" feature in my page rules. I have 2 page rules set up using the following:
www.example.com ===render override ===> app1.example.com
www.example.com/app2/* ===render override ===> app2.example.com
Now in order to access the keystonejs application on app2.example.com. The application is called using app2.example.com/pa
The problem that i'm facing is that render override doesnt allow me to use sub paths, and i do not want to use the forwarding rule. Do i need to make my keystone application accessible through the root url, namely app2.example.com/ ? or is there another way to do this? Otherwise, would i need to use a reverse proxy? such as nginx ?
Thanks
Note: Since you are an enterprise customer, I highly recommend contacting your Customer Success Manager and/or Solutions Engineer at Cloudflare. They are there to help with exactly these kinds of questions. That said, I'll answer the question here for the benefit of self-serve customers.
I think when you say "Render Override" you actually mean "Resolve Override". This setting changes the DNS lookup for the request such that it is routed to a different origin IP address than it would be normally.
Note that Resolve Override does not rewrite the request in any way; it only routes it to a different server. So, a request to www.example.com/app2/foo will go to the server app2.example.com, but the path will still be /app2/foo (not /foo), and the Host header will still be Host: www.example.com.
It sounds like in your case you really want /app2/* to be rewritten to /pa/*, in addition to redirecting to a different origin. You can accomplish this using Cloudflare Workers, which lets you execute arbitrary JavaScript on Cloudflare's edge. Here's what the script might look like:
addEventListener("fetch", event => {
event.respondWith(handle(event.request));
});
async function handle(request) {
let url = new URL(request.url) // parse the URL
if (url.pathname.startsWith("/app2/")) {
// Override the target hostname.
url.host = "app2.example.com"
// Replace /app2/ with /pb/ in the path.
url.pathname = "/pb/" + url.pathname.slice("/app2/".length)
// Send the request on to origin.
return fetch(url, request)
} else {
// Just override the hostname.
url.host = "app1.example.com"
// Send the request on to origin.
return fetch(url, request)
}
}
With this deployed, you can remove your Resolve Override page rules, as they are now covered by the Worker script.
Note that the above script actually does rewrite the Host header in addition to the path. If you want the Host header to stay as www.example.com, then you will need to use the cf.resolveOverride option. This is only available to enterprise customers; ask your CSM or SE if you need help using it. But, for most cases, you actually want the Host header to be rewritten, so you probably don't need this.

How to use Akamai infront of S3 buckets?

I have a static website that is currently hosted in apache servers. I have an akamai server which routes requests to my site to those servers. I want to move my static websites to Amazon S3, to get away from having to host those static files in my servers.
I created a S3 bucket in amazon, gave it appropriate policies. I also set up my bucket for static website hosting. It told me that I can access the site at
http://my-site.s3-website-us-east-1.amazonaws.com
I modified my akamai properties to point to this url as my origin server. When I goto my website, I get Http 504 errors.
What am i missing here?
Thanks
K
S3 buckets don't support HTTPS?
Buckets support HTTPS, but not directly in conjunction with the static web site hosting feature.
See Website Endpoints in the S3 Developer Guide for discussion of the feature set differences between the REST endpoints and the web site hosting endpoints.
Note that if you try to directly connect to your web site hosting endpoint with your browser, you will get a timeout error.
The REST endpoint https://your-bucket.s3.amazonaws.com will work for providing HTTPS between bucket and CDN, as long as there are no dots in the name of your bucket
Or if you need the web site hosting features (index documents and redirects), you can place CloudFront between Akamai and S3, encrypting the traffic inside CloudFront as it left the AWS network on its way to Akamai (it would still be in the clear from S3 to CloudFront, but this is internal traffic on the AWS network). CloudFront automatically provides HTTPS support on the dddexample.cloudfront.net hostname it assigns to each distribution.
I admit, it sounds a bit silly, initially, to put CloudFront behind another CDN but it's really pretty sensible -- CloudFront was designed in part to augment the capabilities of S3. CloudFront also provides Lambda#Edge, which allows injection of logic at 4 trigger points in the request processing cycle (before and after the CloudFront cache, during the request and during the response) where you can modify request and response headers, generate dynamic responses, and make external network requests if needed to implement processing logic.
I faced this problem currently and as mentioned by Michael - sqlbot, putting the CloudFront between Akamai and S3 Bucket could be a workaround, but doing that you're using a CDN behind another CDN. I strongly recommend you to configure the redirects and also customize the response when origin error directly in Akamai (using REST API endpoint in your bucket). You'll need to create three rules, but first, go to CDN > Properties and select your property, Edit New Version based on the last one and click on Add Rule in Property Configuration Settings section. The first rule will be responsible for redirect empty paths to index.html, create it just like the image below:
builtin.AK_PATH is an Akamai's variable. The next step is responsible for redirect paths different from the static ones (html, ico, json, js, css, jpg, png, gif, etc) to \index.html:
The last step is responsible for customize an error response when origin throws an HTTP error code (just like the CloudFront Error Pages). When the origin returns 404 or 403 HTTP status code, the Akamai will call the Failover Hostname Edge Server (which is inside the Akamai network) with the /index.html path. This setup will be triggered when refreshing pages in the browser and when the application has redirection links (which opens new tabs for example). In the Property Hostnames section, add a new hostname that will work as the Failover Hostname Edge Server, the name should has less than 16 characters, then, add the -a.akamaihd.net suffix to it (that's the Akamai pattern). For example: failover-a.akamaihd.net:
Finally, create a new empty rule just like the image below (type the hostname that you just created in the Alternate Hostname in This Property section):
Since you are already using Akamai as a CDN, you could simply use their NetStorage product line to achieve this in a simplified manner.
All you would need to do is to move the content from s3 to Akamai and it would take care of the rest(hosting, distribution, scaling, security, redundancy).
The origin settings on Luna control panel could simply point to the Netstorage FTP location. This will also remove the network latency otherwise present when accessing the S3 bucket from the Akamai Network.

Wildcard subdomains point to appropriate S3/CloudFront subdirectories

I need multiple subdomains to point to individual buckets/subdirectories on Amazon S3 (synched to CloudFront distribution), where I'm hosting some static files.
So that ANY
SUBDOMAINNAME.example.com
automatically points to
s3.amazonaws.com/somebucket/SUBDOMAINNAME
or
somedistributionname.cloudfront.net/SUBDOMAINNAME
Is there a way to accomplish this without running a server for redirection?
Can it be done without changing DNS records for each new subdomain or, if not, adding the DNS rules programmatically?
What is the most efficient way of doing it, in terms of resource usage. (There might be hundreds of subdomains with 100s of daily requests for each)
update: this answer was correct when written, and the techniques described below are still perfectly viable but potentially less desirable since Lambda#Edge can now be used to accomplish this objective, as I explained in my answer to Serving a multitude of static sites from a wildcard domain in AWS.
No, there is no way to do this automatically.
Is there a way to accomplish this without running a server for redirection?
Technically, it isn't redirection that you'd need, to accomplish this. You'd need path rewriting, and that's why the answer to your ultimate question is "no" -- because Route 53 (and DNS in general) can't do anything related to paths.
Route 53 does support wildcard DNS, but that's of limited help without CloudFront and/or S3 supporting a mechanism to put the host header from the HTTP request into the path (which they don't).
Now, this could easily be accomplished in a "zero-touch" mode with a single Route 53 * wildcard entry, a single CloudFront distribution configured for *.example.com, and one or more EC2 instances running HAProxy to do the request path rewriting and proxy the request onward to the S3 bucket. A single line in a basic configuration file would accomplish that request rewrite:
http-request set-path /%[req.hdr(host)]%[path]
Then you'd need the proxy to send the the actual bucket endpoint hostname to S3, instead of the hostname supplied by the browser:
http-request set-header Host example-bucket.s3.amazonaws.com
The proxy would send the modified request to S3, return S3's response to CloudFront, which would return the response to the browser.
However, if you don't want to take this approach, since a server would be required, then the alternative solution looks like this:
Configure a CloudFront distribution for each subdomain, setting the alternate domain name for the distribution to match the specific subdomain.
Configure the Origin for each subdomain's distribution to point to the same bucket, setting the origin path to /one-specific-subdomain.example.com. CloudFront will change a request for GET /images/funny-cat.jpg HTTP/1.1 to GET /one-specific-subdomain.example.com/images/funny-cat.jpg HTTP/1.1 before sending the request to S3, resulting in the behavior you described. (This is the same net result as the behavior I described for HAProxy, but it is static, not dynamic, hence one distribution per subdomain; in neither case would this be a "redirect" -- so the address bar would not change).
Configure an A-record Alias in Route 53 for each subdomain, pointing to the subdomain's specific CloudFront distribution.
This can all be done programmatically through the APIs, using any one of the the SDKs, or using aws-cli, which is a very simple way to test, prototype, and script such things without writing much code. CloudFront and Route 53 are both fully automation-friendly.
Note that there is no significant disadvantage to each site using its own CloudFront distribution, because your hit ratio will be no different, and distributions do not have a separate charge -- only request and bandwidth charges.
Note also that CloudFront has a default limit of 200 distributions per AWS account but this is a soft limit that can be increased by sending a request to AWS support.
Since Lambda#edge this can be done with a lambda function triggered by the Cloud Front "Viewer Request" event.
Here is an example of such a Lambda function where a request like foo.example.com/index.html will return the file /foo/index.html from your origin.
You will need a CF distribution with the CNAME *.example.com, and an A record "*.example.com" pointing to it
exports.handler = (event, context, callback) => {
const request = event.Records[0].cf.request;
const subdomain = getSubdomain(request);
if (subdomain) {
request.uri = '/' + subdomain + request.uri;
}
callback(null, request);
};
function getSubdomain(request) {
const hostItem = request.headers.host.find(item => item.key === 'Host');
const reg = /(?:(.*?)\.)[^.]*\.[^.]*$/;
const [_, subdomain] = hostItem.value.match(reg) || [];
return subdomain;
}
As for the costs take a look at lambda pricing. At current pricing is 0.913$ per million requests
A wildcard works on S3. I just put an A record * that points to an IP and it worked.