I'm using AWS CDK to setup S3 and CloudFront static website hosting. All works well until I want to redirect "http[s]//:www.mydomain.com" to "https ://mydomain.com". I do not want to make the S3 repositories public rather provide bucket permission for the CloudFront "Origin Access Identity". The relevant snippet of my CDK code is as follows:
const wwwbucket = new s3.Bucket(this, "www." + domainName, {
websiteRedirect: {
hostName: domainName,
protocol: s3.RedirectProtocol.HTTPS },
blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL
})
const oaiWWW = new cloudfront.OriginAccessIdentity(this, 'CloudFront-OriginAccessIdentity-WWW', {
comment: 'Allows CloudFront to access the bucket'
})
wwwbucket.grantRead(oaiWWW)
const cloudFrontRedirect = new cloudfront.CloudFrontWebDistribution(this, 'https://www.' + domainname + '.com redirect', {
aliasConfiguration: {
acmCertRef: certificateArn,
names: [ "www." + domainName ],
sslMethod: cloudfront.SSLMethod.SNI,
securityPolicy: cloudfront.SecurityPolicyProtocol.TLS_V1_1_2016,
},
defaultRootObject: "",
originConfigs: [
// {
// customOriginSource: {
// domainName: wwwbucket.bucketWebsiteDomainName
// },
// behaviors : [ {isDefaultBehavior: true}],
// },
{
s3OriginSource: {
s3BucketSource: wwwbucket,
originAccessIdentity: oaiWWW
},
behaviors : [ {isDefaultBehavior: true}],
}
]
});
Unfortunately the result is that rather than redirecting, browsing to www.mydomain.com results in the browser showing an S3 XML bucket listing result. I can fix the problem manually by using the AWS console to edit CloudFront's "Origin Domain Name" within "origin settings" from:
bucketname.s3.eu-west-2.amazonaws.com
to:
bucketname.s3-website.eu-west-2.amazonaws.com
Then all works as expected. I have tried changing my CDK script to use a customOriginSource rather than s3OriginSource (commented-out code above) which results in the correct address in CloudFront's "Origin Domain Name" but then the CloudFront distribution does not have a "Origin Access Identity" and so can't access the S3 bucket.
Does anyone know a way to achieve the redirect without having to make the redirect bucket public or edit the "Origin Domain Name" manually via the AWS console?
I thought I'd found an answer using a CDK escape hatch. After creating the CloudFront distribution for my redirect I modified the CloudFormation JSON behind the CDK class as (in typescript):
type ChangeDomainName = {
origins: {
domainName: string
}[]
}
const cfnCloudFrontRedirect = cloudFrontRedirect.node.defaultChild as cloudfront.CfnDistribution
var distributionConfig = cfnCloudFrontRedirect.distributionConfig as cloudfront.CfnDistribution.DistributionConfigProperty & ChangeDomainName
distributionConfig.origins[0].domainName = wwwbucket.bucketWebsiteDomainName
cfnCloudFrontRedirect.distributionConfig = distributionConfig
Unfortunately although this appeared to generate the CloudFormation template I was aiming for (checked using cdk synthesize) when deploying (cdk deploy) the following error was generated by CloudFormation:
UPDATE_FAILED | AWS::CloudFront::Distribution |
The parameter Origin DomainName does not refer to a valid S3 bucket.
It appears that even though it's possible to set a website endpoint of the form - ${bucketname}.s3-website.${region}.amazonaws.com - manually in the field Origin Domain Name within the CloudFront console this isn't possible using CloudFormation. This leads me to conclude either:
This is a bug with CloudFormation.
It's a bug in the console, in that the console shouldn't allow this setup.
However although currently modifying Origin Domain Name in console works, I don't know if this is a "legal" configuration that could be changed in the future in which case my code might stop working. The current solutions are:
Make the redirect bucket public in which case the customOriginSource will work.
Rather than using a redirect bucket instead use a Lambda to perform the redirect and deploy within CloudFront using "Lambda#Edge".
I would prefer not to make my redirect bucket public as it results in warnings when using security checking tools. The option of deploying "Lambda#Edge" using the cdk outside of us-east-1 currently looks painful so for the moment I'll continue manually editing Origin Domain Name in the console.
For reference the AWS documentation appears to imply that the API prohibits this use, though the console permits it, see: Using Amazon S3 Buckets Configured as Website Endpoints for Your Origin. See also:
Key differences between a website endpoint and a REST API endpoint
Related
I am new to opa server.
I am working on using opa as authorizer for multiple policies and use case is that I upload multiple policies to s3 bucket and then each policy has his own link like /opa/example/allow or /opa/example1/approve.
Now I want to use different requests coming to use this link along with data to check whether they are allowed for specific policy or not.
I am little confused with config as I was going through opa docs. Can someone guide me for same. Following is config I am using and whenever I am hitting opa server it is giving blank response. I have taken this from some blog but not sure if it will work or not.
config.yaml
bundle-s3:
url: https://opa-testing1.s3.us-west-2.amazonaws.com/
credentials:
s3_signing:
environment_credentials: {}
inside this bucket I have a bundle with rego file as follow :
package opa.examples
import input.user
import input.method
default allow = false
allow { user = "alice" }
allow {
user = "bob"
method = "GET"
}
opa server I am running using : opa run --server -c config.yaml
Request I am hitting is on : localhost:8181/v1/data/opa/example/allow
Can someone help me how I can achieve my use case here and some blogs or examples will be very useful
I have a React app built using Serverless NextJS and served behind AWS CloudFront. I am also using AWS Cognito to do authentication of our users.
After a user successfully authenticates through AWS Cognito, they are redirected to my React App with a query string containing OAuth tokens (id_token, access_token, refresh_token, raw[id_token], raw[access_token], raw[refresh_token], raw[expires_in], raw[token_type]).
It seems that the query string is simply larger than AWS CloudFront's limits and it is throwing the following error below:
413 ERROR
The request could not be satisfied.
Bad request. We can't connect to the server for this app...
Generated by cloudfront (CloudFront)
Request ID: FlfDp8raw80pAFCvu3g7VEb_IRYbhHoHBkOEQxYyOTWMsNlRjTA7FQ==
This error has been encountered before by many other users (see example). Keen to know:
Are there any workarounds? Perhaps is there a way to configure AWS Cognito to reduce the number of tokens that it is passing in the query string by default?
Is it possible to configure AWS CloudFront to ignore enforcing its default limits on certain pages (and not cache theme)?
What's the suggestion going forward? The only thing I can imagine is not to use AWS CloudFront.
After analysing the query fields that AWS Cognito sends to a callback URL, I was able to determine that not all fields are required for my usecase. Particularly the raw OAuth token fields.
With that information, I solved the problem by writing a "middleware" to intercept my backend system redirecting to my frontend (that is sitting behind CloudFront) and trimming away query string fields that I do not need to complete authentication.
In case this could inspire someone else stuck with a similar problem, here is what my middleware looks like for my backend system (Strapi):
module.exports = (strapi) => {
return {
initialize() {
strapi.app.use(async (ctx, next) => {
await next();
if (ctx.request.url.startsWith("/connect/cognito/callback?code=")) {
// Parse URL (with OAuth query string) Strapi is redirecting to
const location = ctx.response.header.location;
const { protocol, host, pathname, query } = url.parse(location);
// Parse OAuth query string and remove redundant (and bloated) `raw` fields
const queryObject = qs.parse(query);
const trimmedQueryObject = _.omit(queryObject, "raw");
// Reconstruct original redirect Url with shortened query string params
const newLocation = `${protocol}//${host}${pathname}?${qs.stringify(
trimmedQueryObject
)}`;
ctx.redirect(newLocation);
}
});
},
};
};
My code uses the AWS Javascript SDK to upload to S3 directly from a browser. Before the upload happens, my server sends it a value to use for 'Authorization'.
But I see no way in the AWS.S3.upload() method where I can add this header.
I know that underneath the .upload() method, AWS.S3.ManagedUpload is used but that likewise doesn't seem to return a Request object anywhere for me to add the header.
It works successfully in my dev environment when I hardcode my credentials in the S3() object, but I can't do that in production.
How can I get the Authorization header into the upload() call?
Client Side
this posts explains how to post from a html form with a pre-generated signature
How do you upload files directly to S3 over SSL?
Server Side
When you initialise the S3, you can pass the access key and secret.
const s3 = new AWS.S3({
apiVersion: '2006-03-01',
accessKeyId: '[value]',
secretAccessKey: '[value]'
});
const params = {};
s3.upload(params, function (err, data) {
console.log(err, data);
});
Reference: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
Alternatively if you are running this code inside AWS services such as EC2, Lambda, ECS etc, you can assign a IAM role to the service that you are using. The permissions can be assigned to the IAM Role
I suggest that you use presigned urls.
I am using minio client to access S3. The S3 storage I am using has two endpoints - one (say EP1) which is accessible from a private network and other (say EP2) from the internet. My application creates a presigned URL for downloading an S3 object using EP1 since it cannot access EP2. This URL is used by another application which is not on this private network and hence has access only to EP2. This URL is (obviously) not working when used by the application outside the network since this URL has EP1 in it.
I have gone through minio documentation but did not find anything which help me specify alternate endpoints.
So my question is -
Is there anything which I have missed from minio that can help me?
Is there any S3 feature which allows generating presigned URL for an
object with EP2 in it?
Or is this not solvable without changing
current network layout?
You can use minio-js to manage this
Here is an example that you can use
var Minio = require('minio')
var s3Client = new Minio.Client({
endPoint: "EP2",
port: 9000,
useSSL: false,
accessKey: "minio",
secretKey: "minio123",
region: "us-east-1"
})
var presignedUrl = s3Client.presignedPutObject('my-bucketname', 'my-objectname', 1000, function(e, presignedUrl) {
if (e) return console.log(e)
console.log(presignedUrl)
})
This will not contact the server at all. The only thing here is that you need to know the region that bucket belongs to. If you have not set any location in minio, then you can use us-east-1 by default.
I have a rails app that uses aws cli to sync bunch of content and config with my s3 bucket like so:
aws s3 sync --acl 'public-read' #{some_path} s3://#{bucket_path}
Now I am looking for some easy way to mark everything that was just updated in sync to be marked as invalidated or expired for CloudFront.
I am wondering if there is some way to use -cache-control flag that aws cli provides to make this happen. So that instead of invalidating CouldFont, just mark the files as expired, so CloudFront will be forced to fetch fresh data from bucket.
I am aware of CloudFront POST API to mark files for invalidation, but that will mean I will have detect what changed in the last sync, then make the API call. I might have any where from 1000s to 1 file syncing. Not a pleasent prospect. But if I have to go this route, how would I go about detecting changes without parsing the s3 sync's console output of-course.
Or any other ideas?
Thanks!
You cannot use the --cache-control option that aws cli provides to invalidate files in CloudFront. The --cache-control option maps directly to the Cache-Control header and CloudFront caches the headers along with the file, so if you change a header you must also invalidate to tell CloudFront to pull in the changed headers.
If you want to use the aws cli, then you must parse the output of the sync command and then use the aws cloudfront cli.
Or, you can use s3cmd from s3tools.org. This program provides the the --cf-invalidate option to invalidate the uploaded filed in CloudFront and a sync command synchronize a directory tree to S3.
s3cmd sync --cf-invalidate <local path> s3://<bucket name>
Read, the s3cmd usage page for more details.
What about using the brand new AWS Lambda? Basically, it executes custom code whenever an event is triggered in AWS (in your case, a file is synchronized in S3).
Whenever you synchronize a file you get an event similar to:
{
"Records": [
{
"eventVersion": "2.0",
// ...
"s3":
{
"s3SchemaVersion": "1.0",
// ...
"object":
{
"key": "hello.txt",
"size": 4,
"eTag": "1234"
}
}
}
]
}
Thus, you can check the name of the file that has changed and invalidate it in CloudFront. You receive one event for every file that has changed.
I have created a script that invalidates a path in CloudFront whenever an update occurs in S3, which might be a good starting point if you decide to use this approach. It is written in JavaScript (Node.js) as it is the language used by Lambda.
var aws = require('aws-sdk'),
s3 = new aws.S3({apiVersion: '2006-03-01'}),
cloudfront = new aws.CloudFront();
exports.handler = function(event, context) {
var filePath = '/' + event.Records[0].s3.object.key,
invalidateParams = {
DistributionId: '1234',
InvalidationBatch: {
CallerReference: '1',
Paths: {
Quantity: 1,
Items: [filePath]
}
}
};
console.log('Invalidating file ' + filePath);
cloudfront.createInvalidation(invalidateParams, function(err, data) {
if (err) {
console.log(err, err.stack); // an error occurred
} else {
console.log(data); // successful response
}
});
context.done(null,'');
};
For more info you can check Lambda's and CloudFront's API documentation.
Note however that the service is still in preview and is subject to change.
The AWS CLI tool can output JSON. Collect the JSON results, then submit an invalidation request per the link you included in your post. To make it really simple you could use a gem like CloudFront Invalidator, which will take a list of paths to invalidate.