I have created four S3 buckets, each with a simple index.html file and each with unique content.
I have created a CloudFront distribution and assigned it four origins, one for each of the four buckets.
Each origin has an Origin Access Identity and that OAI has been used in it's related bucket's policy, eg:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity 123456789ABCDE"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-first-test-bucket/*"
}
]
}
I have also set Block all public access to true for each bucket.
When I visit the CloudFront distribution name I see the content for my region.
However, when I use a geo-browser to test the distribution from another region (one closer to one of the other buckets) I see the same content.
How can I configure my CloudFront distribution to serve the closest region-specific content? (eg: us-east-1 bucket content served through CloudFront for New York users.)
Geo-browser is not perfect for testing, you should test this with a good VPN.
to verify what I am saying, try to enter a blocked website in China. geo-browser will take you to it but it is trying to trick the server to think the connection is from China by changing IP address.
This can not Trick AWS. So test with VPN (a paid one is preferable)
More Info:
How does AWS Cloudfront CDN works:
when the first user from a specific region request a file
the file will be streamed (copied) from S3 to the closest Cloudfront server in the user region
the file will stay on this server temporary (usually 24 hours)
when a second user from the same Region request the same file he/she will get the copy from Cloudfront close server too.
if the same file changes on S3 it will be changes in very short time in the Cloudfront too (from 1 second to 5 minutes)
So, only the first request for the file will be affected by the distance of S3 bucket, which is negligible.
My recommendation is to use 1 S3 bucket only with folders specifying content depending on local (us, fr, gb, ...etc) and rely on the Cloudfront CDN to distribute content to different CDN servers for each region. I am using Cloudfront in this way and everything I wrote here is from real experiments I've done before.
Conclusion: if you use CDN then the location of storage server is not a factor for speedy delivery of content.
You can use a Route53 traffic policy. Add a Geolocation rule and then a Cloudfront distribution as an endpoint.
Related
S3 cross region replication is not working for us when we're upload a file with PHP.
When we upload the file from the AWS interface it replicate to the other bucket it's working great, but when we use S3 API for PHP: putObject it's upload but don't replicate to the other bucket.
What are we missing here?
Thanks
As I commented, it would be great to see the bucket policy of the upload bucket, the bucket policy of the destination bucket, and the permissions granted to whatever IAM role / user the PHP is using.
My guess is that there's some difference in config/permissioning between the source bucket's owning account (which is likely what you use when manipulating from the AWS Console interface) and whatever account or role or user is representing your PHP code. For example:
If the owner of the source bucket doesn't own the object in the bucket, the object owner must grant the bucket owner READ and READ_ACP permissions with the object access control list (ACL)
Pending more info from the OP, I'll add some potentially helpful trouble-shooting resources:
Can't get amazon S3 cross-region replication between two accounts to work
AWS Troubleshooting Cross-Region Replication
I don't know if is the same for replicating buckets between accounts, but I use this policy to replicate objects uploaded on a bucket in us-east-1 to a bucket in eu-west-1 and it works like a charm, both uploading files manually of from a python script.
{
"Version": "2008-10-17",
"Id": "S3-Console-Replication-Policy",
"Statement": [
{
"Sid": "S3ReplicationPolicyStmt1",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<AWS Account ID>:root"
},
"Action": [
"s3:GetBucketVersioning",
"s3:PutBucketVersioning",
"s3:ReplicateObject",
"s3:ReplicateDelete"
],
"Resource": [
"arn:aws:s3:::<replicated region ID>.<replicated bucket name>",
"arn:aws:s3:::<replicated region ID>.<replicated bucket name>/*"
]
}
]
}
Where:
- is, of course, your AWS account ID
- is the AWS region ID (eu-west-1, us-east-1, ...) where the replicated bucket will be (in my case is eu-west-1)
- is the name of bucket you want to replicate.
So say you want to replicate a bucket called "my.bucket.com" in eu-west-1, the Resource ARN to put in the policy will be arn:aws:s3:::eu-west-1.my.bucket.com. Same with the leading /*
Also the replication rule is set as follows:
- Source: entire bucket
- Destination: the bucket I mentioned above
- Destination options: leave all unchecked
- IAM role: Create new role
- Rule name: give it a significant name
- Status: Enabled
I have assigned a role with the following policy to my ec2 instance running on us-west-2 region -
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "*"
}
]
}
and trying to access a bucket from ap-southeast-1 region. The problem is every aws s3 operations are timing out. I have also tried specifying region in the command --region ap-southeast-1.
From the documentation, I found this pointer -
Endpoints are supported within the same region only. You cannot create
an endpoint between a VPC and a service in a different region.
So, what is the process to access bucket from a different region using aws-cli or boto client from the instance?
Apparently, to access bucket from a different region, the instance also needs access to the public internet. Therefore, the instance needs to have a public ip or it has to be behind a NAT.
I think it is not necessary to specify the region of the bucket in order to access to it, you can check some boto3 examples from here:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.get_object
I would check to make sure you've given the above permission to the correct User or Role.
Run the command;
aws sts get-caller-identity
You may think the EC2 instance is using credentials you've set when it may be using an IAM role.
I'm trying to solve an issue with my AWS Cloudformation template. The template I have includes a VPC with a private subnet, and a VPC endpoint to allow connections to S3 buckets.
The bucket itself includes 3 buckets, and I have a couple of preexisting buckets already said up in the same region (in this case, eu-west-1).
I use aws-cli to log into an EC2 instance in the private subnet, then use aws-cli commands to access S3 (e.g. sudo aws s3 ls bucketname)
My problem is that I can only list the content of pre-existing buckets in that region, or new buckets that I create manually through the website. When I try to list cloudformation-generated buckets it just hangs and times out:
[ec2-user#ip-10-44-1-129 ~]$ sudo aws s3 ls testbucket
HTTPSConnectionPool(host='vltestbucketxxx.s3.amazonaws.com', port=443): Max retries exceeded with url: /?delimiter=%2F&prefix=&encoding-type=url (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7f2cc0bcf110>, 'Connection to vltestbucketxxx.s3.amazonaws.com timed out. (connect timeout=60)'))
It does not seem to be related to the VPC endpoint (setting the config to allow everything has no effect)
{
"Statement": [
{
"Action": "*",
"Effect": "Allow",
"Resource": "*",
"Principal": "*"
}
]
}
nor does accesscontrol seem to affect it.
{
"Resources": {
"testbucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"AccessControl": "PublicReadWrite",
"BucketName": "testbucket"
}
}
}
}
Bucket policies don't seem to be the issue either (I've generated buckets with no policy attached, and again only the cloudformation generated ones time out). On the website, configuration for a bucket that connects and one that times out looks identical to me.
Trying to access buckets in other regions also times out, but as I understood it cloudformation generates buckets in the same region as the VPC, so that shouldn't be it (the website also shows the buckets to be in the same region).
Does anyone have an idea of what the issue might be?
Edit: I can connect from the VPC public subnet, so maybe it is an endpoint problem after all?
When using a VPC endpoint, make sure that you've configured your client to send requests to the same endpoint that your VPC Endpoint is configured for via the ServiceName property (e.g., com.amazonaws.eu-west-1.s3).
To do this using the AWS CLI, set the AWS_DEFAULT_REGION environment variable or the --region command line option, e.g., aws s3 ls testbucket --region eu-west-1. If you don't set the region explicitly, the S3 client will default to using the global endpoint (s3.amazonaws.com) for its requests, which does not match your VPC Endpoint.
I am trying to block hotlinking of my Cloudfront files from specific domains. Through a combination of online examples and Amazon's own policy generator, I have come up with this:
{
"Version": "2008-10-17",
"Id": "http referer policy",
"Statement": [{
"Sid": "Block image requests",
"Action": "s3:GetObject",
"Effect": "Deny",
"Resource": "arn:aws:s3:::mybucket/subdir/*",
"Condition": {
"StringLike": {
"aws:Referer": [
"http://example.com/*"
]
}
},
"Principal": {
"AWS": "*"
}
}]
}
I sent an invalidation request for a file in the subdirectory of mybucket, then a few minutes later tried reloading the image with the referer header still sent (verified using Chrome's dev tools). Did a hard reload with Ctrl+F5, and the response headers contained "X-Cache:Miss from cloudfront" so it's definitely getting the latest version of the image.
But the image is still displaying fine and is not blocked. The policy generator did not have an option for the "aws:Referer" key, but it's in the Amazon docs here. Have I done something wrong here?
Update 2
Revisiting your policy I wonder how you have actually allowed CloudFront access to your objects in the first place? Have you by chance followed the common advise in e.g. Start Using CloudFront with Amazon S3 that You must ensure that your object permissions are set to Make Everything Public for each object in your Amazon S3 bucket.
In this case you might have stumbled over a related pitfall due to the interaction between the meanwhile three different S3 access control mechanisms available, which can be rather confusing indeed. This is addressed e.g. in Using ACLs and Bucket Policies Together:
When you have ACLs and bucket policies assigned to buckets, Amazon S3
evaluates the existing Amazon S3 ACLs as well as the bucket policy
when determining an account’s access permissions to an Amazon S3
resource. If an account has access to resources that an ACL or policy
specifies, they are able to access the requested resource.
Consequently you would need to migrate your ACL to the bucket policy (i.e. allow CloudFront access before denying via aws:referer) and delete the overly generous ACL thereafter.
Good luck!
Update 1
Okay, now with client caching out the way, I'm afraid this is going to be non trivial (as apparent when you search for aws:referer in the AWS forums), thus might require a couple of iterations (especially given you have researched the topic yourself already):
The most common issue encountered is the leading whitespace error in the AWS documentation (which is particularly annoying, because a simple documentation fix would remedy lots of wasted time on behalf of users and AWS support staff alike)
Your policy doesn't exhibit this issue, however, given you sanitized the real domain you might have replaced the error in your production code in fact?
Also it is important to realize that the HTTP referer header is not necessarily going to be available, see e.g. Referer Hiding (thus your policy won't prevent malicious access anyway, though that's apparently not the issue)
You stated already, that you have verified it to be sent via the Chrome developer tools, so this doesn't apply either (I'm mentioning it to stress the reduced security level).
The policy looks fine at first sight - before digging further into this direction though, I'd recommend to ensure that you are actually bypassing Chrome's cache successfully, which is notoriously less straight forward than people are used to from other browsers; in particular, Ctrl + F5 simply reloads the page, but does not Bypass the cache (not reliable at least)!
As documented there as well, you could use one of the other key combinations To reload a page and bypass the cache (including the confusing 2nd Ctrl + F5 after the 1st one reloaded) , however, I recommend facilitating one of the following two alternatives instead:
Chrome's developer tools offers dedicated support for browsing without a cache - in the bottom right corner of the toolbox panel is a cog icon for settings, clicking this triggers an overlay with an options panel, amongst these you'll find the option Disable cache under section Network.
Chrome's Incognito mode (Ctrl + Shift + N) keeps Google Chrome from storing information about the websites you've visited, which as of today (might change anytime of course) seems to include cached content, cookies, DNS and the like as expected, thus is an even quicker, though less explicit option right now.
I have uploaded some files using the Amazon AWS management console.
I got an HTTP 403 Access denied error. I found out that I needed to set the permission to view.
How would I do that for all the files on the bucket?
I know that it is possible to set permission on each file, but it's time-consuming when having many files that need to be viewable for everyone.
I suggest that you apply a bucket policy1 to the bucket where you want to store public content. This way you don't have to set ACL for every object. Here is an example of a policy that will make all the files in the bucket mybucket public.
{
"Version": "2008-10-17",
"Id": "http better policy",
"Statement": [
{
"Sid": "readonly policy",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::mybucket/sub/dirs/are/supported/*"
}
]
}
That * in "Resource": "arn:aws:s3:::mybucket/sub/dirs/are/supported/*" allows recursion.
1 Note that a Bucket Policy is different than an IAM Policy. (For one you will get an error if you try to include Principal in an IAM Policy.) The Bucket Policy can be edit by going the root of the bucket in your AWS web console and expanding Properties > Permissions. Subdirectories of a bucket also have Properties > Permissions, but there is no option to Edit bucket policy
You can select which directory you want it to be public.
Press on "more" and mark it as public; it will make the directory and all the files to be accessible as public.
You can only modify ACLs for a unique item (bucket or item), soy you will have to change them one by one.
Some S3 management applications allows you to apply the same ACL to all items in a bucket, but internally, it applies the ACL to each one by one.
If you upload your files programmatically, it's important to specify the ACL as you upload the file, so you don't have to modify it later. The problem of using an S3 management application (like Cloudberry, Transmit, ...) is that most of them uses the default ACL (private read only) when you upload each file.
I used Cloudberry Explorer to do the job :)
Using S3 Browser you can update permissions using the gui, also recursively. It's a useful tool and free for non-commercial use.
To make a bulk of files public, do the following:
Go to S3 web interface
Open the required bucket
Select the required files and folders by clicking the checkboxes at the left of the list
Click «More» button at the top of the list, click «Make public»
Confirm by clicking «Make public». The files won't have a public write access despite the warning says «...read this object, read and write permissions».
You could set ACL on each file using aws cli:
BUCKET_NAME=example
BUCKET_DIR=media
NEW_ACL=public-read
aws s3 ls $BUCKET_NAME/$BUCKET_DIR/ | \
awk '{$1=$2=$3=""; print $0}' | \
xargs -t -I _ \
aws s3api put-object-acl --acl $NEW_ACL --bucket $BUCKET_NAME --key "$BUCKET_DIR/_"
I had same problem while uploading the files through program (java) to s3 bucket ..
Error: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'http://localhost:9000' is therefore not allowed access. The response had HTTP status code 403
I added the origin identity and changed the bucket policy and CORS configuration then everything worked fine.
Transmit 5
I wanted to add this here for potential macOS users that already have the beautifully-crafted FTP app called Transmit by Panic.
I already had Panic and it supports S3 buckets (not sure what version this came in but I think the upgrades were free). It also supports recursively updating Read and Write permissions.
You simply right click the directory you want to update and select the Read and Write permissions you want to set them to.
It doesn't seem terribly fast but you can open up the log file by going Window > Transcript so you at least know that it's doing something.
Use AWS policy generator to generate a policy which fits your need. The principal in the policy generator should be the IAM user/role which you'd be using for accessing the object(s).
Resource ARN should be arn:aws:s3:::mybucket/sub/dirs/are/supported/*
Next, click on "Add statement" and follow through. You'll finally get a JSON representing the policy. Paste this in your s3 bucket policy management section which is at "your s3 bucket page in AWS -> permissions -> bucket policy".
This worked for me on digital ocean, which allegedly has the same API as s3:
s3cmd modify s3://[BUCKETNAME]/[DIRECTORY] --recursive --acl-public
The above sets all files to public.