Our AWS statement came in and we noticed we're being doubly charged for the number of requests.
First charge is for Asia Pacific (Tokyo) (ap-northeast-1) and this is straightforward because it's where our bucket is located. But there's another charge against US East (N. Virginia) (us-east-1) with a similar number of requests.
Long story short, it appears this is happening because we're using the aws s3 command and we haven't specified a region either via the --region option or any of the fallback methods.
Typing aws configure list shows region: Value=<not set> Type=None Location=None.
And yet our aws s3 commands succeed, albeit with this seemingly hidden charge. The presumption is, our requests first go to us-east-1, but since there isn't a bucket there by the name we specified, it turns around and comes back to ap-northeast-1, where it ultimately succeeds while getting accounted twice.
The ec2 instance where the aws command is run is itself in ap-northeast-1 if that counts for anything.
So the question is, is the presumption above a reasonable account of what's happening? (i.e. Is it expected behaviour.) And, it seems a bit insidious to me but is there a proper rationale for this?
What you are seeing is correct. The aws s3 command needs to know the region in order to access the S3 bucket.
Since this has not been provided, it will make a request to us-east-1, which is effectively the default - see the AWS S3 region chart to see that us-east-1 does not require a location constraint.
If the S3 receives a request for a bucket which is not in that region then it returns a PermanentRedirect response with the correct region for the Bucket. The AWS CLI handles this transparently and repeats the request with the correct endpoint which includes the region.
The easiest way to see this in action is to run commands in debug mode:
aws s3 ls ap-northeast-1-bucket --debug
The output will include:
DEBUG - Response body:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access
must be addressed using the specified endpoint. Please send all future requests to
this endpoint.</Message>
<Endpoint>ap-northeast-1-bucket.s3.ap-northeast-1.amazonaws.com</Endpoint>
<Bucket>ap-northeast-1</Bucket>
<RequestId>3C4FED2EFFF915E9</RequestId><HostId>...</HostId></Error>
The AWS CLI does not assume the Region is the same as the calling EC2 instance, it's a long running confusion/feature request.
Additional Note: Not all AWS services will auto-discover the region in this way and will fail if the Region is not set. S3 works because it uses a Global Namespace which inherently requires some form of discovery service.
Related
I am trying to code an AWS ConfigRule in CloudFormation. A rule to make sure S3 buckets are encrypted, ALB access logs are logging, also that EBS volumes are encrypted.
Do I need to code a new scope and source for each one of these mentioned above, or can I code them all together? meaning, is this below ok
Update:
The short version of the issue that I wanted to cross-reference some values in different stacks in different regions but the documentations were so confusing to let me think it's not possible to do that but it's possible, just I had to output those values as exported values in that stack and then use:
${cf.us-east-1:another-stack.theNeededArn} in the other stack.
The Long version if you are interested:
I am maintaining the infrastructure code using a Serverless framework. I have CloudFront that connecting to an S3 bucket that hosted in Europe. I got a client request to limit access to this bucket through the CloudFront to be limited only to the authenticated users (custom auth). Lambda#Edge was the best solution and I already implemented it that way but because Lambda#Edge has to be deployed to us-east-1, I ended up moving the S3 and the Cloudfront to the same region us-east-1 (That's because Cloufront depends on the lambda edge and S3 depends on the CloudFront so I have to keep them in the same stack or at least the same region). But I don't want to move my bucket to the US for legal stuff and I want to keep the data in Europe, Also my S3 has a lambda trigger function that listens to it and writes some data to a DynamoDb hosted in Europe.
So The problem:
I had S3 in Europe and I want to keep it in Europe but due to using lambda edge and because that the cloud formation or the serverless doesn't support a cross-region stack reference I ended up moving this S3 to the US but that's not the requirements!
I think even though we use the cloud front but having all of our customers in Europe and putting the S3 bucket in the US will increase the latency.
For legal stuff, I want to keep the European user's data inside Europe and not move it outside.
Here in this question's answer, I specified my approach and the full code example in case you are interested: How to access AWS CloudFront that connected with S3 Bucket via Bearer token of a specific user (JWT Custom Auth)
Any Suggestions?
Update (Steps to show the exact problem):
In my serverless.yml, I created this stack contains this lambda
edge:
I didn't want to configure lambda edge using a serverless framework but instead, I used cloud formation to configure the cloud front and everything.
In the cloud formation resources file, I added the cloud front
origins and configure the private origin through its behavior to use
the lambda edge (Please check the highlights in the picture):
Please note that now I am using the lambda edge ARN inside my Cloudfront so they need to be in the same region and because lambda#edge should be in us-east-1 so I decided to move the CloudFront to the same region and that does not really matter cause it has edge behavior by design.
Als if you are interested here I defined all the needed roles for
the predefined lambda edge in the step.1 (And that includes
publishing the correct policies and also the lambda edge version as
in lambda edge, you have to reference the version, not the function
and all set in this step I just put it here for completeness.):
Now I have the cloud front configuration, lambda edge configuration,
and in the cloud front, we are referencing the lambda edge ARN which
leads me to put them in the same region but now I will define my S3
bucket and will make it private so no one can access it, just the
cloud front CloudFrontOriginAccessIdentity can do that:
As you may see in the role I just gave access to CloudFront to get and gave access to the (lambda edge to get and put but not sure if that's correct) but anyways even if we need the CloudFrontOriginAccessIdentity only to be connected to my bucket so now the bucket linked with my CloudFront which is also linked to the lambda edge so I can't separate them to put the S3 in Europe only??
And by that even if I have an S3 trigger lambda function so I should put this function in the US even if this function is doing some stuff related to DynamoDb in Europe? So what is the point? Also even if the Cloudfront is edged but the bucket is regional so if I really need to process some data related to it that means by putting it in us I increased the latency? So that's my full detailed problem.
Update2:
I wanted to post the code as screenshots so I can highlight some lines and make it easier but for whom interested to check the code itself, I already posted the full version of it in my answer to this question here: How to access AWS CloudFront that connected with S3 Bucket via Bearer token of a specific user (JWT Custom Auth)
The solution was instead of moving everything over to us-east-1, just maintain two stacks which are the primary stack and the lambda edge stack. The primary stack resides in EU and the lambda edge stack resides in us-east-1. You can reference the lambda edge functions in us-east-1 by using ${cf.us-east-1:another-stack.lambdaEdgeArn}.
It is impossible to reference Lambda#Edge deployed to us-east-1 from other regions by Fn:ImportValue. In the CloudFormation template, there is a workaround by looking up the version-specific ARN dynamically and passing it as a template parameter for the CloudFront template, in a task file.
- name: Get Lambda Version-ARN
shell:
cmd: "
aws lambda list-versions-by-function \
--function-name '{{ lambda_func_name }}' \
--region '{{ lambda_region }}' \
--query \"max_by(Versions, &to_number(to_number(Version) || '0'))\" \
| jq -r '.FunctionArn'
"
register: lambda_output
- set_fact:
lambda_arn: "{{ lambda_output.stdout }}"
- name: CloudFront
cloudformation:
stack_name: "{{ stack_name }}"
state: "{{ state }}"
region: "{{ region }}"
template: "roles/{{ role_name }}/templates/cloudfront-template.yml"
template_parameters:
LambdaARN: "{{ lambda_arn }}"
...
I would like to start using serverless-framework to manage lambda deploys at my company, but we handle PHI so security’s tight. Our compliance director and CTO had concerns about passing our AWS key and secret to another company.
When doing a serverless deploy, do AWS credentials ever actually pass through to Serverless, Inc?
If not, can someone point me to where in the code I can prove that?
Thanks!
Running serverless deploy isn't just one call, it's many.
AWS example (oversimplification):
Check if deployment s3 bucket already exists
Create an S3 bucket
Upload packages to s3 bucket
Call CloudFormation
Check CloudFormation stack status
Get info of created recourses (e.g. endpoint urls of created APIs)
And those calls can change dependent on what you are doing and what you have done before.
The point I'm trying to make is is that these calls which contain your credentials are not all located in one place and if you want to do a full code review of Serverless Framework and all it's dependencies, have fun with that.
But under the hood, we know that it's actually using the JavaScript aws-sdk (go check out the package.json), and we know what endpoints that uses {service}.{region}.amazonaws.com.
So to prove to your employers that nothing with your credentials is going anywhere except AWS you can just run a serverless deploy with wireshark running (other network packet analyzers are available). That way you can see anything that's not going to amazonaws.com
But wait, why are calls being made to serverless.com and serverlessteam.com when I run a deploy?
Well that's just tracking some stats and you can see what they track here. But if you are uber paranoid, this can be turned off with serverless slstats --disable.
After making some changes for an aws hosted static website, I deleted an aws s3 bucket through the AWS console. However, the bucket is now orphaned. Although it is not listed in the AWS console, I can see still reach what is left of it through the CLI and through the URI.
When I try to recreate a www bucket with the same name, the AWS console returns the following error:
Bucket already exists
The bucket with issues has a www prefix, so now I have two different versions (www and non-www) of the same website.
The problem URI is:
www.michaelrieder.com and www.michaelrieder.com.s3-eu-west-1.amazonaws.com
I made many failed attempts to delete the bucket using the aws s3 CLI utility. I tried aws rb force, aws rm, and any other command I remotely thought might work.
I need to delete and recreate the bucket with exactly the same name so I can have www website redirection working correctly as aws enforces static website naming conventions strictly.
When I execute the aws s3 CLI command for example:
aws s3 rb s3://www.michaelrieder.com --force --debug
A typical CLI error message is:
An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied
It thought it might be a cache related issue and that the the bucket would flush itself after a period of time, but the issue has persisted for over 48 hours.
It seems to be a permissions issue, but I cannot find a way to change the phantom bucket’s permissions or any method of deleting the bucket or even it’s individual objects, since I do not have access to the bucket via the AWS console or the aws s3 CLI.
Appreciate any ideas. Please help.
I'd like to know if there's a way for me to have bucket-level stats in amazon s3.
Basically i want to charge customers for storage and GET requests on my system (which is hosted on s3).
So i created a specific bucket for each client, but i can't seem to get the stats just for a specific bucket.
I see the API lets me
GET Bucket
or
GET Bucket requestPayment
But i just can't find how to get the number of requests issued to said bucket and the total size of the bucket.
Thanks for help !
Regards
I don't think that what you are trying to achieve is possible using Amazon API. The GET Bucket request does not contain usage statistics (requests, etc) other than the timestamp of the latest modification (LastModified).
My suggestion would be that you enable logging in your buckets and perform the analysis that you want from there.
S3 starting page gives you an overview on it:
Amazon S3 also supports logging of requests made against your Amazon S3 resources. You can configure your Amazon S3 bucket to create access log records for the requests made against it. These server access logs capture all requests made against a bucket or the objects in it and can be used for auditing purposes.
And I am sure there is plenty of documentation on that matter.
HTH.