Situation
It's pretty simple. I want to have more fine-tuned control of the s3 buckets created by serverless by managing them in the Resources section. I used to reference the bucket like this:
my-function:
handler: src/functions/my-function.handler
events:
- s3:
bucket: my-bucket-${opt:stage,self:provider.stage}
But now I want to reference it like this:
my-function:
handler: src/functions/my-function.handler
events:
- s3:
bucket: ${self:resources.Resources.MyBucket.Properties.BucketName}
...
Resources:
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-${self:provider.stage}
My understanding is that this will allow me more configuration options, let me include the bucket in outputs, etc...
However, I get this error when deploying to my dev stage:
MyBucket - my-bucket-dev already exists in stack <stack info>
Question
Is there a way I can make this syntax change in Serverless without it throwing an error? I also didn't want to use existing: true because I'll be deploying to stages where the bucket won't exist
You need to match the Logical IDs for the bucket and lambda permission.
See this example taken from the docs:
functions:
resize:
handler: resize.handler
events:
- s3: photos
resources:
Resources:
S3BucketPhotos:
Type: AWS::S3::Bucket
Properties:
BucketName: my-custom-bucket-name
# add additional custom bucket configuration here
ResizeLambdaPermissionPhotosS3:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName:
'Fn::GetAtt':
- ResizeLambdaFunction
- Arn
Principal: 's3.amazonaws.com'
Action: 'lambda:InvokeFunction'
SourceAccount:
Ref: AWS::AccountId
SourceArn: 'arn:aws:s3:::my-custom-bucket-name'
https://serverless.com/framework/docs/providers/aws/events/s3#custom-bucket-configuration
Related
I am trying to build s3 event application by referring the environment variable defined.
Below the template i am referring
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Sub "${S3}"
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: ""How can refer the s3 bucket form the environment here"
Events: s3:ObjectCreated:*
i actually need to refer the 'BUCKET NAME' from the 'Environment variable' and also need to add event trigger from the same s3.
can anyone help on this?
Thanks
There are two options,
Either you have defined the S3 bucket in the same template, and then you can reference it using the logical CFN ID:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-name
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Ref S3Bucket
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: !Ref S3Bucket
Events: s3:ObjectCreated:*
Or you do not have it defined in the same template, in which case you have to pass it to the stack as a parameter during deployment like this:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
S3Bucket:
Description: Name of the S3 Bucket
Type: String
Resources:
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Ref S3Bucket
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: !Ref S3Bucket
Events: s3:ObjectCreated:*
I have defined an S3 bucket in my serverless.yml file as follows:
resources:
Resources:
nameOfS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:custom.stage}-name-of-s3-bucket
AccessControl: Private
What I would like to do now is to reference the arn of the bucket in an IAM role which is also defined as a resource e.g.
resources:
Resources:
# s3 bucket definition removed for brevity
iamRoleName:
Type: AWS::IAM::Role
Properties:
RoleName: ${self:custom.stage}-iam-role-name
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: ${self:custom.stage}-iam-role-policy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "s3:PutObject"
Resource: *** Need the ARN of the s3 bucket here***
Is there a way to do this?
In my experience, the intrinsic functions in cloudformation work fine inside serverless.yml, but not the yaml short form.
i.e. use Fn::GetAtt over !GetAtt
As long as you use the full syntax for yaml, it should work.
Try this (untested)
resources:
Resources:
nameOfS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: ${self:custom.stage}-name-of-s3-bucket
AccessControl: Private
iamRoleName:
Type: AWS::IAM::Role
Properties:
RoleName: ${self:custom.stage}-iam-role-name
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: ${self:custom.stage}-iam-role-policy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: "s3:PutObject"
Resource:
"Fn::GetAtt": [ nameOfS3Bucket, Arn ]
I am writing a CloudFormation that creates an S3 bucket and an SQS queue. In the same CFN, I am trying to allow the S3 bucket to send messages to the SQS queue. I keep getting this error:
Unable to validate the following destination configurations
I spent about 12 hours in trial and error yesterday. I think I have now read every article out there about this topic. Most seem to be on Lambdas or SNS Topics, but the process should be the same. I have tried adding a DependsOn in the S3 bucket so that the SQS policy gets created before the bucket, but that didn't work. I then did the other solution suggested which was to create the bucket and sqs, then create the policy for the sqs bucket that allows the s3 to publish to it, and finally create the notification. I was successful up to creating the notification.
Here is my sqs creation:
SQSBiaData:
Type: AWS::SQS::Queue
Properties:
QueueName: !Ref Queue
Tags:
- Key: 'Environment'
Value: !Ref Environment
Here is my sqs policy:
SQSBiaPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref SQSBiaData
PolicyDocument:
Statement:
- Sid: SendMessage
Effect: Allow
Principal:
AWS: !Ref AccountNumber
Action: SQS:SendMessage
Resource:
- 'arn:aws:sqs:us-east-1:123456:bia-data'
Condition:
ArnLike:
aws:SourceArn:
Fn::Join:
- ''
- - 'arn:aws:s3:*:*:'
- 'bia-data'
Here is my bucket creation:
S3BucketBiaData:
Type: AWS::S3::Bucket
DependsOn:
- SQSBiaPolicy
DeletionPolicy: Retain
Properties:
AccessControl: Private
BucketName: !Ref BucketName
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
Tags:
- Key: 'Environment'
Value: !Ref Environment
NotificationConfiguration:
QueueConfigurations:
- Event: 's3:ObjectCreated:*'
Queue: 'arn:aws:sqs:us-east-1:123456:bia-data'
Filter:
S3Key:
Rules:
- Name: 'prefix'
Value: 'manifest/'
Finally, here is my bucket policy:
S3BucketBiaPolicies:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref BucketName
PolicyDocument:
Statement:
- Sid: DenyInsecureConnections
Effect: Deny
Principal: '*'
Action: s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
Bool:
aws:SecureTransport: 'false'
- Sid: DenyPublicReadGrant
Effect: Deny
Principal:
AWS: "*"
Action:
- s3:PutObject
- s3:PutObjectAcl
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
StringLike:
s3:x-amz-grant-read:
- "*http://acs.amazonaws.com/groups/global/AllUsers*"
- "*http://acs.amazonaws.com/groups/global/AuthenticatedUsers*"
- Sid: OnlyAllowVPCAccess
Effect: Deny
Principal: '*'
Action: s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
ForAnyValue:StringNotEquals:
'aws: sourceVpce': !Ref VPCs
I am expecting the CFN to happen in this order:
SQS
SQS Policy
S3 Bucket
S3 Bucket Policy
Can anyone tell me what I am doing wrong? Keep in mind that the CFN works UNTIL I add the NotificationConfiguration section.
Thanks!
While I do not know the exact issue of your problem, I'd like to suggest heavier usage of Pseudo params to make debugging easier.
I would replace the "us-east-1"s with ${AWS::Region} and "123456"s with ${AWS::AccountId}.
I would also use a !GetAtt when referencing the Queue in the S3 definition. Instead of Queue: 'arn:aws:sqs:us-east-1:123456:bia-data'
Queue: !GetAtt SQSBiaData.Arn
Finally, I would use the parameter "BucketName" as a Ref in the last line of your SQS policy.
These changes might make debugging easier for someone on here.
Created cloud formation template to create bucket with notification.
Following is code:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
CBRS3ToS3IADelay:
Description: Number of days before an S3 object is transitioned from S3 to S3-IA
Type: Number
Default: 365
CBRS3ToGlacierDelay:
Description: Number of days before an S3-IA object is transitioned from S3-IA to Glacier.
Type: Number
Default: 1460
CBRBucketName:
Description: S3 bucket name
Type: String
Default: "my-bucket-test0011"
Resources:
CBRS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Ref: CBRBucketName
AccessControl: Private
LifecycleConfiguration:
Rules:
- Id: CbrCertReportGlacierArchiveRule
Status: Enabled
Transitions:
- StorageClass: STANDARD_IA
TransitionInDays: !Ref CBRS3ToS3IADelay
- StorageClass: GLACIER
TransitionInDays: !Ref CBRS3ToGlacierDelay
NotificationConfiguration:
LambdaConfigurations:
-
Function: "arn:aws:lambda:xxxx:xxxx:function:xxxx"
Event: "s3:ObjectCreated:Put"
Filter:
S3Key:
Rules:
-
Name: suffix
Value: ".gz"
Tags:
- Key: PRODUCT
Value: CRAWS
VersioningConfiguration:
Status: Enabled
Code working with notification block.
But above template is not working with notification.
Getting following error:
Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument
I able to do from console.
Anyone help me to fix this issue?
this is late, so more of answering myself for this question (just managed to fix the same problem): it fails due to a preliminary check on s3 to invoke that lambda function, we will need this:
CBRS3BucketCanInvokeFunctionX:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName: ARN_OF_FUNCTION_X
Action: 'lambda:InvokeFunction'
Principal: s3.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
SourceArn: !Sub 'arn:aws:s3:::${CBRBucketName}'
your CBRS3Bucket will also need to let above resource run first:
CBRS3Bucket:
Type: AWS::S3::Bucket
DependsOn: CBRS3BucketCanInvokeFunctionX
Try taking the .gz and put in just gz.
Here is a snippet from my serverless.yml file:
Resources:
LogGroupInfo:
Type: 'AWS::Logs::LogGroup'
Properties:
RetentionInDays: 3
FirehoseInstance:
Properties:
DeliveryStreamName: ${opt:stage}-analytics
DeliveryStreamType: DirectPut
RedshiftDestinationConfiguration:
CloudWatchLoggingOptions:
Enabled: true
LogGroupName: !Ref LogGroupInfo
Here is the error I receive:
unknown tag !<!Ref> in "/Users/code/Project1/serverless.yml" at line 56, column 42:
... LogGroupName: !Ref LogGroupInfo
This template works perfectly well as it is when used in cloudformation to create a stack.
Why is !Ref being rejected by serverless.yml ?
The shortcut syntax of !Ref is not yet supported within the serverless framework.
As the bug ticket below suggests, you have to use the object based form for now.
LogGroupName:
Ref: LogGroupInfo
Failure to Create Resource Using !Ref
The feature is currently tracked through that issue:
Fn::Sub and !Sub