Created cloud formation template to create bucket with notification.
Following is code:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
CBRS3ToS3IADelay:
Description: Number of days before an S3 object is transitioned from S3 to S3-IA
Type: Number
Default: 365
CBRS3ToGlacierDelay:
Description: Number of days before an S3-IA object is transitioned from S3-IA to Glacier.
Type: Number
Default: 1460
CBRBucketName:
Description: S3 bucket name
Type: String
Default: "my-bucket-test0011"
Resources:
CBRS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Ref: CBRBucketName
AccessControl: Private
LifecycleConfiguration:
Rules:
- Id: CbrCertReportGlacierArchiveRule
Status: Enabled
Transitions:
- StorageClass: STANDARD_IA
TransitionInDays: !Ref CBRS3ToS3IADelay
- StorageClass: GLACIER
TransitionInDays: !Ref CBRS3ToGlacierDelay
NotificationConfiguration:
LambdaConfigurations:
-
Function: "arn:aws:lambda:xxxx:xxxx:function:xxxx"
Event: "s3:ObjectCreated:Put"
Filter:
S3Key:
Rules:
-
Name: suffix
Value: ".gz"
Tags:
- Key: PRODUCT
Value: CRAWS
VersioningConfiguration:
Status: Enabled
Code working with notification block.
But above template is not working with notification.
Getting following error:
Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument
I able to do from console.
Anyone help me to fix this issue?
this is late, so more of answering myself for this question (just managed to fix the same problem): it fails due to a preliminary check on s3 to invoke that lambda function, we will need this:
CBRS3BucketCanInvokeFunctionX:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName: ARN_OF_FUNCTION_X
Action: 'lambda:InvokeFunction'
Principal: s3.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
SourceArn: !Sub 'arn:aws:s3:::${CBRBucketName}'
your CBRS3Bucket will also need to let above resource run first:
CBRS3Bucket:
Type: AWS::S3::Bucket
DependsOn: CBRS3BucketCanInvokeFunctionX
Try taking the .gz and put in just gz.
Related
Transform: AWS::Serverless-2016-10-31
Description: >
patientcheckout
Sample SAM Template for patientcheckout
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 20
Runtime: java11
#Architectures:
# - x86_64
MemorySize: 512
Resources:
PatientCheckoutBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub ${AWS::StackName}-${AWS::AccountId}-${AWS::Region}
PatientCheckoutFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: patientcheckout
Handler: com.harsha.aws.lambda.patientcheckout.PatientCheckoutLambda::handler
Policies:
- S3ReadPolicy:
BucketName: !Sub ${AWS::StackName}-${AWS::AccountId}-${AWS::Region}
Events:
S3Event:
Type: S3
Properties:
Bucket: !Ref PatientCheckoutBucket
Events: s3:ObjectCreate:*
The event is not supported notifications (Service: Amazon S3 Status Code: 400; Error InvalidArgument;
I get this when error I run the code with sam cli. sam build is success, but s3 bucket creation fails
I am trying to build s3 event application by referring the environment variable defined.
Below the template i am referring
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Sub "${S3}"
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: ""How can refer the s3 bucket form the environment here"
Events: s3:ObjectCreated:*
i actually need to refer the 'BUCKET NAME' from the 'Environment variable' and also need to add event trigger from the same s3.
can anyone help on this?
Thanks
There are two options,
Either you have defined the S3 bucket in the same template, and then you can reference it using the logical CFN ID:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
S3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-name
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Ref S3Bucket
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: !Ref S3Bucket
Events: s3:ObjectCreated:*
Or you do not have it defined in the same template, in which case you have to pass it to the stack as a parameter during deployment like this:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
S3Bucket:
Description: Name of the S3 Bucket
Type: String
Resources:
CreateThumbnail:
Type: AWS::Serverless::Function
Properties:
Handler: handler
Runtime: runtime
Timeout: 60
Policies: AWSLambdaExecute
Environment:
Variables:
BUCKET_NAME: !Ref S3Bucket
Events:
CreateThumbnailEvent:
Type: S3
Properties:
Bucket: !Ref S3Bucket
Events: s3:ObjectCreated:*
Situation
It's pretty simple. I want to have more fine-tuned control of the s3 buckets created by serverless by managing them in the Resources section. I used to reference the bucket like this:
my-function:
handler: src/functions/my-function.handler
events:
- s3:
bucket: my-bucket-${opt:stage,self:provider.stage}
But now I want to reference it like this:
my-function:
handler: src/functions/my-function.handler
events:
- s3:
bucket: ${self:resources.Resources.MyBucket.Properties.BucketName}
...
Resources:
MyBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: my-bucket-${self:provider.stage}
My understanding is that this will allow me more configuration options, let me include the bucket in outputs, etc...
However, I get this error when deploying to my dev stage:
MyBucket - my-bucket-dev already exists in stack <stack info>
Question
Is there a way I can make this syntax change in Serverless without it throwing an error? I also didn't want to use existing: true because I'll be deploying to stages where the bucket won't exist
You need to match the Logical IDs for the bucket and lambda permission.
See this example taken from the docs:
functions:
resize:
handler: resize.handler
events:
- s3: photos
resources:
Resources:
S3BucketPhotos:
Type: AWS::S3::Bucket
Properties:
BucketName: my-custom-bucket-name
# add additional custom bucket configuration here
ResizeLambdaPermissionPhotosS3:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName:
'Fn::GetAtt':
- ResizeLambdaFunction
- Arn
Principal: 's3.amazonaws.com'
Action: 'lambda:InvokeFunction'
SourceAccount:
Ref: AWS::AccountId
SourceArn: 'arn:aws:s3:::my-custom-bucket-name'
https://serverless.com/framework/docs/providers/aws/events/s3#custom-bucket-configuration
I am creating a cloudformation stack where the templates creates Cloud trail and then S3 bucket and pushes all logs to S3 bucket.
I have tried creating Cloud trail,s3 bucket and tried attaching the s3 Bucket Policy to the bucket
Parameters:
loggroupname:
Type: String
trailname:
Type: String
s3bucketname:
Type: String
Resources:
createloggroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub ${loggroupname}
creates3bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub ${s3bucketname}
s3bucketpolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Sub ${s3bucketname}
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: 'AWSCloudTrailAclCheck20150319'
Effect: 'Allow'
Principal:
Service: 'cloudtrail.amazonaws.com'
Action: 's3:GetBucketAcl'
Resource:
!Sub 'arn:aws:s3:::${s3bucketname}'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AWSCloudTrailWrite20150319
Effect: 'Allow'
Principal:
Service: 'cloudtrail.amazonaws.com'
Action: 's3:PutObject'
Resource:
!Sub 'arn:aws:s3:::${s3bucketname}/AWSLogs/${AWS::AccountId}/*'
Condition:
StringsEquals:
s3:x-amz-acl: 'bucket-owner-full-control'
myvpctrail:
DependsOn:
- s3bucketpolicy
Type: AWS::CloudTrail::Trail
Properties:
IsLogging: true
IsMultiRegionTrail: true
IncludeGlobalServiceEvents: true
S3BucketName: !Ref creates3bucket
Invalid Condition type : StringsEquals (Service: Amazon S3; Status Code: 400; Error Code: MalformedPolicy; Request ID: F7439B111E82A3FA; S3 Extended Request ID: IGU1L7BB77WcrhPtmydd5j6viQdMK0vqA3Qo4RTS209FAvjT3q6wBIsyabdt5B7pBFvdr2MT+sM=)
Simple typo.
It's StringEquals not StringsEquals.
Source: https://docs.aws.amazon.com/AmazonS3/latest/dev/amazon-s3-policy-keys.html
I am writing a CloudFormation that creates an S3 bucket and an SQS queue. In the same CFN, I am trying to allow the S3 bucket to send messages to the SQS queue. I keep getting this error:
Unable to validate the following destination configurations
I spent about 12 hours in trial and error yesterday. I think I have now read every article out there about this topic. Most seem to be on Lambdas or SNS Topics, but the process should be the same. I have tried adding a DependsOn in the S3 bucket so that the SQS policy gets created before the bucket, but that didn't work. I then did the other solution suggested which was to create the bucket and sqs, then create the policy for the sqs bucket that allows the s3 to publish to it, and finally create the notification. I was successful up to creating the notification.
Here is my sqs creation:
SQSBiaData:
Type: AWS::SQS::Queue
Properties:
QueueName: !Ref Queue
Tags:
- Key: 'Environment'
Value: !Ref Environment
Here is my sqs policy:
SQSBiaPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref SQSBiaData
PolicyDocument:
Statement:
- Sid: SendMessage
Effect: Allow
Principal:
AWS: !Ref AccountNumber
Action: SQS:SendMessage
Resource:
- 'arn:aws:sqs:us-east-1:123456:bia-data'
Condition:
ArnLike:
aws:SourceArn:
Fn::Join:
- ''
- - 'arn:aws:s3:*:*:'
- 'bia-data'
Here is my bucket creation:
S3BucketBiaData:
Type: AWS::S3::Bucket
DependsOn:
- SQSBiaPolicy
DeletionPolicy: Retain
Properties:
AccessControl: Private
BucketName: !Ref BucketName
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
Tags:
- Key: 'Environment'
Value: !Ref Environment
NotificationConfiguration:
QueueConfigurations:
- Event: 's3:ObjectCreated:*'
Queue: 'arn:aws:sqs:us-east-1:123456:bia-data'
Filter:
S3Key:
Rules:
- Name: 'prefix'
Value: 'manifest/'
Finally, here is my bucket policy:
S3BucketBiaPolicies:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref BucketName
PolicyDocument:
Statement:
- Sid: DenyInsecureConnections
Effect: Deny
Principal: '*'
Action: s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
Bool:
aws:SecureTransport: 'false'
- Sid: DenyPublicReadGrant
Effect: Deny
Principal:
AWS: "*"
Action:
- s3:PutObject
- s3:PutObjectAcl
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
StringLike:
s3:x-amz-grant-read:
- "*http://acs.amazonaws.com/groups/global/AllUsers*"
- "*http://acs.amazonaws.com/groups/global/AuthenticatedUsers*"
- Sid: OnlyAllowVPCAccess
Effect: Deny
Principal: '*'
Action: s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
ForAnyValue:StringNotEquals:
'aws: sourceVpce': !Ref VPCs
I am expecting the CFN to happen in this order:
SQS
SQS Policy
S3 Bucket
S3 Bucket Policy
Can anyone tell me what I am doing wrong? Keep in mind that the CFN works UNTIL I add the NotificationConfiguration section.
Thanks!
While I do not know the exact issue of your problem, I'd like to suggest heavier usage of Pseudo params to make debugging easier.
I would replace the "us-east-1"s with ${AWS::Region} and "123456"s with ${AWS::AccountId}.
I would also use a !GetAtt when referencing the Queue in the S3 definition. Instead of Queue: 'arn:aws:sqs:us-east-1:123456:bia-data'
Queue: !GetAtt SQSBiaData.Arn
Finally, I would use the parameter "BucketName" as a Ref in the last line of your SQS policy.
These changes might make debugging easier for someone on here.