Cloudwatch Event Rule doesn't invoke when source state machine fails - amazon-cloudwatch

I have a step function that runs 2 separate lambdas. If the step function fails or times out, I want to get an email via SNS telling me the step function failed. I created the event rule using cloudformation and specified the statemachine ARN in the event pattern. When the step function fails, no email is sent out. If I remove the stateMachineArn parameter and run my step function, I get the failure email. I've double checked numerous times that I'm entering the correct ARN for the state machine. CF for the Event Rule is below (in YAML format). Thanks.
FailureEvent:
Type: AWS::Events::Rule
DependsOn:
- StateMachine
Properties:
Name: !Ref FailureRuleName
Description: "EventRule"
EventPattern:
detail-type:
- "Step Functions Execution Status Change"
detail:
status:
- "FAILED"
- "TIMED_OUT"
stateMachineArn: ["arn:aws:states:region:account#:stateMachine:statemachine"]
Targets:
-
Arn:
Ref: SNSARN
Id: !Ref SNSTopic

I did get this fixed and expanded on it to invoke a lambda that publishes a custom SNS email using a lambda. My alignment was off in my EventPattern section. See below. Thanks to #Marcin.
FailureEvent:
Type: AWS::Events::Rule
DependsOn:
- FMIStateMachine
Properties:
Description: !Ref FailureRuleDescription
Name: !Ref FailureRuleName
State: "ENABLED"
RoleArn:
'Fn::Join': ["", ['arn:aws:iam::', !Ref 'AWS::AccountId', ':role/', !Ref LambdaExecutionRole]]
EventPattern:
detail-type:
- "Step Functions Execution Status Change"
detail:
status:
- "FAILED"
- "TIMED_OUT"
stateMachineArn: [!Ref StateMachine]
Targets:
- Arn:
'Fn::Join': ["", ['arn:aws:lambda:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':function:', !Ref FailureLambda]]
Id: !Ref FailureLambda
Input: !Sub '{"failed_service": "${StateMachineName}","sns_arn": "arn:aws:sns:${AWS::Region}:${AWS::AccountId}:${SNSTopic}"}'

Related

EventRule matched by default EventBridge EventBus ignored by custom EventBus

I have a Cloudformation template which implements a CI/CD process for Lambda functions (at bottom)
Essentially it -
watches a Github repo
pulls source code on new git tag creation
starts a CodeBuild process which runs unit tests, zips source code (on test success) and pushes archive to S3
enables CodeBuild notifications
implements a CloudWatch EventRule to pattern match raw CodeBuild notifications, format and push them to SNS
binds a Lambda function to SNS, which pushes the notifications to Slack via a webhook
This works fine with the default EventBridge EventBus, but the pattern matching seems to fail if I switch to a custom EventBus.
(see resources EventBus and EventRule in the stack; it's currently set up to use the custom EventBus, and fails / ignores new git tags; if you comment out the reference to property EventBusName in EventBus, it defaults to using the default EventBus and works)
Why would a custom EventBus behave differently to the default EventBus in this situation ?
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
AppName:
Type: String
RepoOwner:
Type: String
RepoName:
Type: String
RepoBranch:
Type: String
Default: master
RepoAuth:
Type: String
Default: master
WebhookUrl:
Type: String
WebhookLambda:
Type: String
CodeBuildBuildSpec:
Type: String
CodeBuildType:
Type: String
Default: LINUX_CONTAINER
CodeBuildComputeType:
Type: String
Default: BUILD_GENERAL1_SMALL
CodeBuildImage:
Type: String
Default: aws/codebuild/standard:4.0
LambdaHandler:
Type: String
Default: "index.handler"
LambdaMemory:
Type: Number
Default: 128
LambdaTimeout:
Type: Number
Default: 30
LambdaRuntime:
Type: String
Default: python3.8
Resources:
ArtifactsBucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Fn::Sub:
- ${app_name}-lambda-artifacts
- app_name:
Ref: AppName
CodeBuildProject:
Properties:
Environment:
ComputeType:
Ref: CodeBuildComputeType
Image:
Ref: CodeBuildImage
Type:
Ref: CodeBuildType
Name:
Fn::Sub:
- ${app_name}-lambda-ci
- app_name:
Ref: AppName
ServiceRole:
Fn::GetAtt:
- CodeBuildRole
- Arn
Source:
Auth:
Resource:
Ref: RepoAuth
Type: OAUTH
Location:
Fn::Sub:
- "https://github.com/${repo_owner}/${repo_name}.git"
- repo_owner:
Ref: RepoOwner
repo_name:
Ref: RepoName
Type: GITHUB
BuildSpec:
Fn::Sub:
- "${build_spec}"
- build_spec:
Ref: CodeBuildBuildSpec
Artifacts:
Type: NO_ARTIFACTS
SourceVersion:
Ref: RepoBranch
Triggers:
Webhook: true
FilterGroups:
- - Type: EVENT
Pattern: PUSH
ExcludeMatchedPattern: false
- Type: HEAD_REF
Pattern: "refs/tags/.*"
ExcludeMatchedPattern: false
Type: AWS::CodeBuild::Project
CodeBuildRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- codebuild:*
- events:*
- s3:PutObject
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName: code-build-role-policy
Type: AWS::IAM::Role
WebhookFunction:
Properties:
FunctionName:
Fn::Sub:
- ${app_name}-lambda-webhook
- app_name:
Ref: AppName
Code:
ZipFile:
Ref: WebhookLambda
Environment:
Variables:
WEBHOOK_URL:
Ref: WebhookUrl
Handler:
Ref: LambdaHandler
MemorySize:
Ref: LambdaMemory
Role:
Fn::GetAtt:
- WebhookFunctionRole
- Arn
Runtime:
Ref: LambdaRuntime
Timeout:
Ref: LambdaTimeout
Type: AWS::Lambda::Function
WebhookFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName: webhook-role-policy
Type: AWS::IAM::Role
WebhookFunctionPermission:
Properties:
Action: "lambda:InvokeFunction"
FunctionName:
Ref: WebhookFunction
Principal: "sns.amazonaws.com"
SourceArn:
Ref: WebhookTopic
Type: AWS::Lambda::Permission
WebhookTopic:
Properties:
Subscription:
- Protocol: lambda
Endpoint:
Fn::GetAtt:
- WebhookFunction
- Arn
Type: AWS::SNS::Topic
WebhookTopicPolicy:
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: "events.amazonaws.com"
Action:
- "sns:Publish"
Resource:
Ref: WebhookTopic
Topics:
- Ref: WebhookTopic
Type: AWS::SNS::TopicPolicy
EventBus:
Type: AWS::Events::EventBus
Properties:
Name:
Fn::Sub:
- ${app_name}-bus
- app_name:
Ref: AppName
EventRule:
Type: AWS::Events::Rule
Properties:
EventBusName: # CURRENTLY USING CUSTOM EVENT BUS (PATTERN MATCHING FAILS); REMOVE THIS PROPERTY TO SWITCH TO DEFAULT EVENT BUS (PATTERN MATCHING WORKS)
Ref: EventBus
EventPattern:
source:
- "aws.codebuild"
detail-type:
- "CodeBuild Build Phase Change"
detail:
completed-phase:
- SUBMITTED
- PROVISIONING
- DOWNLOAD_SOURCE
- INSTALL
- PRE_BUILD
- BUILD
- POST_BUILD
- UPLOAD_ARTIFACTS
- FINALIZING
completed-phase-status:
- TIMED_OUT
- STOPPED
- FAILED
- SUCCEEDED
- FAULT
- CLIENT_ERROR
project-name:
- Ref: CodeBuildProject
State: ENABLED
Targets:
- Arn:
Ref: WebhookTopic
Id:
Fn::Sub:
- "${project_name}-codebuild-notifications"
- project_name:
Ref: CodeBuildProject
InputTransformer:
InputPathsMap:
build-id: "$.detail.build-id"
project-name: "$.detail.project-name"
completed-phase: "$.detail.completed-phase"
completed-phase-status: "$.detail.completed-phase-status"
InputTemplate: |
"{'build-id': '<build-id>', 'project-name': '<project-name>', 'completed-phase': '<completed-phase>', 'completed-phase-status': '<completed-phase-status>'}"
"Custom event buses serve a use case of receiving events from your custom applications and services. Unfortunately, it is not possible for AWS services to push events to a custom event bus." (AWS Support)

SNS topic not receiving CodeBuild notifications

Trying to get CodeBuild to push notifications to an SNS topic (bound to a Lambda), via a CloudWatch Events Rule.
Cloudformation template (see below) deploys fine.
CodeBuild process works fine (have tested).
SNS topic and bound Lambda work fine - I can push a message to the topic via AWS CLI and see Lambda dump that message to Cloudwatch logs.
The Cloudwatch Event Rule seems to be configured fine - I can see it in the cosole, it looks well formed, seems to be bound to SNS topic.
In addition I have been careful to give the Event Rule a role with permissions to sns:Publish, and also defined an AWS::SNS::TopicPolicy for the SNS topic -
Unable to successfully set up SNS on CodeBuild project through CFT but works manually
But still nothing - CodeBuild successfully completes but I don't receive any notifications.
Any thoughts as to what might be wrong ?
TIA :)
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
AppName:
Type: String
StagingBucket:
Type: String
RepoOwner:
Type: String
RepoName:
Type: String
RepoBranch:
Type: String
RepoPAT:
Type: String
CodeBuildBuildSpec:
Type: String
CodeBuildType:
Type: String
Default: LINUX_CONTAINER
CodeBuildComputeType:
Type: String
Default: BUILD_GENERAL1_SMALL
CodeBuildImage:
Type: String
Default: aws/codebuild/standard:4.0
LambdaHandler:
Type: String
Default: "index.handler"
LambdaMemory:
Type: Number
Default: 128
LambdaTimeout:
Type: Number
Default: 30
LambdaRuntime:
Type: String
Default: python3.8
Resources:
CodeBuildProject:
DependsOn:
- CodeBuildSourceCredential
Properties:
Environment:
ComputeType:
Ref: CodeBuildComputeType
Image:
Ref: CodeBuildImage
Type:
Ref: CodeBuildType
Name:
Ref: AppName
ServiceRole:
Fn::GetAtt:
- CodeBuildRole
- Arn
Source:
Location:
Fn::Sub:
- "https://github.com/${repo_owner}/${repo_name}.git"
- repo_owner:
Ref: RepoOwner
repo_name:
Ref: RepoName
Type: GITHUB
BuildSpec:
Fn::Sub:
- "${build_spec}"
- build_spec:
Ref: CodeBuildBuildSpec
Artifacts:
Type: S3
Location:
Ref: StagingBucket
SourceVersion:
Ref: RepoBranch
Triggers:
Webhook: true
FilterGroups:
- - Type: EVENT
Pattern: PUSH
ExcludeMatchedPattern: false
- Type: HEAD_REF
Pattern: "refs/tags/.*"
ExcludeMatchedPattern: false
Type: AWS::CodeBuild::Project
CodeBuildSourceCredential:
Type: AWS::CodeBuild::SourceCredential
Properties:
Token:
Ref: RepoPAT
ServerType: GITHUB
AuthType: PERSONAL_ACCESS_TOKEN
CodeBuildRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Version: '2012-10-17'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
Path: /
Type: AWS::IAM::Role
CodeBuildNotificationFunction:
Properties:
Code:
ZipFile: "def handler(event, context):\n print (event)"
Handler:
Ref: LambdaHandler
MemorySize:
Ref: LambdaMemory
Role:
Fn::GetAtt:
- CodeBuildNotificationFunctionRole
- Arn
Runtime:
Ref: LambdaRuntime
Timeout:
Ref: LambdaTimeout
Type: AWS::Lambda::Function
CodeBuildNotificationFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName: code-build-notification-role-policy
Type: AWS::IAM::Role
CodeBuildNotificationTopic:
Properties:
Subscription:
- Protocol: lambda
Endpoint:
Fn::GetAtt:
- CodeBuildNotificationFunction
- Arn
Type: AWS::SNS::Topic
CodeBuildNotificationTopicPolicy:
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: "events.amazonaws.com"
Action:
- "sns:Publish"
Resource:
Ref: CodeBuildNotificationTopic
Topics:
- Ref: CodeBuildNotificationTopic
Type: AWS::SNS::TopicPolicy
CodeBuildNotificationLambdaInvokePermission:
Properties:
Action: "lambda:InvokeFunction"
FunctionName:
Ref: CodeBuildNotificationFunction
Principal: "sns.amazonaws.com"
SourceArn:
Ref: CodeBuildNotificationTopic
Type: AWS::Lambda::Permission
SampleNotificationRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
Fn::Sub:
- '{"source": ["aws.codebuild"], "detail-type": ["Codebuild Build Phase Change"], "detail": {"completed-phase": ["SUBMITTED", "PROVISIONING", "DOWNLOAD_SOURCE", "INSTALL", "PRE_BUILD", "BUILD", "POST_BUILD", "UPLOAD_ARTIFACTS", "FINALIZING"], "completed-phase-status": ["TIMED_OUT", "STOPPED", "FAILED", "SUCCEEDED", "FAULT", "CLIENT_ERROR"], "project-name": ["${project_name}"]}}'
- project_name:
Ref: CodeBuildProject
State: ENABLED
RoleArn:
Fn::GetAtt:
- SampleNotificationRuleRole
- Arn
Targets:
- Arn:
Ref: CodeBuildNotificationTopic
Id: sample-notification
SampleNotificationRuleRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: events.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- "sns:Publish"
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName: sample-notification-rule-role-policy
Type: AWS::IAM::Role
Fixed - AWS::Events::Rule target was missing an InputTransformer block like so
Targets:
- Arn:
Ref: CodeBuildNotificationTopic
Id: sample-notification
InputTransformer:
InputPathsMap:
build-id: "$.detail.build-id"
project-name: "$.detail.project-name"
completed-phase: "$.detail.completed-phase"
completed-phase-status: "$.detail.completed-phase-status"
InputTemplate: |
"Build '<build-id>' for build project '<project-name>' has completed the build phase of '<completed-phase>' with a status of '<completed-phase-status>'."

How to fix "Unable to validate the following destination configurations" in AWS CloudFormation?

I am writing a CloudFormation that creates an S3 bucket and an SQS queue. In the same CFN, I am trying to allow the S3 bucket to send messages to the SQS queue. I keep getting this error:
Unable to validate the following destination configurations
I spent about 12 hours in trial and error yesterday. I think I have now read every article out there about this topic. Most seem to be on Lambdas or SNS Topics, but the process should be the same. I have tried adding a DependsOn in the S3 bucket so that the SQS policy gets created before the bucket, but that didn't work. I then did the other solution suggested which was to create the bucket and sqs, then create the policy for the sqs bucket that allows the s3 to publish to it, and finally create the notification. I was successful up to creating the notification.
Here is my sqs creation:
SQSBiaData:
Type: AWS::SQS::Queue
Properties:
QueueName: !Ref Queue
Tags:
- Key: 'Environment'
Value: !Ref Environment
Here is my sqs policy:
SQSBiaPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref SQSBiaData
PolicyDocument:
Statement:
- Sid: SendMessage
Effect: Allow
Principal:
AWS: !Ref AccountNumber
Action: SQS:SendMessage
Resource:
- 'arn:aws:sqs:us-east-1:123456:bia-data'
Condition:
ArnLike:
aws:SourceArn:
Fn::Join:
- ''
- - 'arn:aws:s3:*:*:'
- 'bia-data'
Here is my bucket creation:
S3BucketBiaData:
Type: AWS::S3::Bucket
DependsOn:
- SQSBiaPolicy
DeletionPolicy: Retain
Properties:
AccessControl: Private
BucketName: !Ref BucketName
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
SSEAlgorithm: AES256
Tags:
- Key: 'Environment'
Value: !Ref Environment
NotificationConfiguration:
QueueConfigurations:
- Event: 's3:ObjectCreated:*'
Queue: 'arn:aws:sqs:us-east-1:123456:bia-data'
Filter:
S3Key:
Rules:
- Name: 'prefix'
Value: 'manifest/'
Finally, here is my bucket policy:
S3BucketBiaPolicies:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref BucketName
PolicyDocument:
Statement:
- Sid: DenyInsecureConnections
Effect: Deny
Principal: '*'
Action: s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
Bool:
aws:SecureTransport: 'false'
- Sid: DenyPublicReadGrant
Effect: Deny
Principal:
AWS: "*"
Action:
- s3:PutObject
- s3:PutObjectAcl
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
StringLike:
s3:x-amz-grant-read:
- "*http://acs.amazonaws.com/groups/global/AllUsers*"
- "*http://acs.amazonaws.com/groups/global/AuthenticatedUsers*"
- Sid: OnlyAllowVPCAccess
Effect: Deny
Principal: '*'
Action: s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketBiaData}/*'
Condition:
ForAnyValue:StringNotEquals:
'aws: sourceVpce': !Ref VPCs
I am expecting the CFN to happen in this order:
SQS
SQS Policy
S3 Bucket
S3 Bucket Policy
Can anyone tell me what I am doing wrong? Keep in mind that the CFN works UNTIL I add the NotificationConfiguration section.
Thanks!
While I do not know the exact issue of your problem, I'd like to suggest heavier usage of Pseudo params to make debugging easier.
I would replace the "us-east-1"s with ${AWS::Region} and "123456"s with ${AWS::AccountId}.
I would also use a !GetAtt when referencing the Queue in the S3 definition. Instead of Queue: 'arn:aws:sqs:us-east-1:123456:bia-data'
Queue: !GetAtt SQSBiaData.Arn
Finally, I would use the parameter "BucketName" as a Ref in the last line of your SQS policy.
These changes might make debugging easier for someone on here.

unable to add notificationconfiguration to s3 bucket

Created cloud formation template to create bucket with notification.
Following is code:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
CBRS3ToS3IADelay:
Description: Number of days before an S3 object is transitioned from S3 to S3-IA
Type: Number
Default: 365
CBRS3ToGlacierDelay:
Description: Number of days before an S3-IA object is transitioned from S3-IA to Glacier.
Type: Number
Default: 1460
CBRBucketName:
Description: S3 bucket name
Type: String
Default: "my-bucket-test0011"
Resources:
CBRS3Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Ref: CBRBucketName
AccessControl: Private
LifecycleConfiguration:
Rules:
- Id: CbrCertReportGlacierArchiveRule
Status: Enabled
Transitions:
- StorageClass: STANDARD_IA
TransitionInDays: !Ref CBRS3ToS3IADelay
- StorageClass: GLACIER
TransitionInDays: !Ref CBRS3ToGlacierDelay
NotificationConfiguration:
LambdaConfigurations:
-
Function: "arn:aws:lambda:xxxx:xxxx:function:xxxx"
Event: "s3:ObjectCreated:Put"
Filter:
S3Key:
Rules:
-
Name: suffix
Value: ".gz"
Tags:
- Key: PRODUCT
Value: CRAWS
VersioningConfiguration:
Status: Enabled
Code working with notification block.
But above template is not working with notification.
Getting following error:
Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument
I able to do from console.
Anyone help me to fix this issue?
this is late, so more of answering myself for this question (just managed to fix the same problem): it fails due to a preliminary check on s3 to invoke that lambda function, we will need this:
CBRS3BucketCanInvokeFunctionX:
Type: 'AWS::Lambda::Permission'
Properties:
FunctionName: ARN_OF_FUNCTION_X
Action: 'lambda:InvokeFunction'
Principal: s3.amazonaws.com
SourceAccount: !Ref 'AWS::AccountId'
SourceArn: !Sub 'arn:aws:s3:::${CBRBucketName}'
your CBRS3Bucket will also need to let above resource run first:
CBRS3Bucket:
Type: AWS::S3::Bucket
DependsOn: CBRS3BucketCanInvokeFunctionX
Try taking the .gz and put in just gz.

Serverless.yml rejecting cloudformation "Ref" functions

Here is a snippet from my serverless.yml file:
Resources:
LogGroupInfo:
Type: 'AWS::Logs::LogGroup'
Properties:
RetentionInDays: 3
FirehoseInstance:
Properties:
DeliveryStreamName: ${opt:stage}-analytics
DeliveryStreamType: DirectPut
RedshiftDestinationConfiguration:
CloudWatchLoggingOptions:
Enabled: true
LogGroupName: !Ref LogGroupInfo
Here is the error I receive:
unknown tag !<!Ref> in "/Users/code/Project1/serverless.yml" at line 56, column 42:
... LogGroupName: !Ref LogGroupInfo
This template works perfectly well as it is when used in cloudformation to create a stack.
Why is !Ref being rejected by serverless.yml ?
The shortcut syntax of !Ref is not yet supported within the serverless framework.
As the bug ticket below suggests, you have to use the object based form for now.
LogGroupName:
Ref: LogGroupInfo
Failure to Create Resource Using !Ref
The feature is currently tracked through that issue:
Fn::Sub and !Sub