How to define CF resource as function event source in serverless framework - serverless-framework

I'm trying to create a AWS Lambda with the serverless framework. The Lambda is triggered through an AWS IoT Topic Rule. In case the execution of the Rule fails I want to have an error action executed. The entire configuration should take place within the serverless.yml.
As far as I can tell from the documentation there is no option to describe an errorAction for an iot event:
functions:
foobar:
events:
- iot:
errorAction: ?
It is possible though to define a Cloud Formation resource with an ErrorAction inside the serverless.yml:
resources:
Resources:
FoobarIotTopicRule1:
Type: AWS::IoT::TopicRule
Properties:
ErrorAction:
Republish:
RoleArn: arn:aws:iam::1234567890:role/service-role/iot_execution_role
Topic: FAILURE
But then I don't know how to link the resource to act as a trigger of the Lambda function.
functions:
foobar:
handler: index.handler
events:
- iot:
name: iot_magic_rule
sql: "SELECT * FROM 'my/dedicated/topic'"
enabled: true
sqlVersion: '2016-03-23'
resources:
Resources:
FoobarIotTopicRule1:
Type: AWS::IoT::TopicRule
Properties:
RuleName: iot_magic_rule
TopicRulePayload:
AwsIotSqlVersion: '2016-03-23'
RuleDisabled: false
Sql: "SELECT * FROM 'my/dedicated/topic'"
ErrorAction:
Republish:
RoleArn: arn:aws:iam::1234567890:role/service-role/iot_execution_role
Topic: FAILURE
With the above configuration, trying to deploy on AWS fails as Cloud Formation tries to create the AWS IoT Topic Rule twice. Once for the definition in events and once as the defined resource FoobarIoTTopicRule1.
EDIT1
Defining the Lambda action inside the IoTTopicRule resource, creates the rule as intended, with Lambda action and error event. Unfortunately the rule does not show up as a trigger within the Lambda.

To be able to define an AWS IoT Topic Rule with an ErrorAction that will also show up as a trigger event on AWS Lambda, the configuration should look somewhat like this:
functions:
foobar:
handler: index.handler
resources:
Resources:
FoobarIotTopicRule1:
Type: AWS::IoT::TopicRule
Properties:
RuleName: iot_magic_rule
TopicRulePayload:
AwsIotSqlVersion: '2016-03-23'
RuleDisabled: false
Sql: "SELECT * FROM 'my/dedicated/topic'"
Actions:
- Lambda:
FunctionArn: { "Fn::GetAtt": ['FoobarLambdaFunction', 'Arn']}
ErrorAction:
Republish:
RoleArn: arn:aws:iam::1234567890:role/service-role/iot_execution_role
Topic: FAILURE
FoobarLambdaPermissionIotTopicRule1:
Type: AWS::Lambda::Permission
Properties:
FunctionName: { "Fn::GetAtt": [ "FoobarLambdaFunction", "Arn" ] }
Action: lambda:InvokeFunction
Principal: { "Fn::Join": ["", [ "iot.", { "Ref": "AWS::URLSuffix" } ]]}
SourceArn:
Fn::Join:
- ""
- - "arn:"
- "Ref": "AWS::Partition"
- ":iot:"
- "Ref": "AWS::Region"
- ":"
- "Ref": "AWS::AccountId"
- ":rule/"
- "Ref": "FoobarIotTopicRule1"

Related

Aurora PostgreSQL permissions to access s3

Im trying to give my Aurora PostgreSQL permissions to access an s3 bucket. I'm using the serverless framework and have the following code.
RDSCluster:
Type: 'AWS::RDS::DBCluster'
Properties:
MasterUsername: AuserName
MasterUserPassword: Apassword
DBSubnetGroupName:
Ref: RDSClusterGroup
AvailabilityZones:
- eu-central-1a
- eu-central-1b
Engine: aurora-postgresql
EngineVersion: 11.9
EngineMode: provisioned
EnableHttpEndpoint: true
DatabaseName: initialbase
DBClusterParameterGroupName:
Ref: RDSDBParameterGroup
AssociatedRoles:
- RoleArn:
{ Fn::GetAtt: [ AuroraPolicy, Arn ] }
VpcSecurityGroupIds:
- Ref: RdsSecurityGroup
AuroraPolicy:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- rds.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: AuroraRolePolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- s3:AbortMultipartUpload
- s3:GetBucketLocation
- s3:GetObject
- s3:ListBucket
- s3:ListBucketMultipartUploads
- s3:PutObject
Resource:
- { Fn::GetAtt: [ S3BucketEgresbucket, Arn ] }
- Fn::Join:
- ""
- - { Fn::GetAtt: [ S3BucketEgresbucket, Arn ] }
- "/*"
This should grant the DB permission to execute query's using SELECT aws_commons.create_s3_ur
However when I try and deploy I get the error message:
The feature-name parameter must be provided with the current operation for the Aurora (PostgreSQL) engine.
The issue comes from the AssociatedRoles object, cloudformation states that the FeatureName field is not needed however if you are wishing for your cluster to access other AWS services it is required. In this case as I was wanting to have my cluster access an s3 bucket I had to change my AssociatedRoles object so it looked like this:
AssociatedRoles:
- RoleArn: { Fn::GetAtt: [ roleServiceIntegration, Arn ] }
FeatureName: s3Import

Cloudwatch Event Rule doesn't invoke when source state machine fails

I have a step function that runs 2 separate lambdas. If the step function fails or times out, I want to get an email via SNS telling me the step function failed. I created the event rule using cloudformation and specified the statemachine ARN in the event pattern. When the step function fails, no email is sent out. If I remove the stateMachineArn parameter and run my step function, I get the failure email. I've double checked numerous times that I'm entering the correct ARN for the state machine. CF for the Event Rule is below (in YAML format). Thanks.
FailureEvent:
Type: AWS::Events::Rule
DependsOn:
- StateMachine
Properties:
Name: !Ref FailureRuleName
Description: "EventRule"
EventPattern:
detail-type:
- "Step Functions Execution Status Change"
detail:
status:
- "FAILED"
- "TIMED_OUT"
stateMachineArn: ["arn:aws:states:region:account#:stateMachine:statemachine"]
Targets:
-
Arn:
Ref: SNSARN
Id: !Ref SNSTopic
I did get this fixed and expanded on it to invoke a lambda that publishes a custom SNS email using a lambda. My alignment was off in my EventPattern section. See below. Thanks to #Marcin.
FailureEvent:
Type: AWS::Events::Rule
DependsOn:
- FMIStateMachine
Properties:
Description: !Ref FailureRuleDescription
Name: !Ref FailureRuleName
State: "ENABLED"
RoleArn:
'Fn::Join': ["", ['arn:aws:iam::', !Ref 'AWS::AccountId', ':role/', !Ref LambdaExecutionRole]]
EventPattern:
detail-type:
- "Step Functions Execution Status Change"
detail:
status:
- "FAILED"
- "TIMED_OUT"
stateMachineArn: [!Ref StateMachine]
Targets:
- Arn:
'Fn::Join': ["", ['arn:aws:lambda:', !Ref 'AWS::Region', ':', !Ref 'AWS::AccountId', ':function:', !Ref FailureLambda]]
Id: !Ref FailureLambda
Input: !Sub '{"failed_service": "${StateMachineName}","sns_arn": "arn:aws:sns:${AWS::Region}:${AWS::AccountId}:${SNSTopic}"}'

Can I customize partitioning in Kinesis Firehose before delivering to S3?

I have a Firehose stream that is intended to ingest millions of events from different sources and of different event-types. The stream should deliver all data to one S3 bucket as a store of raw\unaltered data.
I was thinking of partitioning this data in S3 based on metadata embedded within the event message like event-souce, event-type and event-date.
However, Firehose follows its default partitioning based on record arrival time. Is it possible to customize this partitioning behavior to fit my needs?
Update: Accepted answer updated as a new answer suggests the feature is available as of Sep 2021
As of writing this, the dynamic partitioning feature Vlad has mentioned is still pretty new. I needed it to be a part of CloudFormation template, which was still not properly documented. I had to add in DynamicPartitioningConfiguration to get it working properly. MetadataExtractionQuery syntax was also not properly documented.
MyKinesisFirehoseStream:
Type: AWS::KinesisFirehose::DeliveryStream
...
Properties:
ExtendedS3DestinationConfiguration:
Prefix: "clients/client_id=!{client_id}/dt=!{timestamp:yyyy-MM-dd}/"
ErrorOutputPrefix: "errors/!{firehose:error-output-type}/"
DynamicPartitioningConfiguration:
Enabled: "true"
RetryOptions:
DurationInSeconds: "300"
ProcessingConfiguration:
Enabled: "true"
Processors:
- Type: AppendDelimiterToRecord
- Type: MetadataExtraction
Parameters:
- ParameterName: MetadataExtractionQuery
ParameterValue: "{client_id:.client_id}"
- ParameterName: JsonParsingEngine
ParameterValue: JQ-1.6
Since September 1st, 2021, AWS Kinesis Firehose supports this feature. Read the announcement blog post here.
From the documentation:
You can use the Key and Value fields to specify the data record parameters to be used as dynamic partitioning keys and jq queries to generate dynamic partitioning key values. ...
Here is how it looks like from UI:
No. You cannot 'partition' based upon event content.
Some options are:
Send to separate Firehose streams
Send to a Kinesis Data Stream (instead of Firehose) and write your own custom Lambda function to process and save the data (See: AWS Developer Forums: Athena and Kinesis Firehose)
Use Kinesis Analytics to process the message and 'direct' it to different Firehose streams
If you are going to use the output with Amazon Athena or Amazon EMR, you could also consider converting it into Parquet format, which has much better performance. This would require post-processing of the data in S3 as a batch rather than converting the data as it arrives in a stream.
To build on John's answer, if you don't have the near real-time streaming requirements, we've found batch-processing with Athena to be a simple solution for us.
Kinesis streams to a given table unpartitioned_event_data, which can make use of the native record arrival time partitioning.
We define another Athena table partitioned_event_table which can be defined with custom partition keys and make use of the INSERT INTO capabilities that Athena has. Athena will automatically repartition your data in the format you want without requiring any custom consumers or new infrastructure to manage. This can be scheduled with a cron, SNS, or something like Airflow.
What's cool is you can create a view that does a UNION of the two tables to query historical and real-time data in one place.
We actually dealt with this problem at Radar and talk about more trade-offs in this blog post.
To expand on Murali's answer, we have implemented it in CDK:
Our incomming json data looks something like this:
{
"data":
{
"timestamp":1633521266990,
"defaultTopic":"Topic",
"data":
{
"OUT1":"Inactive",
"Current_mA":3.92
}
}
}
The CDK code looks as following:
const DeliveryStream = new CfnDeliveryStream(this, 'deliverystream', {
deliveryStreamName: 'deliverystream',
extendedS3DestinationConfiguration: {
cloudWatchLoggingOptions: {
enabled: true,
},
bucketArn: Bucket.bucketArn,
roleArn: deliveryStreamRole.roleArn,
prefix: 'defaultTopic=!{partitionKeyFromQuery:defaultTopic}/!{timestamp:yyyy/MM/dd}/',
errorOutputPrefix: 'error/!{firehose:error-output-type}/',
bufferingHints: {
intervalInSeconds: 60,
},
dynamicPartitioningConfiguration: {
enabled: true,
},
processingConfiguration: {
enabled: true,
processors: [
{
type: 'MetadataExtraction',
parameters: [
{
parameterName: 'MetadataExtractionQuery',
parameterValue: '{Topic: .data.defaultTopic}',
},
{
parameterName: 'JsonParsingEngine',
parameterValue: 'JQ-1.6',
},
],
},
{
type: 'AppendDelimiterToRecord',
parameters: [
{
parameterName: 'Delimiter',
parameterValue: '\\n',
},
],
},
],
},
},
})
My scenario is:
Firehose needs to send data to s3, which is tied to glue table, parquet as format, and dynamic partitioning enabled since I want to consider the year, month, and day from the data I push to firehose instead of the default.
Below is the working code
rawdataFirehose:
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
DeliveryStreamName: !Join ["-", [rawdata, !Ref AWS::StackName]]
DeliveryStreamType: DirectPut
ExtendedS3DestinationConfiguration:
BucketARN: !GetAtt rawdataS3bucket.Arn
Prefix: parquetdata/year=!{partitionKeyFromQuery:year}/month=!{partitionKeyFromQuery:month}/day=!{partitionKeyFromQuery:day}/
BufferingHints:
IntervalInSeconds: 300
SizeInMBs: 128
ErrorOutputPrefix: errors/
RoleARN: !GetAtt FirehoseRole.Arn
DynamicPartitioningConfiguration:
Enabled: true
ProcessingConfiguration:
Enabled: true
Processors:
- Type: MetadataExtraction
Parameters:
- ParameterName: MetadataExtractionQuery
ParameterValue: "{year:.year,month:.month,day:.day}"
- ParameterName: "JsonParsingEngine"
ParameterValue: "JQ-1.6"
DataFormatConversionConfiguration:
Enabled: true
InputFormatConfiguration:
Deserializer:
HiveJsonSerDe: {}
OutputFormatConfiguration:
Serializer:
ParquetSerDe: {}
SchemaConfiguration:
CatalogId: !Ref AWS::AccountId
RoleARN: !GetAtt FirehoseRole.Arn
DatabaseName: !Ref rawDataDB
TableName: !Ref rawDataTable
Region:
Fn::ImportValue: AWSRegion
VersionId: LATEST
FirehoseRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: firehose.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: !Sub firehose-glue-${Envname}
PolicyDocument: |
{
"Version": "2012-10-17",
"Statement":
[
{
"Effect": "Allow",
"Action":
[
"glue:*",
"iam:ListRolePolicies",
"iam:GetRole",
"iam:GetRolePolicy",
"tag:GetResources",
"s3:*",
"cloudwatch:*",
"ssm:*"
],
"Resource": "*"
}
]
}
Note:
rawDataDB is a reference to glue database
rawDataTable is a reference to table
rawdataS3bucket is a reference to s3 bucket

Unable to create S3Buckets through serverless.yml

I simply tried to add a new S3Bucket into the Resources section and the stack is not being built anymore:
resources:
Resources:
myBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: prefix-${self:custom.env.myvar}-myBucket
and the error I'm getting is not helping too much:
Template format error: Unresolved resource dependencies
[] in the Resources block of the template (nothing between the [] that could indicate what to look for)
Any idea what's going on?
I'm running serverless v1.5.0
serverless.yml
service: myService
frameworkVersion: "=1.5.0"
custom:
env: ${file(./.variables.yml)}
provider:
name: aws
runtime: nodejs4.3
stage: ${opt:stage, self:custom.env.stage}
region: ${self:custom.env.region}
profile: myProfile-${opt:stage, self:custom.env.stage}
memorySize: 128
iamRoleStatements:
- Effect: "Allow"
Action:
- "lambda:InvokeFunction"
Resource: "*"
- Effect: "Allow"
Action:
- "s3:ListBucket"
Resource: { "Fn::Join" : ["", ["arn:aws:s3:::", { "Ref" : "ServerlessDeploymentBucket" } ] ] }
- Effect: "Allow"
Action:
- "s3:PutObject"
Resource:
Fn::Join:
- ""
- - "arn:aws:s3:::"
- "Ref" : "ServerlessDeploymentBucket"
- "Ref" : ""
functions:
myFunction:
handler: functions/myFunction.handler
name: ${opt:stage, self:custom.env.stage}-myFunction
resources:
Resources:
myBucket:
Type: AWS::S3::Bucket
Properties:
BucketName: myService-${opt:stage, self:custom.env.myVar}-myBucket
The Reference to an empty-string in your iamRoleStatements section, - "Ref" : "", is likely causing the Unresolved resource dependencies [] error. Remove this line from your template, since it seems to be unnecessary.

AWS CloudFormation environmental conditional for ses role

I'm trying to make a reusable CloudFormation template and would like to do some kind of conditional where if the Environment parameter is "test" (or any other environment other than "prod"), then send SES emails to only gmail accounts (i.e., corporate accounts), but for "prod", send SES emails anywhere. Would I have to do two different roles and have conditions on each one? Or is there a way to do this inside of just the one role below? Thanks for any help!
Parameters:
Environment:
Description: Environment, which can be "test", "stage", "prod", etc.
Type: String
Resources:
Role:
Type: AWS::IAM::Role
Properties:
RoleName: myRole
Path: /
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "ecs.amazonaws.com"
Action:
- "sts:AssumeRole"
Policies:
-
PolicyName: "ses-policy"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "ses:SendEmail"
- "ses:SendRawEmail"
Resource: "*"
Condition:
"ForAllValues:StringLike":
"ses:Recipients":
- "*#gmail.com"
Conditions are perfectly suited for adding this sort of conditional logic to CloudFormation Resource Properties. In your example, you could use the Fn::If Intrinsic Function to include the existing Policy Condition (not to be confused with the CloudFormation Condition!) if the environment is not prod, and AWS::NoValue otherwise (removing the Policy Condition entirely when environment is prod):
Parameters:
Environment:
Description: Environment, which can be "test", "stage", "prod", etc.
Type: String
AllowedValues: [test, stage, prod]
Conditions:
IsProdEnvironment: !Equals [ !Ref Environment, prod ]
Resources:
Role:
Type: AWS::IAM::Role
Properties:
RoleName: myRole
Path: /
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "ecs.amazonaws.com"
Action:
- "sts:AssumeRole"
Policies:
-
PolicyName: "ses-policy"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- "ses:SendEmail"
- "ses:SendRawEmail"
Resource: "*"
Condition: !If
- IsProdEnvironment
- !Ref AWS::NoValue
- "ForAllValues:StringLike":
"ses:Recipients":
- "*#gmail.com"