How to use input transformer for ECS Fargate launch type with Terraform CloudWatch event trigger - amazon-s3

Question

I'm using terraform to create a CloudWatch Event Trigger with a ECS Fargate launch type which the event source is S3. When I use the input_transformer field to pass in the bucket and key into the ECS task, my event rule results in a failed invocation.
This is the aws_cloudwatch_event_rule:
resource "aws_cloudwatch_event_rule" "event_rule" {
name = "dev-gnss-source-put-rule-tf"
description = "Capture S3 events on uploads bucket"
event_pattern = <<PATTERN
{
"source": [
"aws.s3"
],
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"s3.amazonaws.com"
],
"eventName": [
"PutObject"
],
"requestParameters": {
"bucketName": [
"example-bucket-name"
]
}
}
}
PATTERN
}
This is the aws_cloudwatch_event_target:
resource "aws_cloudwatch_event_target" "event_target" {
target_id = "dev-gnss-upload-event-target-tf"
arn = "example-cluster-arn"
rule = aws_cloudwatch_event_rule.event_rule.name
role_arn = aws_iam_role.uploads_events.arn
ecs_target {
launch_type = "FARGATE"
task_count = 1 # Launch one container / event
task_definition_arn = "example-task-definition-arn"
network_configuration {
subnets = ["example-subnet"]
security_groups = []
}
}
input_transformer {
input_paths = {
s3_bucket = "$.detail.requestParameters.bucketName"
s3_key = "$.detail.requestParameters.key"
}
input_template = <<TEMPLATE
{
"containerOverrides": [
{
"name": "myproject-task",
"environment": [
{ "name": "S3_BUCKET", "value": <s3_bucket> },
{ "name": "S3_KEY", "value": <s3_key> }
]
}
]
}
TEMPLATE
}
}
If I remove the input_transformer section, it will work fine, but I need to pass in the s3 bucket and key to process the particular file.
My rationale for doing this is to remove the need for an intermediary Lambda and was guided by this Medium post: https://medium.com/#bowbaq/trigger-an-ecs-job-when-an-s3-upload-completes-3559c44c37d1
Any advice is appreciated.

Answer 1

After hours of going in circles, I found an answer!
So the first step is to check what the cause of the failed invocation is. You can do this by checking CloudTrail logs by navigating to Cloud Trail > Event history > Search by Event name and type RunTask in the search box. You should see a series of events from the event source ecs.amazonaws.com. Find one that relates to your the Failed Invocation you experienced.
When you click into the event, you can see under the Event record section an errorMessage. In my case, it was the following:
"errorCode": "InvalidParameterException",
"errorMessage": "Override for container named myproject-task is not a container in the TaskDefinition.",
This may be different for you. For me, it was because my containerOverride name was incorrect. This field refers to: The name of the container that receives the override. This parameter is required if any override is specified. ref: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerOverride.html
Correcting this field fixed my issue.