Filebeat is not creating my custom index? - filebeat

I hope everyone is doing well.
I am using filebeat to ship cisco syslog (with using filebeat cisco module) to elasticsearch. And apparently it is not using my custom index, instead logs go to default index filebeat-*.
Note: the field host.name is a custom field.
-also tried disabling and enabling ILM but no luck.
Below is my filebeat.yml config:
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: filestream
id: my-filestream-id
enabled: false
paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
- type: syslog
enabled: false
paths:
- /var/log/*.log
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# ======================= Elasticsearch template setting =======================
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 1
# =================================== Kibana ===================================
setup.kibana:
host: "http://172.30.169.50:5000"
ssl.verification_mode: "none"
username: "elastic"
password: "m4gN3s1#.123"
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["172.30.169.50:9200"]
ssl.verification_mode: "none"
username: "elastic"
password: "secret"
indices:
- index: "sbx-%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "SBX"
- index: "core-%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "-CS0"
- index: "access%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "-AS0"
- index: "bgp-%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "RTRBGP"
- index: "iplc-%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "IPLC0"
- index: "asp-%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "ASP0"
- index: "sfs-%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "SFS0"
- index: "Local_Loop-%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "LL0"
- index: "mpls-%{+yyyy.MM.dd}"
when.contains:
fields:
host.name: "MPLS"
- index: "fortinet-%{+yyyy.MM.dd}"
when.contains:
fields:
event.module: "fortinet"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
- drop_fields:
when:
equals:
event.module: "cisco"
fields: ["agent.ephemeral_id","agent.hostname","agent.id","agent.type","agent.version","agent.name","cisco.ios.facility","ecs.version","event.cod>
- drop_fields:
when:
equals:
event.module: "fortinet"
fields: ["agent.ephemeral_id","agent.hostname","agent.id","agent.type","agent.version","agent.name","ecs.version","event.code","event.dataset","e>
- dissect:
when:
equals:
event.module: "cisco"
tokenizer: "%{address}:%{}"
field: "log.source.address"
target_prefix: "host"
- dissect:
when:
equals:
event.module: "cisco"
tokenizer: "%{} %{name} %{}"
field: "log.original"
target_prefix: "host"

Related

Serverless: TypeError: Cannot read property 'stage' of undefined

frameworkVersion: '2'
plugins:
- serverless-step-functions
- serverless-python-requirements
- serverless-parameters
- serverless-pseudo-parameters
provider:
name: aws
region: us-east-2
stage: ${opt:stage, 'dev'}
runtime: python3.7
versionFunctions: false
iam:
role: arn:aws:iam::#{AWS::AccountId}:role/AWSLambdaVPCAccessExecutionRole
apiGateway:
shouldStartNameWithService: true
lambdaHashingVersion: 20201221
package:
exclude:
- node_modules/**
- venv/**
# Lambda functions
functions:
generateAlert:
handler: handler.generateAlert
generateData:
handler: handler.generateDataHandler
timeout: 600
approveDenied:
handler: handler.approveDenied
timeout: 600
stepFunctions:
stateMachines:
"claims-etl-and-insight-generation-${self:provider.stage}":
loggingConfig:
level: ALL
includeExecutionData: true
destinations:
- Fn::GetAtt: ["ETLStepFunctionLogGroup", Arn]
name: "claims-etl-and-insight-generation-${self:provider.stage}"
definition:
Comment: "${self:provider.stage} ETL Workflow"
StartAt: RawQualityJob
States:
# Raw Data Quality Check Job Start
RawQualityJob:
Type: Task
Resource: arn:aws:states:::glue:startJobRun.sync
Parameters:
JobName: "data_quality_v2_${self:provider.stage}"
Arguments:
"--workflow-name": "${self:provider.stage}-Workflow"
"--dataset_id.$": "$.datasetId"
"--client_id.$": "$.clientId"
Next: DataQualityChoice
Retry:
- ErrorEquals: [States.ALL]
MaxAttempts: 2
IntervalSeconds: 10
BackoffRate: 5
Catch:
- ErrorEquals: [States.ALL]
Next: GenerateErrorAlertDataQuality
# End Raw Data Quality Check Job
DataQualityChoice:
Type: Task
Resource:
Fn::GetAtt: [approveDenied, Arn]
Next: Is Approved ?
Is Approved ?:
Type: Choice
Choices:
- Variable: "$.quality_status"
StringEquals: "Denied"
Next: FailState
Default: HeaderLineJob
FailState:
Type: Fail
Cause: "Denied status"
# Header Line Job Start
HeaderLineJob:
Type: Parallel
Branches:
- StartAt: HeaderLineIngestion
States:
HeaderLineIngestion:
Type: Task
Resource: arn:aws:states:::glue:startJobRun.sync
Parameters:
JobName: headers_lines_etl_rs_v2
Arguments:
"--workflow-name.$": "$.Arguments.--workflow-name"
"--dataset_id.$": "$.Arguments.--dataset_id"
"--client_id.$": "$.Arguments.--client_id"
End: True
Retry:
- ErrorEquals: [States.ALL]
MaxAttempts: 2
IntervalSeconds: 10
BackoffRate: 5
Catch:
- ErrorEquals: [States.ALL]
Next: GenerateErrorAlertHeaderLine
End: True
# Header Line Job End
GenerateErrorAlertDataQuality:
Type: Task
Resource:
Fn::GetAtt: [generateAlert, Arn]
End: true
resources:
Resources:
# Cloudwatch Log
"ETLStepFunctionLogGroup":
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: "ETLStepFunctionLogGroup_${self:provider.stage}"
This is what my serverless.yml file looks like.
When I run the command:
sls deploy --stage staging
It show
Type Error ----------------------------------------------
TypeError: Cannot read property 'stage' of undefined
at Variables.getValueFromOptions (/snapshot/serverless/lib/classes/Variables.js:648:37)
at Variables.getValueFromSource (/snapshot/serverless/lib/classes/Variables.js:579:17)
at /snapshot/serverless/lib/classes/Variables.js:539:12
Your Environment Information ---------------------------
Operating System: linux
Node Version: 14.4.0
Framework Version: 2.30.3 (standalone)
Plugin Version: 4.5.1
SDK Version: 4.2.0
Components Version: 3.7.4
How I can fix this? I tried with different version of serverless.
There is error in yamlParser file, which is provided by serverless-step-functions.
Above is my serverless config file.
It looks like a $ sign is missing from your provider -> stage?
provider:
name: aws
region: us-east-2
stage: ${opt:stage, 'dev'} # $ sign is missing?
runtime: python3.7
versionFunctions: false
iam:
role: arn:aws:iam::#{AWS::AccountId}:role/AWSLambdaVPCAccessExecutionRole
apiGateway:
shouldStartNameWithService: true
lambdaHashingVersion: 20201221

EventRule matched by default EventBridge EventBus ignored by custom EventBus

I have a Cloudformation template which implements a CI/CD process for Lambda functions (at bottom)
Essentially it -
watches a Github repo
pulls source code on new git tag creation
starts a CodeBuild process which runs unit tests, zips source code (on test success) and pushes archive to S3
enables CodeBuild notifications
implements a CloudWatch EventRule to pattern match raw CodeBuild notifications, format and push them to SNS
binds a Lambda function to SNS, which pushes the notifications to Slack via a webhook
This works fine with the default EventBridge EventBus, but the pattern matching seems to fail if I switch to a custom EventBus.
(see resources EventBus and EventRule in the stack; it's currently set up to use the custom EventBus, and fails / ignores new git tags; if you comment out the reference to property EventBusName in EventBus, it defaults to using the default EventBus and works)
Why would a custom EventBus behave differently to the default EventBus in this situation ?
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
AppName:
Type: String
RepoOwner:
Type: String
RepoName:
Type: String
RepoBranch:
Type: String
Default: master
RepoAuth:
Type: String
Default: master
WebhookUrl:
Type: String
WebhookLambda:
Type: String
CodeBuildBuildSpec:
Type: String
CodeBuildType:
Type: String
Default: LINUX_CONTAINER
CodeBuildComputeType:
Type: String
Default: BUILD_GENERAL1_SMALL
CodeBuildImage:
Type: String
Default: aws/codebuild/standard:4.0
LambdaHandler:
Type: String
Default: "index.handler"
LambdaMemory:
Type: Number
Default: 128
LambdaTimeout:
Type: Number
Default: 30
LambdaRuntime:
Type: String
Default: python3.8
Resources:
ArtifactsBucket:
Type: AWS::S3::Bucket
Properties:
BucketName:
Fn::Sub:
- ${app_name}-lambda-artifacts
- app_name:
Ref: AppName
CodeBuildProject:
Properties:
Environment:
ComputeType:
Ref: CodeBuildComputeType
Image:
Ref: CodeBuildImage
Type:
Ref: CodeBuildType
Name:
Fn::Sub:
- ${app_name}-lambda-ci
- app_name:
Ref: AppName
ServiceRole:
Fn::GetAtt:
- CodeBuildRole
- Arn
Source:
Auth:
Resource:
Ref: RepoAuth
Type: OAUTH
Location:
Fn::Sub:
- "https://github.com/${repo_owner}/${repo_name}.git"
- repo_owner:
Ref: RepoOwner
repo_name:
Ref: RepoName
Type: GITHUB
BuildSpec:
Fn::Sub:
- "${build_spec}"
- build_spec:
Ref: CodeBuildBuildSpec
Artifacts:
Type: NO_ARTIFACTS
SourceVersion:
Ref: RepoBranch
Triggers:
Webhook: true
FilterGroups:
- - Type: EVENT
Pattern: PUSH
ExcludeMatchedPattern: false
- Type: HEAD_REF
Pattern: "refs/tags/.*"
ExcludeMatchedPattern: false
Type: AWS::CodeBuild::Project
CodeBuildRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- codebuild:*
- events:*
- s3:PutObject
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName: code-build-role-policy
Type: AWS::IAM::Role
WebhookFunction:
Properties:
FunctionName:
Fn::Sub:
- ${app_name}-lambda-webhook
- app_name:
Ref: AppName
Code:
ZipFile:
Ref: WebhookLambda
Environment:
Variables:
WEBHOOK_URL:
Ref: WebhookUrl
Handler:
Ref: LambdaHandler
MemorySize:
Ref: LambdaMemory
Role:
Fn::GetAtt:
- WebhookFunctionRole
- Arn
Runtime:
Ref: LambdaRuntime
Timeout:
Ref: LambdaTimeout
Type: AWS::Lambda::Function
WebhookFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName: webhook-role-policy
Type: AWS::IAM::Role
WebhookFunctionPermission:
Properties:
Action: "lambda:InvokeFunction"
FunctionName:
Ref: WebhookFunction
Principal: "sns.amazonaws.com"
SourceArn:
Ref: WebhookTopic
Type: AWS::Lambda::Permission
WebhookTopic:
Properties:
Subscription:
- Protocol: lambda
Endpoint:
Fn::GetAtt:
- WebhookFunction
- Arn
Type: AWS::SNS::Topic
WebhookTopicPolicy:
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: "events.amazonaws.com"
Action:
- "sns:Publish"
Resource:
Ref: WebhookTopic
Topics:
- Ref: WebhookTopic
Type: AWS::SNS::TopicPolicy
EventBus:
Type: AWS::Events::EventBus
Properties:
Name:
Fn::Sub:
- ${app_name}-bus
- app_name:
Ref: AppName
EventRule:
Type: AWS::Events::Rule
Properties:
EventBusName: # CURRENTLY USING CUSTOM EVENT BUS (PATTERN MATCHING FAILS); REMOVE THIS PROPERTY TO SWITCH TO DEFAULT EVENT BUS (PATTERN MATCHING WORKS)
Ref: EventBus
EventPattern:
source:
- "aws.codebuild"
detail-type:
- "CodeBuild Build Phase Change"
detail:
completed-phase:
- SUBMITTED
- PROVISIONING
- DOWNLOAD_SOURCE
- INSTALL
- PRE_BUILD
- BUILD
- POST_BUILD
- UPLOAD_ARTIFACTS
- FINALIZING
completed-phase-status:
- TIMED_OUT
- STOPPED
- FAILED
- SUCCEEDED
- FAULT
- CLIENT_ERROR
project-name:
- Ref: CodeBuildProject
State: ENABLED
Targets:
- Arn:
Ref: WebhookTopic
Id:
Fn::Sub:
- "${project_name}-codebuild-notifications"
- project_name:
Ref: CodeBuildProject
InputTransformer:
InputPathsMap:
build-id: "$.detail.build-id"
project-name: "$.detail.project-name"
completed-phase: "$.detail.completed-phase"
completed-phase-status: "$.detail.completed-phase-status"
InputTemplate: |
"{'build-id': '<build-id>', 'project-name': '<project-name>', 'completed-phase': '<completed-phase>', 'completed-phase-status': '<completed-phase-status>'}"
"Custom event buses serve a use case of receiving events from your custom applications and services. Unfortunately, it is not possible for AWS services to push events to a custom event bus." (AWS Support)

SNS topic not receiving CodeBuild notifications

Trying to get CodeBuild to push notifications to an SNS topic (bound to a Lambda), via a CloudWatch Events Rule.
Cloudformation template (see below) deploys fine.
CodeBuild process works fine (have tested).
SNS topic and bound Lambda work fine - I can push a message to the topic via AWS CLI and see Lambda dump that message to Cloudwatch logs.
The Cloudwatch Event Rule seems to be configured fine - I can see it in the cosole, it looks well formed, seems to be bound to SNS topic.
In addition I have been careful to give the Event Rule a role with permissions to sns:Publish, and also defined an AWS::SNS::TopicPolicy for the SNS topic -
Unable to successfully set up SNS on CodeBuild project through CFT but works manually
But still nothing - CodeBuild successfully completes but I don't receive any notifications.
Any thoughts as to what might be wrong ?
TIA :)
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
AppName:
Type: String
StagingBucket:
Type: String
RepoOwner:
Type: String
RepoName:
Type: String
RepoBranch:
Type: String
RepoPAT:
Type: String
CodeBuildBuildSpec:
Type: String
CodeBuildType:
Type: String
Default: LINUX_CONTAINER
CodeBuildComputeType:
Type: String
Default: BUILD_GENERAL1_SMALL
CodeBuildImage:
Type: String
Default: aws/codebuild/standard:4.0
LambdaHandler:
Type: String
Default: "index.handler"
LambdaMemory:
Type: Number
Default: 128
LambdaTimeout:
Type: Number
Default: 30
LambdaRuntime:
Type: String
Default: python3.8
Resources:
CodeBuildProject:
DependsOn:
- CodeBuildSourceCredential
Properties:
Environment:
ComputeType:
Ref: CodeBuildComputeType
Image:
Ref: CodeBuildImage
Type:
Ref: CodeBuildType
Name:
Ref: AppName
ServiceRole:
Fn::GetAtt:
- CodeBuildRole
- Arn
Source:
Location:
Fn::Sub:
- "https://github.com/${repo_owner}/${repo_name}.git"
- repo_owner:
Ref: RepoOwner
repo_name:
Ref: RepoName
Type: GITHUB
BuildSpec:
Fn::Sub:
- "${build_spec}"
- build_spec:
Ref: CodeBuildBuildSpec
Artifacts:
Type: S3
Location:
Ref: StagingBucket
SourceVersion:
Ref: RepoBranch
Triggers:
Webhook: true
FilterGroups:
- - Type: EVENT
Pattern: PUSH
ExcludeMatchedPattern: false
- Type: HEAD_REF
Pattern: "refs/tags/.*"
ExcludeMatchedPattern: false
Type: AWS::CodeBuild::Project
CodeBuildSourceCredential:
Type: AWS::CodeBuild::SourceCredential
Properties:
Token:
Ref: RepoPAT
ServerType: GITHUB
AuthType: PERSONAL_ACCESS_TOKEN
CodeBuildRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Version: '2012-10-17'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
Path: /
Type: AWS::IAM::Role
CodeBuildNotificationFunction:
Properties:
Code:
ZipFile: "def handler(event, context):\n print (event)"
Handler:
Ref: LambdaHandler
MemorySize:
Ref: LambdaMemory
Role:
Fn::GetAtt:
- CodeBuildNotificationFunctionRole
- Arn
Runtime:
Ref: LambdaRuntime
Timeout:
Ref: LambdaTimeout
Type: AWS::Lambda::Function
CodeBuildNotificationFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName: code-build-notification-role-policy
Type: AWS::IAM::Role
CodeBuildNotificationTopic:
Properties:
Subscription:
- Protocol: lambda
Endpoint:
Fn::GetAtt:
- CodeBuildNotificationFunction
- Arn
Type: AWS::SNS::Topic
CodeBuildNotificationTopicPolicy:
Properties:
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: "events.amazonaws.com"
Action:
- "sns:Publish"
Resource:
Ref: CodeBuildNotificationTopic
Topics:
- Ref: CodeBuildNotificationTopic
Type: AWS::SNS::TopicPolicy
CodeBuildNotificationLambdaInvokePermission:
Properties:
Action: "lambda:InvokeFunction"
FunctionName:
Ref: CodeBuildNotificationFunction
Principal: "sns.amazonaws.com"
SourceArn:
Ref: CodeBuildNotificationTopic
Type: AWS::Lambda::Permission
SampleNotificationRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
Fn::Sub:
- '{"source": ["aws.codebuild"], "detail-type": ["Codebuild Build Phase Change"], "detail": {"completed-phase": ["SUBMITTED", "PROVISIONING", "DOWNLOAD_SOURCE", "INSTALL", "PRE_BUILD", "BUILD", "POST_BUILD", "UPLOAD_ARTIFACTS", "FINALIZING"], "completed-phase-status": ["TIMED_OUT", "STOPPED", "FAILED", "SUCCEEDED", "FAULT", "CLIENT_ERROR"], "project-name": ["${project_name}"]}}'
- project_name:
Ref: CodeBuildProject
State: ENABLED
RoleArn:
Fn::GetAtt:
- SampleNotificationRuleRole
- Arn
Targets:
- Arn:
Ref: CodeBuildNotificationTopic
Id: sample-notification
SampleNotificationRuleRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: events.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- "sns:Publish"
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName: sample-notification-rule-role-policy
Type: AWS::IAM::Role
Fixed - AWS::Events::Rule target was missing an InputTransformer block like so
Targets:
- Arn:
Ref: CodeBuildNotificationTopic
Id: sample-notification
InputTransformer:
InputPathsMap:
build-id: "$.detail.build-id"
project-name: "$.detail.project-name"
completed-phase: "$.detail.completed-phase"
completed-phase-status: "$.detail.completed-phase-status"
InputTemplate: |
"Build '<build-id>' for build project '<project-name>' has completed the build phase of '<completed-phase>' with a status of '<completed-phase-status>'."

Cloud Formation to add multiple S3 triggers to same S3 bucket in LambdaConfigurations

My requirement is to trigger Lambda_Function_1 if input.txt file creates in S3 bucket and trigger Lambda_Function_2 if output.txt file creates in same S3 bucket.
The below cfn is not working, but it works fine if I put only one event instead of two events in same LambdaConfigurations.
Can some one please help me here?
Parameters:
S3BucketBaseName:
Type: String
Description: The base name of the Amazon S3 bucket.
Default: dw-trip
Resources:
LambdaStart:
DependsOn:
- LambdaStartStopEC2
Type: "AWS::Lambda::Function"
Properties:
FunctionName: "dw-trip-start-ec2"
Handler: "index.handler"
Role: !GetAtt LambdaStartStopEC2.Arn
Runtime: python3.7
MemorySize: 3008
Timeout: 900
Code:
ZipFile: |
import boto3
region = 'us-east-1'
instances = ['i-05d5fbec4c82956b6']
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
ec2.start_instances(InstanceIds=instances)
print('started your instances: ' + str(instances))
ProcessingLambdaPermissionStart:
Type: AWS::Lambda::Permission
DependsOn:
- LambdaStart
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref LambdaStart
Principal: s3.amazonaws.com
SourceArn:
Fn::Join:
- ''
- - 'arn:aws:s3:::'
- !Join ["-",[!Ref "S3BucketBaseName",!Ref "AWS::AccountId"]]
SourceAccount: !Ref AWS::AccountId
LambdaStop:
DependsOn:
- ProcessingLambdaPermissionStart
Type: "AWS::Lambda::Function"
Properties:
FunctionName: "dw-trip-stop-ec2"
Handler: "index.handler"
Role: !GetAtt LambdaStartStopEC2.Arn
Runtime: python3.7
MemorySize: 3008
Timeout: 900
Code:
ZipFile: |
import boto3
region = 'us-east-1'
instances = ['i-05d5fbec4c82956b6']
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
ec2.stop_instances(InstanceIds=instances)
print('stopping your instances: ' + str(instances))
ProcessingLambdaPermissionStop:
Type: AWS::Lambda::Permission
DependsOn:
- LambdaStop
Properties:
Action: 'lambda:InvokeFunction'
FunctionName: !Ref LambdaStop
Principal: s3.amazonaws.com
SourceArn:
Fn::Join:
- ''
- - 'arn:aws:s3:::'
- !Join ["-",[!Ref "S3BucketBaseName",!Ref "AWS::AccountId"]]
SourceAccount: !Ref AWS::AccountId
S3KmsKey:
Type: AWS::KMS::Key
DependsOn:
- ProcessingLambdaPermissionStop
Properties:
Description: KMS key for trip S3 bucket.
Enabled: true
EnableKeyRotation: true
KeyPolicy:
Statement:
- Sid: Administration
Effect: Allow
Principal:
AWS:
- Fn::Join:
- ''
- - 'arn:aws:iam::'
- Ref: AWS::AccountId
- ':role/DW01-codepipeline-action-us-east-1'
- Fn::Join:
- ''
- - 'arn:aws:iam::'
- Ref: AWS::AccountId
- ':root'
Action: 'kms:*'
Resource: '*'
S3bucketCreate:
DependsOn:
- S3KmsKey
Type: AWS::S3::Bucket
Properties:
BucketName: !Join ["-",[!Ref "S3BucketBaseName",!Ref "AWS::AccountId"]]
BucketEncryption:
ServerSideEncryptionConfiguration:
- ServerSideEncryptionByDefault:
KMSMasterKeyID: !Ref S3KmsKey
SSEAlgorithm: "aws:kms"
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function: !GetAtt LambdaStart.Arn
Filter:
S3Key:
Rules:
- Name: prefix
Value: input.txt
- Event: s3:ObjectCreated:*
Function: !GetAtt LambdaStop.Arn
Filter:
S3Key:
Rules:
- Name: prefix
Value: output.txt
S3bucketPolicy:
DependsOn:
- S3bucketCreate
Type: AWS::S3::BucketPolicy
Properties:
Bucket:
Ref: 'S3bucketCreate'
PolicyDocument:
Statement:
- Sid: AllowEc2AccesstoBucket
Action:
- 's3:GetObject'
- 's3:PutObject'
Effect: Allow
Principal:
AWS:
- Fn::Join:
- ''
- - 'arn:aws:iam::'
- Ref: AWS::AccountId
- ':role/DevDW01-EC2-us-east-1'
Resource:
Fn::Join:
- ''
- - 'arn:aws:s3:::'
- Ref: 'S3bucketCreate'
- '/*'
LambdaStartStopEC2:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
RoleName: Lambda-StartStop-EC2
MaxSessionDuration: 43200
Policies:
- PolicyName: StartStop-EC2
PolicyDocument:
Statement:
- Action:
- s3:*
Effect: Allow
Resource: '*'
- Action:
- ec2:*
Effect: Allow
Resource: '*'
- PolicyName: logs
PolicyDocument:
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:DescribeLogGroups
- logs:DescribeLogStreams
- logs:PutLogEvents
- logs:GetLogEvents
- logs:FilterLogEvents
Effect: Allow
Resource: '*'
Outputs:
S3bucketCreateName:
Value:
Ref: S3bucketCreate
Export:
Name: S3bucketCreateName
S3bucketCreateArn:
Value:
Fn::GetAtt: S3bucketCreate.Arn
Export:
Name: S3bucketCreateArn
S3KmsKeyArn:
Value:
Fn::GetAtt: S3KmsKey.Arn
Export:
Name: S3KmsKeyArn
Multiple filter rules with prefix and suffix as name are allowed as long as they do not overlap. Refer here for various examples explaining how overlapping may occur and how to avoid them.
In this case, the error Template format error: YAML not well-formed is possibly due to improper YAML formatting. Use cfn-lint to validate the templates.
Adding a snippet that explicitly specifies the expected prefix and suffix of the S3 object.
NotificationConfiguration:
LambdaConfigurations:
- Event: s3:ObjectCreated:*
Function: !GetAtt LambdaStart.Arn
Filter:
S3Key:
Rules:
- Name: prefix
Value: input
- Name: suffix
Value: txt
- Event: s3:ObjectCreated:*
Function: !GetAtt LambdaStop.Arn
Filter:
S3Key:
Rules:
- Name: prefix
Value: output
- Name: suffix
Value: txt
I actually had to do it like this create multiple LambdaConfigurations.
"NotificationConfiguration": {
"LambdaConfigurations": [{
"Event": "s3:ObjectCreated:*",
"Function": {
"Fn::GetAtt": ["lambdaVodFunction", "Arn"]
},
"Filter": {
"S3Key": {
"Rules": [{
"Name": "suffix",
"Value": ".mp4"
}]
}
}
},
{
"Event": "s3:ObjectCreated:*",
"Function": {
"Fn::GetAtt": ["lambdaVodFunction", "Arn"]
},
"Filter": {
"S3Key": {
"Rules": [{
"Name": "suffix",
"Value": ".mov"
}]
}
}
}
]
}

filebeat add_fields processor with condition

I'd like to add a field "app" with the value "apache-access" to every line that is exported to Graylog by the Filebeat "apache" module.
The following configuration should add the field as I see a "event_dataset"="apache.access" field in Graylog but to does not do anything.
If I remove the condition, the "add_fields" processor does add a field though.
filebeat.inputs:
- type: log
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.logstash:
hosts: [ "localhost:5044" ]
processors:
- add_fields:
when:
equals:
event_dataset: "apache.access"
target: ""
fields:
app: "apache-access"
logging.level: info
For whatever reason the field is called "event.dataset" in filebeat but displayed as "event_dataset" in Graylog.