I have a Python process writing the following example JSON log line:
{"levelname": "DEBUG", "asctime": "2020-02-04 08:37:42,128", "module": "scale_out", "thread": 139793342834496, "filename": "scale_out.py", "lineno": 130, "funcName": "_check_if_can_remove_inactive_components", "message": "inactive_components: set([]), num_of_components_active: 0, max num_of_components_to_keep: 1"}
In the filebeat.yml, I'm trying to exclude all DEBUG logs from being sent into Elasticsearch.
I've tried using the exclude_lines keyword, but Filebeat still publish these events.
I've also tried using a processor with drop event
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/my_service/*.log
json.keys_under_root: true
json.add_error_key: true
json.message_key: "module"
exclude_lines: ['DEBUG'] # also tried ['.*DEBUG.*']
keep_null: true
processors:
- drop_event:
when:
levelname: 'DEBUG'
Any ideas what am I may be doing wrong?
Well..
It was much more easier (and stupid) that I expected it to be.
While the exclude_lines doesn't work (still),
I was able to get the drop_event to work.
The problem was that the 'DEBUG' should had been written without quotes.
processors:
- drop_event:
when:
levelname: DEBUG
Related
I'm trying to setup some processors in a filebeat.yml to process some logs before sending to ELK.
An important part of the processing is determining the "level" of the event, which is not always included in the line in the log file.
This is the idea I have for it right now:
# /var/log/messages
- type: log
processors:
- dissect:
tokenizer: "%{month} %{day} %{time} %{hostname} %{service}: {%message}"
field: "message"
target_prefix: "dissect"
- if:
when:
regexp:
message: ((E|e)rror|(f|F)ault)
then:
- add_fields:
target: 'dissect'
fields:
level: error
else:
- if:
when:
regexp:
message: (W|W)arning
then:
- add_fields:
target: 'dissect'
fields:
level: warning
else:
- add_fields:
target: 'dissect'
fields:
level: information
- drop_fields:
#duplicate
fields: ["dissect.month","dissect.day","dissect.time","dissect.hostname","message"]
# Change to true to enable this input configuration.
enabled: true
paths:
- /var/log/messages
I'm still not sure about those patterns I'm trying... but right now I don't think they're what's causing me to fail.
When trying to run filebeat with console output for a test with
filebeat -e -c filebeat.yml
I get the following error:
2022-01-26T17:45:27.174+0200 ERROR instance/beat.go:877 Exiting: Error while initializing input: failed to make if/then/else processor: missing or invalid condition
Exiting: Error while initializing input: failed to make if/then/else processor: missing or invalid condition
I'm very new to yaml in general, and the only other beat I've done before is an AuditBeat (which works, and has conditions, but not "if"s).
Does anyone know what the problem might be?
To clarify: I commented out all other "input" entries, leaving just this one, and still got this error.
Edit: Version: 7.2.0
The if part of the if-then-else processor doesn't use the when label to introduce the condition. The correct usage is:
- if:
regexp:
message: [...]
You have to correct the two if processors in your configuration.
Additionally, there's a mistake in your dissect expression. {%message} should be %{message}. Also, the regexp for warning should be (W|w)arning not (W|W)arning (both W's are uppercase in your config).
This is the corrected processors configuration:
processors:
- dissect:
tokenizer: "%{month} %{day} %{time} %{hostname} %{service}: %{message}"
field: "message"
target_prefix: "dissect"
- if:
regexp:
message: ((E|e)rror|(f|F)ault)
then:
- add_fields:
target: 'dissect'
fields:
level: error
else:
- if:
regexp:
message: (W|w)arning
then:
- add_fields:
target: 'dissect'
fields:
level: warning
else:
- add_fields:
target: 'dissect'
fields:
level: information
I'm trying to achieve the following using Ansible:
Define a YAML file with some variables in the dotted format inside it (variables.yml)
database.hosts[0]: "db0"
database.hosts[1]: "db1"
database.hosts[2]: "db2"
foo.bar: 1
foo.baz: 2
Load the variables in variables.yml by using the include_vars module in my playbook (playbook.yml) and print them in a tree structure
- hosts: all
gather_facts: not
tasks:
- name: "Loading vars"
run_once: true
include_vars:
file: 'variables.yml'
- name: "Testing"
debug:
msg: "{{ foo }}"
- name: "Testing"
debug:
msg: "{{ database }}"
Running this results in the following error:
fatal: [host0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'foo' is undefined\n\nThe error appears to be in '.../playbook.yml': line 9, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - name: \"Testing\"\n ^ here\n"}
Which makes it clear that each property in the YAML file has been loaded as a separate property and not as properties within two trees rooted in database and foo.
Of course, the playbook works as expected if I specify the properties as follows:
database:
hosts:
- "db0"
- "db1"
- "db2"
foo:
bar: 1
baz: 2
However, I need the YAML variables file to be in the dotted format instead of in the classic indented format. Is there any way to achieve this? E.g.: a module different from include_vars or some configuration that I can add to the ansible.cfg file? I have already tried to use hash_behaviour=merge, but that didn't help.
Q: "I need the YAML variables file to be in the dotted format instead of in the classic indented format. Is there any way to achieve this?"
A: No. It's' not possible. See Creating valid variable names.
I want multiline in one log merge into one record in ES, here is my config part for multiline.
multiline.negate: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
multiline.negate: true
multiline.match: after
But it can't work as expected, It always record each line into ES.
multi-line fields should under filebeat.inputs:
example:
filebeat.inputs:
-type: log
...
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
multiline.negate: true
multiline.match: after
...
I am trying to use wildcard for my ansible variable but it seems like i cant manage to use it.
I have tried something from here but still the same.
the ansible output
"reboot_required": false,
"updates": {
"0720a128-90b1-4b21-a8cf-3c5c86239435": {
"kb": [
"2267602"
],
"installed": false,
"id": "0720a128-90b1-4b21-a8cf-3c5c86239435",
"categories": [
"Definition Updates",
"Windows Defender"
],
"title": "Definition Update for Windows Defender Antivirus - KB2267602 (Definition 1.297.412.0)"
},
"60bbf4af-afd3-45fe-aad2-6d72beddeba2": {
"kb": [
"4509475"
],
"installed": false,
"id": "60bbf4af-afd3-45fe-aad2-6d72beddeba2",
"categories": [
"Updates",
"Windows Server 2016"
],
"title": "2019-06 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4509475)"
I am trying to get the title, or id
- name: debug
debug:
msg: "{{ item.updates.*.id }}"
with_items:
- "{{ result }}"
appreciate the help
Given the ansible output above is stored in the variable result the tasks below
- set_fact:
id_list: "{{ result.updates|
json_query('*.id')
}}"
- debug:
var: id_list
give the list of id (similar titles)
id_list:
- 0720a128-90b1-4b21-a8cf-3c5c86239435
- 60bbf4af-afd3-45fe-aad2-6d72beddeba2
And the tasks below
- set_fact:
my_list: "{{ result.updates|
json_query('*.{ id: id, title: title }')
}}"
- debug:
var: my_list
give the list of the id, title hashes
my_list:
- id: 0720a128-90b1-4b21-a8cf-3c5c86239435
title: Definition Update for Windows Defender Antivirus - KB2267602 (Definition 1.297.412.0)
- id: 60bbf4af-afd3-45fe-aad2-6d72beddeba2
title: 2019-06 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4509475)
The wildcard he's using in the example that you linked is apart of the json_query filter. He's piping to the json_query filter and then using the wildcard as part of that syntax.
results | json_query('[].block_device_mapping.*.snapshot_id')
You're not using json_query in your example and therefore, this syntax is not available and won't work.
Try piping your results to json_query and then including the path that you want to get to. If {{ results }} is already created you can leave off the with_items and go with something like:
{{ results | json_query('updates.*.id') }}
I'm guessing here at the exact syntax but you definitely have to start with json_query.
To figure out the exact syntax you want, start small piping to json_query and then grabbing the top most element(updates, in your case), adding pieces to the filter until you've narrowed it down to the information you want. I've linked to a pathfinder below that helps.
Reference:
json_query filter documentation
json_query documentation
jsonpath finder to help you figure stuff out easier.
edit: The syntax in the first part of the answer from Vladimir looks way sexier than what I'm guessing at. Try his syntax to get to what works, use my answer to understand what's wrong. Then mark him as the correct answer.
Try below. I have not tested it though.
- name: debug
debug:
msg: "{{ item|first }}:{{ item[item|first].title }}"
with_items:
- "{{ result.updates }}"
I am using yaml2json for the first time. My OS is Windows 7 and I am using git bash.
May be I am missing something very basic, can you guys please help/guide me here.
I tried sending the output of the bash text processing command to test.yml and I can see the test.yml file is created properly. But once I feed it as a input to yaml2json, it parses just the first line "version" :1 and exits without any error.
However, If I try to convert test.yml file contents online via site:--http://yamltojson.com/-- the resulting .json is proper.
Following are the contents of test.yml file generated:--
version: 1
layout: post
lang: en
slug: "checklist"
type: "modal"
title: "Checklist"
published: "true"
categories: "mobile"
tags: "mobile"
action:
title: "Disguise Now" link: "close"
title: "Cancel" link: "home-ready" status: disabled checklist:
title: "Review security plan and update contacts regularly"
I encountered the same problem and solved it by starting the document with
---
So for example ...
---
version: 1
layout: post
lang: en
slug: "checklist"
type: "modal"
title: "Checklist"
published: "true"
categories: "mobile"
tags: "mobile"
... works well, but may not solve your problem because you are using a generated yaml file.
There are more problems with yaml2json (e.g. interpreting the sign of a negative number as list item indicator). So in many cases I use a simple python script (python 2.7 or higher provided) instead of using yaml2json. Only disadvantage I can see is that as opposed to yaml2json the order of dictionary entries is not preserved, but that's just a cosmetical issue, not a logical one:
python -c 'import sys, json, yaml; print json.dumps(yaml.load(sys.stdin), indent=4)' < myyamlfile.yaml
#yaccob's solution worked for me. Just had to add the Loader yaml.load(sys.stdin, Loader=yaml.FullLoader parameter to avoid the deprecation warning:
python2 -c 'import sys, json, yaml; print json.dumps(yaml.load(sys.stdin, Loader=yaml.FullLoader), indent=4)' < sample.yaml