Rewriting log data - syslog-ng

Question

I am sending syslog data to my LogZilla server and am unable to rewrite the data using the
Event message:
{"event_type":"Threat_Event","ipv4":"172.31.100.13","hostname":"server1.something.net","source_uuid":"df4df304c3-93f2a-41f89-8dfefd-7f54bdsf5e429f","occured":"06-Aug-2019 02:38:44","severity":"Warning","threat_type":"test file","threat_name":"Eicar","scanner_id":"Real-time file system protection","engine_version":"1498036 (20190805)","object_type":"file","object_uri":"file:///home/admin/g4.txt","action_taken":"cleaned by deleting","threat_handled":true,"need_restart":false,"username":"root","processname":"/usr/bin/vi","circumstances":"Event occurred on a newly created file.","firstseen":"06-Aug-2019 02:38:44","hash":"CF8BD9DFDDFF007F75ADF4C2BE48005CEA317C62"}
Code for automatic key value detection to rewrite message above:
{
"rewrite_rules": [
{
"match": {
"field": "program",
"value": "ESServer"
},
"update": {
"message": "${event_type}, ${ipv4}"
},
"kv": {"separator": ":", "delimiter": ","
}
}
]
}
'''
I am expecting the message to be parsed so that I can setup dashboards based on various fields from the message.

Answer 1

LogZilla doesn't parse kv pairs within quotes, so first you'll need to strip those out. Here's a syslog-ng rule that will do that:
filter f_program {program("ESServer")};
rewrite r_quotes { subst("\"", "", value("MESSAGE") flags("global") condition( filter(f_program))); };
log {
source(s_logzilla);
rewrite (r_quotes);
#filter(f_fwdrops);
destination(d_logzilla_network);
# Uncomment line below for debug/testing of incoming events
#destination(df_debug);
#destination(d_unix_stream);
flags(flow-control,final);
};
You should create a 'rules' directory to store any custom configurations in. Save the above in that directory as syslog.conf (or any name you prefer). The copy it to the conainer and restart syslog-ng:
docker cp syslog.conf lz_syslog:/etc/logzilla/syslog-ng
docker restart lz_syslog
Now those events should have the quotes removed when they come in. Next, create a LogZilla parser rule with the following:
first_match_only: true
rewrite_rules:
- comment:
- 'Name: ESET Security Manager KV'
- 'Sample: "event_type":"Threat_Event","ipv4":"172.31.100.13","hostname":"server1.something.net","source_uuid":"df4df304c3-93f2a-41f89-8dfefd-7f54bdsf5e429f","occured":"06-Aug-2019 02:38:44","severity":"Warning","threat_type":"test file","threat_name":"Eicar","scanner_id":"Real-time file system protection","engine_version":"1498036 (20190805)","object_type":"file","object_uri":"file:///home/admin/g4.txt","action_taken":"cleaned by deleting","threat_handled":true,"need_restart":false,"username":"root","processname":"/usr/bin/vi","circumstances":"Event occurred on a newly created file.","firstseen":"06-Aug-2019 02:38:44","hash":"CF8BD9DFDDFF007F75ADF4C2BE48005CEA317C62"'
- 'Description: ESET K/V Detection and User Tag creation'
match:
field: program
op: =~
value: 'lzadmin'
kv:
delimiter: ""
separator: ":"
pair_separator: ","
tag:
ut_event_type: ${event_type}
ut_ipv4: ${ipv4}
ut_hostname: ${hostname}
Then add the rule:
logzilla rules add kv.json