With syslog-ng how do you embed regex's in templates - syslog-ng

Question

I am converting a rsyslog template to syslog-ng and I cannot find in the syslog-ng docs how to embed regex's in a template. The incoming message body looks like this:
123 1.2.3.4 4.3.2.1:80 someone#somewhere.com US
The original rsyslog template is:
$template graylog_json,"{\"version\":\"1.1\", \"host\":\"%HOSTNAME:::json%\", \"short_message\":\"Mail Authentication Log\", \"_LogDateTime\":\"%timereported:::date-rfc3339,json%\", \"_Cluster\":\"c25\", \"_ResponseCode\":\"%msg:R,ERE,1,BLANK:^[^ ]*? ([0-9]{3}) --end:json%\", \"_SourceIP\":\"%msg:R,ERE,2,BLANK:^ ([0-9]{3}) ([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})--end:json%\", \"_DestinationIP\":\"%msg:R,ERE,1,BLANK: ([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}):[0-9]{2,4}--end:json%\", \"_DestinationPort\":\"%msg:R,ERE,1,BLANK: [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}:([0-9]{2,4})--end:json%\", \"_UserAccount\":\"%msg:R,ERE,1,BLANK::[0-9]{2,4} ([^ ]{1,})--end:json%\", \"_Country\":\"%msg:R,ERE,2,BLANK::[0-9]{2,4} ([^ ]{1,})( [A-Z?]{2})?--end:json%\"}\n"
The regex bits in the template parse out the relevant fields in the original message. I can't just dump messages to graylog because we use custom fields. I believe I want to use a template in syslog-ng, but I can't find examples, or even docs, showing how to embed regex's inside a template.

Answer 1

looking at the body of your message, you have the following options:
Parse the message with a csv-parser, using the whitespace as separator character. Note that the csv-parser will not split the IP:port, but you can run another csv-parser on the address (this time with : as separator) to do that. You can find examples for that in the syslog-ng documentation
Alternatively, you can write a custom syslog-ng parser in Python to process this message, and use the standard python string functions to separate the message into words and split the IP:port pair.
Using the csv-parser is probably easier and has better performance.
Also, syslog-ng version 3.13 includes a graylog destination (that's not included in the docs yet, but you can find an example in this blog post Graylog as destination in syslog-ng)