Add custom field on augmented Apache log parsing with Grok in Logstash - apache

Question

here is my problem : Let's say I have some standard Apache logs, like so :
IP1 IP2 - - [13/Jun/2016:14:45:05 +0200] "GET /page/requested.html HTTP/1.1" 200 4860 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0"
I can sucessfully parse these logs with my actual configuration of Logstash :
input {
file {
path => '/home/user/logsDir/*'
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
elasticsearch { }
stdout { codec => rubydebug }
}
But on these logs, I apply some machine learning algorithm and I give them a score. So the new log line looks like that :
IP1 IP2 - - [13/Jun/2016:14:45:05 +0200] "GET /page/requested.html HTTP/1.1" 200 4860 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" 0.00950628507703
Note the 0.00950628507703 at the end of the line, which is the actual score
Now, I would like to parse this line so I could use score for visualisation in Kibana (Logstash is integeated in the whole ELK stack ). So it would be great if the score could be parse as a float.
NB: I can place the score before or after the standard Apache log message and insert any kind of characters between the two (currently it is just a space).
Any idea on how to tackle this problem ?
Thanks in advance !

Answer 1

Eventually I found how to process. I add a little keyword before the score : the word pred
So my lines are know like this :
IP1 IP2 - - [13/Jun/2016:14:45:05 +0200] "GET /page/requested.html HTTP/1.1" 200 4860 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0" pred:0.00950628507703
And I use this configuration for logstash :
input {
file {
path => '/home/user/logsDir/*'
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG} pred:%{NUMBER:prediction_score}"}
}
# I convert the score into a float in order to vizualise it in Kibana
mutate {
convert => {"prediction_score" => "float"}
}
}
output {
elasticsearch { }
stdout { codec => rubydebug }
}
I hope this will help you if you are stuck with the same problem !
Cheers !