Logstash count by unique IP - apache

Question

I'm trying to do some log analysis with Logstash.
I need to count unique IPs from an Apache access log, then I need to match them with a count filter, to determine if an email will be sent.
Something like this:
If 10+ access from an unique IP in a 5 minutes interval is found, them I need to send an email with this IP on it.
What would be the best solution for this?

Answer 1

Doing this is surprisingly hard -- to do it you need to create a meter per IP address. Once you have a meter per IP address, you then need to look at it's rate_5m and decide if it's over your threashold (note rate_5m is the per second rate over the last 5 minutes). Once you've decided that you need to send off the alert, you'll probably want to include the IP address in it (so we need to extract that using a ruby filter)... all in all, not sure I'd ever use something like this in production because it would likely chew up memory like crazy (because of the meter per ip address).
filter {
metrics {
meter => "%{ip}"
add_tag => ["metric"]
}
ruby { code => '
ip = nil
if event["tags"].include? "metric"
event.to_hash.each do |key,value|
if key.end_with?(".rate_5m") and value > 0.2
ip = key[0..-9]
end
end
end
if ip
event["ip"] = ip
event["tags"] = ["alert"]
end
'
}
}
output {
if "alert" in [tags] {
email { ... }
}
}
You could probably write a custom filter that is smarter about it using something like the trending algorithm to find IP addresses that are trending higher in count.