Logstash: modify apache date format - apache

Question

The grok-filter %{COMBINEDAPACHELOG} formats the timestamp as dd/MMM/YYYY:HH:mm:ss Z however I need the timestamp in the format of yyyy-MM-dd HH:mm:ss
I tried the below configuration
grok {
match => [
"message", "%{COMBINEDAPACHELOG}",
]
break_on_match => false
}
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss" ]
target => ["datetime"]
}
but got the below parsing error:
Failed parsing date from field {:field=>"timestamp", :value=>"19/May/2012:12:40:18 -0700", :exception=>java.lang.IllegalArgumentException: Invalid format: "19/May/2012:12:40:18 -0700" is malformed at "/May/2012:12:40:18 -0700", :level=>:warn}
Would highly appreciate if anyone can throw more light on the same.

Answer 1

The COMBINEDAPACHELOG pattern is expecting the date in the log entry to match the format so it can shove it into the "timestamp" field. It doesn't format your timestamp at all.
Once the date has been grok'ed out into "timestamp", you can use the date{} filter to move it into #timestamp. The pattern you supply there should match whatever's in the field.
So, pass "dd/MMM/yyyy:HH:mm:ss Z" as the format to date{} and you should be all set.
EDIT:
Based on your additional details, I was hoping that you could match each component of the input date and then combine them into a new field. That would work if you were trying to swap, say, firstName and lastName in a string, but dates are more complicated. A simple string swap wouldn't handle converting "Jan" to "01" or deal with timezones at all.
So, we're back to creating a date object and then outputting that as a string in the format you desire.
# convert "timestamp" to a date field "datetime"
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => ["datetime"]
}
# convert "datetime" to a string "datestring"
ruby {
code => "
event['datestring'] = event['datetime'].strftime('%Y-%m-%d %H:%M:%S')
"
}

Answer 2

For the latest version of Logstash the code would be:
# convert "datetime" to a string "datestring"
ruby {
code => "event.set('datestring', event.get('datetime').strftime('%Y-%m-%d %H:%M:%S'))"
}