Creating Logstash input and grok filter for a custom time format - input

Question

Been almost two months and I cannot figure out how to make the following logs to parse. Challenges faced:
There are double quotes around the logs and the format of the logs is not very consistent
Many tab and odd spaces between the logs
Appreciate any guide on how to start
"[5/10/22 16:07:39:393 GTS] 00000330 SystemErr R at com.ibm.mdr.DrStateMgr.eventFromUser(DrStateMgr.java:2952)"
"[5/10/22 16:07:39:393 GTS] 00000330 SystemErr R at com.ibm.mdr.DrStateMgr.dequeueAndFireEvents(DrStateMgr.java:5010)"
[5/10/22 16:03:49:982 GTS] 000000a4 WebContainer E com.ibm.ws.webcontainer.internal.WebContainer handleRequest TEST_SERVER: A WebGroup/Virtual Host to handle / has not been defined.
[5/8/22 6:43:42:236 GTS] 00000001 SSLConfigMana W AAPKI0003A: The runtime has at least one SSL configuration that supports only weak TLSv1 or TLSv1.1 handshake protocols. For increased security, modify the configuration to use only stronger protocols such as TLSv1.2 or later. Find instructions to update your configuration at https://www.ibm.com/support/pages/node/1077951. SSL configurations that use the weaker SSL protocols include: [XDADefaultSSLSettings((cell):AFDJP01PCell01)].
[5/8/22 6:43:42:220 GTS] 00000001 WSKeyStore W SSPKI0002A: One or more key stores are using the default password.
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I DDPKI0004A: The process has the java security property jdk.tls.disabledAlgorithms set to [SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, DH keySize < 1024, DESede, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, DES_CBC]. The WebSphere Application server is setting the java security property jdk.tls.disabledAlgorithms to [SSLv3, RC4, DH keySize < 768, MD5withRSA].
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I DDPKI0004A: The process has the java security property jdk.certpath.disabledAlgorithms set to [MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224]. The WebSphere Application server is setting the java security property jdk.certpath.disabledAlgorithms to [MD2, RSA keySize < 1024, MD5].
[5/8/22 6:43:42:204 GTS] 00000001 FIPSManager I EEPKI0005A: FIPS security mode is : No FIPS property found.
[5/8/22 6:43:42:204 GTS] 00000001 SSLConfigMana I GGPKI0007A: The SSL configuration is initializing.
[5/8/22 6:43:42:189 GTS] 00000001 SSLComponentI I HHPKI0008A: SSL service is initializing the configuration
[5/8/22 6:43:42:095 GTS] 00000001 PluginConfigS I PLGC0044B: The plug-in configuration service started successfully.
[5/8/22 6:43:41:345 GTS] 00000001 AdminInitiali A ADMN0054E: The administration service is initialized.
[5/8/22 6:43:41:048 GTS] 00000001 ProviderTrack I com.ibm.ffdc.osgi.ProviderTracker AddingService FFDC1007I: FFDC Provider Installed: com.ibm.ws.ffdc.impl.FfdcProvider#ed46329b
[5/8/22 6:43:40:908 GTS] 00000001 ComponentMeta I ASVR0150U: The runtime provisioning feature is disabled. All components will be started.
[5/8/22 6:43:39:923 GTS] 00000001 ModelMgr I ASVR0180U: Initializing core configuration models
[5/8/22 6:43:39:783 GTS] 00000001 ManagerAdmin I TRAS0555T: The message IDs that are in use are deprecated
[5/8/22 6:43:39:783 GTS] 00000001 ManagerAdmin I TRAS0787K: The startup trace state is *=info.
"[5/8/22 7:37:18:809 GTS] FFDC Exception:java.io.FileNotFoundException SourceId:com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters -IOE ProbeId:1044
java.io.FileNotFoundException: DEAV0180D: File not found: /favicon.ico
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:958)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:486)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4075)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1019)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:213)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Reporter:null"
Expected output
{
"month": [
[
"5"
]
],
"day": [
[
"10"
]
],
"year": [
[
"22"
]
],
"time": [
[
"16:03:49:982"
]
],
"instance": [
[
"000000a4"
]
]
"process": [
[
"WebContainer E com.ibm.ws.webcontainer.internal.WebContainer handleRequest TEST_SERVER: A WebGroup/Virtual Host to handle / has not been"
]
]
"server": [
[
"TEST_SERVER"
]
]
"error": [
[
"A WebGroup/Virtual Host to handle / has not been"
]
]
}
Grok pattern in use
\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{GREEDYDATA:host}

Answer 1

For this multiple pattern in a single log you can try like the below. if any of grok parsing failed it might be due to extra space in between hence constructed it and add those patterns in this.
filter
{
grok
{
match => {"message" => ["\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{DATA:host} %{WORD:source} %{WORD:logtype} %{DATA:lib} %{WORD:request_type} %{DATA:server}: %{GREEDYDATA:detailed_message}","\[%{MONTHNUM:month}\/%{MONTHDAY:day}\/%{YEAR:year} %{TIME:time} GTS\] %{DATA:host} %{WORD:source} %{WORD:logtype} %{DATA:code}: %{GREEDYDATA:detailed_message}"]}
}
}
However, the above pattern would only work single line for W,R,I,A log type but it won't work in multi line
i.e.,
"[5/8/22 7:37:18:809 GTS] FFDC Exception:java.io.FileNotFoundException SourceId:com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters -IOE ProbeId:1044
java.io.FileNotFoundException: DEAV0180D: File not found: /favicon.ico
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.processEDR(DefaultExtensionProcessor.java:958)
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor.handleRequest(DefaultExtensionProcessor.java:486)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1114)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:4075)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1019)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1817)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:213)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:316)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:287)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1187)
at com.ibm.ws.ssl.channel.impl.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:694)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1833)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)
Reporter:null"
Keep posted on how it goes !!! Thanks !!!