Operational Analytics - ZERO Client and Server logs document - ibm-mobilefirst

Question

Server version: 7.1.0.00.20160229-1240
WebSphere ND 8.5.5.9
Mobile app sends logs to the server via WLClientLogReceiver adapter by calling method WL.Logger.send(); I do see the logs from Mobile app in application server SystemOut.log via terminal.
But the Operational Analytics console not loading any Client and Server logs. It shows ZERO documents in the administration tab. But it has documents for other transactions like session, adapter calls, device, etc.
While trying to search the logs using some keywords getting the below error in SystemOut.log file of Application Server.
[4/5/16 16:05:31:029 CDT] 000000fb JavaScriptInt I com.worklight.integration.js.JavaScriptIntegrationLibraryImplementation info Received device logs [project metlifepoc]
[4/5/16 16:05:55:428 CDT] 0000011e RequestProces E org.apache.wink.server.internal.RequestProcessor handleRequest An unhandled exception occurred which will be propagated to the container.
[4/5/16 16:05:55:429 CDT] 0000011e ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: An exception was thrown by one of the service methods of the servlet [data] in application [IMF Operational Analytics]. Exception created : [org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed; shardFailures {[Kdo8LGq9QTy-ghZsOC-8Xw][worklight][0]: SearchParseException[[worklight][0]: from[0],size[10]: Parse Failure [Failed to parse source [{"from":0,"size":10,"query":{"filtered":{"query":{"query_string":{"query":"message: met* ","default_operator":"and","analyzer":"normalization","lowercase_expanded_terms":true,"analyze_wildcard":true}},"filter":{"bool":{"must":{"range":{"timestamp":{"from":"1459310400000","to":"1459915199999","include_lower":true,"include_upper":true}}}}}}},"sort":[{"timestamp":{"order":"desc"}}]}]]]; nested: NoClassDefFoundError[org.apache.lucene.analysis.icu.ICUFoldingFilter (initialization failure)]; nested: RuntimeException[java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; nested: IOException[ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; }{[Kdo8LGq9QTy-ghZsOC-8Xw][worklight]1: SearchParseException[[worklight]1: from[0],size[10]: Parse Failure [Failed to parse source [{"from":0,"size":10,"query":{"filtered":{"query":{"query_string":{"query":"message: met* ","default_operator":"and","analyzer":"normalization","lowercase_expanded_terms":true,"analyze_wildcard":true}},"filter":{"bool":{"must":{"range":{"timestamp":{"from":"1459310400000","to":"1459915199999","include_lower":true,"include_upper":true}}}}}}},"sort":[{"timestamp":{"order":"desc"}}]}]]]; nested: NoClassDefFoundError[org.apache.lucene.analysis.icu.ICUFoldingFilter (initialization failure)]; nested: RuntimeException[java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; nested: IOException[ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; }{[Kdo8LGq9QTy-ghZsOC-8Xw][worklight][2]: SearchParseException[[worklight][2]: from[0],size[10]: Parse Failure [Failed to parse source [{"from":0,"size":10,"query":{"filtered":{"query":{"query_string":{"query":"message: met* ","default_operator":"and","analyzer":"normalization","lowercase_expanded_terms":true,"analyze_wildcard":true}},"filter":{"bool":{"must":{"range":{"timestamp":{"from":"1459310400000","to":"1459915199999","include_lower":true,"include_upper":true}}}}}}},"sort":[{"timestamp":{"order":"desc"}}]}]]]; nested: NoClassDefFoundError[org.apache.lucene.analysis.icu.ICUFoldingFilter (initialization failure)]; nested: RuntimeException[java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; nested: IOException[ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; }{[Kdo8LGq9QTy-ghZsOC-8Xw][worklight][3]: SearchParseException[[worklight][3]: from[0],size[10]: Parse Failure [Failed to parse source [{"from":0,"size":10,"query":{"filtered":{"query":{"query_string":{"query":"message: met* ","default_operator":"and","analyzer":"normalization","lowercase_expanded_terms":true,"analyze_wildcard":true}},"filter":{"bool":{"must":{"range":{"timestamp":{"from":"1459310400000","to":"1459915199999","include_lower":true,"include_upper":true}}}}}}},"sort":[{"timestamp":{"order":"desc"}}]}]]]; nested: NoClassDefFoundError[org.apache.lucene.analysis.icu.ICUFoldingFilter (initialization failure)]; nested: RuntimeException[java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; nested: IOException[ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; }{[Kdo8LGq9QTy-ghZsOC-8Xw][worklight][4]: SearchParseException[[worklight][4]: from[0],size[10]: Parse Failure [Failed to parse source [{"from":0,"size":10,"query":{"filtered":{"query":{"query_string":{"query":"message: met* ","default_operator":"and","analyzer":"normalization","lowercase_expanded_terms":true,"analyze_wildcard":true}},"filter":{"bool":{"must":{"range":{"timestamp":{"from":"1459310400000","to":"1459915199999","include_lower":true,"include_upper":true}}}}}}},"sort":[{"timestamp":{"order":"desc"}}]}]]]; nested: NoClassDefFoundError[org.apache.lucene.analysis.icu.ICUFoldingFilter (initialization failure)]; nested: RuntimeException[java.io.IOException: ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; nested: IOException[ICU data file error: Header authentication failed, please check if you have a valid ICU data file]; }
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:276)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$3.onFailure(TransportSearchTypeAction.java:224)
at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:205)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:790)
]
[4/5/16 16:05:55:443 CDT] 0000011e FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /metlife/runtime/installed/WebSphere/AppServer/profiles/AppSrv01/logs/ffdc/MobileFirstAppServer_4475157a_16.04.05_16.05.55.4322080935388977381237.txt com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters -RE 1123

Answer 1

This issue was fixed by setting the analytics-service.war's classLoader to parentLast.
instructions to do so found here:
https://www.ibm.com/support/knowledgecenter/SSHS8R_7.0.0/com.ibm.worklight.installconfig.doc/monitor/t_installing_op_analytics_websphere.html