WSO2 api manager is not sending data to WSO2 DAS (Data analytics server) - ssl

Question

I configured API manager to send data to WSO2 data analytics server.
My configuration on API server is:
Event Receiver Configurations: {tcp://wso2-dac-svc.libre-dev.com:7611}
Data Analyzer Configurations: https://wso2-dac-svc.libre-dev.com:8443
On DAC server I installed API_Manager_Analytics.car which has event receiver definitions.
On API server I have imported certificate from DAC server using keytool. I restarted both servers.
I am seeing following error in the log:
TID: [-1] [] [2016-05-16 16:06:11,417] ERROR {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher} - Error while connection to event receiver {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher}
org.wso2.carbon.databridge.commons.exception.AuthenticationException: Access denied for user admin to login TCP,wso2-dac-svc.libre-dev.com:7611,TCP,wso2-dac-svc.libre-dev.com:7711
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.authenticator.AgentAuthenticator.connect(AgentAuthenticator.java:54)
at org.wso2.carbon.databridge.agent.thrift.DataPublisher.start(DataPublisher.java:273)
at org.wso2.carbon.databridge.agent.thrift.DataPublisher.<init>(DataPublisher.java:161)
at org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher$ReceiverConnectionWorker.run(AsyncDataPublisher.java:843)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.agent.thrift.exception.AgentAuthenticatorException: Thrift exception
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.authenticator.ThriftAgentAuthenticator.connect(ThriftAgentAuthenticator.java:51)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.authenticator.AgentAuthenticator.connect(AgentAuthenticator.java:51)
... 8 more
Caused by: org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.send_connect(ThriftSecureEventTransmissionService.java:82)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:73)
at org.wso2.carbon.databridge.agent.thrift.internal.publisher.authenticator.ThriftAgentAuthenticator.connect(ThriftAgentAuthenticator.java:47)
... 9 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: No trusted certificate found
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
... 15 more
Caused by: sun.security.validator.ValidatorException: No trusted certificate found
at sun.security.validator.SimpleValidator.buildTrustedChain(SimpleValidator.java:394)
at sun.security.validator.SimpleValidator.engineValidate(SimpleValidator.java:133)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
... 23 more
TID: [-1] [] [2016-05-16 16:06:41,363] ERROR {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher} - Reconnection failed fortcp://wso2-dac-svc.libre-dev.com:7611 {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher}
TID: [-1] [] [2016-05-16 16:07:11,004] WARN {org.apache.synapse.core.axis2.TimeoutHandler} - Expiring message ID : urn:uuid:6ed9fae5-d1fb-4cdf-885b-e101e79faf40; dropping message after timeout of : 30 seconds {org.apache.synapse.core.axis2.TimeoutHandler}
TID: [-1] [] [2016-05-16 16:07:11,371] ERROR {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher} - Reconnection failed fortcp://wso2-dac-svc.libre-dev.com:7611 {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher}
TID: [-1] [] [2016-05-16 16:07:34,514] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-9: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
Basically I cannot get API stats sent to DAS server. Any help is appreciated.

Answer 1

Based on the error log the issue seems to be wit
The Data Analytic Server Thrift runs on port 7711 uses the 'wso2carbon.jks' located in /repository/resources/security by deault. As a result when we need to configure this on /repository/conf/carbon.xml as shown below
<Security> <br>
<!--<br>
KeyStore which will be used for encrypting/decrypting passwords<br>
and other sensitive information.
--><br>
<KeyStore><br>
<!-- Keystore file location--><br>
<Location>${carbon.home}/repository/resources/security/wso2carbon.jks</Location><br>
<!-- Keystore type (JKS/PKCS12 etc.)--><br>
<Type>JKS</Type><br>
<!-- Keystore password--><br>
<Password>wso2carbon</Password>
<!-- Private Key alias--><br>
<KeyAlias>wso2carbon</KeyAlias>
<!-- Private Key password-->
<KeyPassword>wso2carbon</KeyPassword><br>
</KeyStore><br>
For adding a new key store please use the below steps
1. Place the key store in '/repository/resources/security/' folder
2. Update the section 'Security/KeyStore/' of /repository/conf/carbon.xml accordingly
3. Update the keystore references of the data-agent-config.xml accordingly
4. Import the certificate of the new keystore to ESB's trust store located at /repository/resources/security/client-truststore.jks.
Finally, once the private key is changed, its relevent certificate should be imported to the API-Manager trust store located at /repository/resources/security/client-truststore.jks.
Hope this steps will sort the issues on the given error log.