Google Cloud Messaging SSL error peer not authenticated - google-cloud-messaging

I have an issue regarding GCM. When my server app tries to send a message to GCM, it sometimes throws an error:
16-01-20 18:13:47,993 ERROR [com.chopper.ivolley.server.association.gcm.GcmNoticifationClient] (pool-5-thread-2) GCM returned an error: javax.ws.rs.ProcessingException: Unable to invoke request
at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:287) [resteasy-client-3.0.10.Final.jar:]
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:407) [resteasy-client-3.0.10.Final.jar:]
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:450) [resteasy-client-3.0.10.Final.jar:]
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation$5.call(ClientInvocation.java:513) [resteasy-client-3.0.10.Final.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65]
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:431) [jsse.jar:1.8.0_65]
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:283) [resteasy-client-3.0.10.Final.jar:]
... 7 more
I imported the certificate from android.googleapis.com in my TrustStore and my app uses it, because i can also see log messages with successful m
I have no idea why this happens only from time to time. Could anybody help?

Related

CloudHUB Notification - Fails connections from workers in VPC

I developed an API which will trigger email alerts based on some business requirements. To raise email alerts, I used cloudHub create-notification connector. The connector configuration is below:
<cloudhub:config name="CloudHub_Config" doc:name="CloudHub Config">
<cloudhub:connection
username="${ch.username}"
password="${ch.password}"
environment="${environment.id}"/>
</cloudhub:config>
Username and password are configured in Runtime manager > Application > properties, and the user has required permissions which I verified in Access Management. A Custom Application Notification alert is configured on correct environment in Runtime manager.
I am using below code to generate alerts from mule flows.
<cloudhub:create-notification doc:name="Success" config-ref="CloudHub_Config" domain="${domain}" priority="INFO">
<cloudhub:message><![CDATA[#["
Success scenario
Hello user"]]]>
</cloudhub:message>
<cloudhub:custom-properties><![CDATA[#[output application/java
---
{
"category" : "Info",
"status" : "Success"
}]]]>
</cloudhub:custom-properties>
</cloudhub:create-notification>
I tested this flow on 4.4.0 (latest) runtime on Studio pointing to AnyPoint platform TEST environment, and it was success. I deployed the same code to cloudHub worker (within VPC) and it failed test-connection during application start-up. The deployment went through and started successfully, but it fails when executing the component from flows. The error details is:
Message : Unexpected Error Occurred
Error type : MULE:UNKNOWN
Payload Type : org.mule.runtime.core.internal.streaming.bytes.ManagedCursorStreamProvider
Root Exception stack trace:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:289)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1675)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:1015)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:1012)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1504)
at org.glassfish.grizzly.ssl.SSLUtils.executeDelegatedTask(SSLUtils.java:274)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:735)
at org.glassfish.grizzly.ssl.SSLFilter.doHandshakeStep(SSLFilter.java:345)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:648)
at org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:349)
at com.ning.http.client.providers.grizzly.SwitchingSSLFilter.handleRead(SwitchingSSLFilter.java:74)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:540)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:151)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
CloudHub Notification component looks like not able to connect to platform APIs. The below logs shows the error while start-up of application:
08:45:46.011 07/26/2022 Worker-0 [MuleRuntime].uber.02: [myCompany-api-v1].uber#org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationInstance.testConnectivity:189 #5176f3d4 INFO
Connectivity test failed for config 'CloudHub_Config'. Application deployment will continue. Error was: Unexpected Error occurred trying to validate the connection.
java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:111)
at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:42)
at org.mule.runtime.module.extension.internal.runtime.config.ClassLoaderConnectionProviderWrapper.validate(ClassLoaderConnectionProviderWrapper.java:72)
at org.mule.runtime.core.internal.connection.ReconnectableConnectionProviderWrapper.validate(ReconnectableConnectionProviderWrapper.java:50)
at org.mule.runtime.core.internal.connection.ErrorTypeHandlerConnectionProviderWrapper.validate(ErrorTypeHandlerConnectionProviderWrapper.java:85)
at org.mule.runtime.core.internal.connection.DefaultConnectionManager.doTestConnectivity(DefaultConnectionManager.java:186)
at org.mule.runtime.core.internal.connection.DefaultConnectionManager.lambda$testConnectivity$1(DefaultConnectionManager.java:169)
at org.mule.runtime.core.internal.connection.DefaultConnectionManager.doTestConnectivity(DefaultConnectionManager.java:176)
at org.mule.runtime.core.internal.connection.DefaultConnectionManager.testConnectivity(DefaultConnectionManager.java:152)
at org.mule.runtime.core.internal.connection.DelegateConnectionManagerAdapter$EagerConnectionManagerAdapter.testConnectivity(DelegateConnectionManagerAdapter.java:176)
at org.mule.runtime.core.internal.connection.DelegateConnectionManagerAdapter.testConnectivity(DelegateConnectionManagerAdapter.java:98)
at org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationInstance$1.doWork(LifecycleAwareConfigurationInstance.java:204)
at org.mule.runtime.core.api.retry.policy.AbstractPolicyTemplate.execute(AbstractPolicyTemplate.java:62)
at org.mule.runtime.core.internal.retry.async.RetryWorker.run(RetryWorker.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:152)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:93)
... 20 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1566)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:545)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1217)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1185)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:471)
at org.glassfish.grizzly.ssl.SSLUtils.sslEngineWrap(SSLUtils.java:451)
at org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:360)
at org.glassfish.grizzly.ssl.SSLUtils.handshakeWrap(SSLUtils.java:327)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:724)
at org.glassfish.grizzly.ssl.SSLFilter.doHandshakeStep(SSLFilter.java:345)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:648)
at org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:349)
at com.ning.http.client.providers.grizzly.SwitchingSSLFilter.handleRead(SwitchingSSLFilter.java:74)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:540)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
... 7 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1729)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:333)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:1015)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:1012)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1504)
at org.glassfish.grizzly.ssl.SSLUtils.executeDelegatedTask(SSLUtils.java:274)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:735)
... 25 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:450)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:289)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1675)
... 33 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
... 39 more
Scenarios I tried:
Running the same code on 4.3.0 runtime on same cloudHub
worker within VPC - IT FAILED.
Running the same code on 4.4.0 (latest) runtime on another
AnyPoint platform's cloudHub worker which is not on VPC - IT WORKED FINE.
Seems like connectivity of cloudHub-notification component running on worker within VPC is causing SSL handshake error. Strange that this component doesn't provide any settings to configure SSL.
Has anyone faced this issue before, if yes - please help me here?

Apache Beam pipeline running on Dataflow failed to read from KafkaIO: SSL handshake failed

I'm building an Apache Beam pipeline to read from Kafka as an unbounded source.
I was able to run it locally using direct runner.
However, the pipeline would fail with the attached exception stack trace, when run using Google Cloud Dataflow runner on the cloud.
It seems it's ultimately the Conscrypt Java library that's throwing javax.net.ssl.SSLException: Unable to parse TLS packet header. I'm not really sure how to address this issue.
java.io.IOException: Failed to start reading from source: org.apache.beam.sdk.io.kafka.KafkaUnboundedSource#33b5ff70
com.google.cloud.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:783)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:360)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:193)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1227)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:135)
com.google.cloud.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:966)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
org.apache.beam.sdk.io.kafka.KafkaUnboundedReader.start(KafkaUnboundedReader.java:126)
com.google.cloud.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:778)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:360)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:193)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1227)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:135)
com.google.cloud.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:966)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:206)
org.apache.beam.sdk.io.kafka.KafkaUnboundedReader.start(KafkaUnboundedReader.java:112)
com.google.cloud.dataflow.worker.WorkerCustomSources$UnboundedReaderIterator.start(WorkerCustomSources.java:778)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation$SynchronizedReaderIterator.start(ReadOperation.java:360)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:193)
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1227)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:135)
com.google.cloud.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:966)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLException: Unable to parse TLS packet header
org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:782)
org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:723)
org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:688)
org.conscrypt.Java8EngineWrapper.unwrap(Java8EngineWrapper.java:236)
org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:464)
org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:328)
org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:255)
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:79)
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:460)
org.apache.kafka.common.network.Selector.poll(Selector.java:398)
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238)
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:190)
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:219)
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:205)
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.fetchCommittedOffsets(ConsumerCoordinator.java:468)
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.refreshCommittedOffsetsIfNeeded(ConsumerCoordinator.java:450)
org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:1772)
org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1411)
org.apache.beam.sdk.io.kafka.KafkaUnboundedReader.setupInitialOffset(KafkaUnboundedReader.java:641)
org.apache.beam.sdk.io.kafka.KafkaUnboundedReader.lambda$start$0(KafkaUnboundedReader.java:106)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Looks like Conscrypt causes SSL errors in many cotexts like this. Dataflow worker in Beam 2.9.0 has an option to disable this. Please try. --experiment=disable_conscrypt_security_provider. Alternately, you can try Beam 2.4.x, which does not enable Conscrypt.

wso2 iot analytics geofencing error

I've downloaded WSO2 IOT Server 3.0.0 and installed the product by following steps in wso2 documentation. I use Ubuntu 14.04 LTS.
https://docs.wso2.com/display/IoTS300/Installing+on+Linux+or+OS+X
Then, I started broker, core and analytics servers in given order. I did not make any additional configuration to follow link below:
https://docs.wso2.com/display/IoTS300/Running+the+Product#RunningtheProduct-StartingtheServer
I created sample users and user roles by running mobile-qsg.sh script.
I successfully enrolled to wso2 iot server with my Android( 6.0.1) device( Samsung SM-G900FQ). I used account for alex who has iotMobileUser role. I can see my device info on devicemgt console. In order to add a geofencing rule, I opened geo dashboard running on https//localhost:9445/portal/dashboards/geo-dashboard/, authenticated with admin account and able to login.
When I send a location request to my device via devicemgt api, it throws EventConversionException.
[2017-05-28 13:03:46,391] [IoT-Analytics] INFO {org.wso2.carbon.dashboard.deployment.DashboardDeployer} - Gadget directory [geo-dashboard] has been copied to path /home/dozc/wso2iot-3.0.0/analytics/repository/deployment/server/jaggeryapps/portal/store/carbon.super/fs/gadget/geo-dashboard
[2017-05-28 13:03:46,427] [IoT-Analytics] INFO {org.wso2.carbon.dashboard.deployment.DashboardDeployer} - Dashboard definition [geo-dashboard] has been created.
[2017-05-28 13:03:46,428] [IoT-Analytics] ERROR {org.wso2.carbon.dashboard.deployment.DashboardDeployer} - A theme already exists with the name geo-dashboard
[2017-05-28 13:03:46,442] [IoT-Analytics] INFO {org.wso2.carbon.application.deployer.internal.ApplicationManager} - Successfully Deployed Carbon Application : org.wso2.carbon.iot.geo.dashboard_1.0.0 {super-tenant}
[2017-05-28 13:03:46,751] [IoT-Analytics] INFO {org.wso2.carbon.ntask.core.impl.AbstractQuartzTaskManager} - Task scheduled: [-1234][ANALYTICS_SPARK_EVENTING][STORE_EVENT_ROUTER_TASK]
[2017-05-28 13:03:46,877] [IoT-Analytics] INFO {org.wso2.carbon.core.init.JMXServerManager} - JMX Service URL : service:jmx:rmi://localhost:11113/jndi/rmi://localhost:10001/jmxrmi
[2017-05-28 13:03:46,880] [IoT-Analytics] INFO {org.wso2.carbon.analytics.eventsink.AnalyticsEventStoreDeployer} - Deploying analytics event store: org_wso2_geo_FusedSpatialEvent.xml
[2017-05-28 13:03:46,918] [IoT-Analytics] INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} - Server : WSO2 IoT - Analytics-3.0.0
[2017-05-28 13:03:46,920] [IoT-Analytics] INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} - WSO2 Carbon started in 95 sec
[2017-05-28 13:03:46,990] [IoT-Analytics] INFO {org.wso2.carbon.event.stream.core.internal.EventJunction} - WSO2EventConsumer added to the junction. Stream:org.wso2.geo.FusedSpatialEvent:1.0.0
[2017-05-28 13:03:46,992] [IoT-Analytics] INFO {org.wso2.carbon.analytics.eventsink.AnalyticsEventStoreDeployer} - Deployed successfully analytics event store: org_wso2_geo_FusedSpatialEvent.xml
[2017-05-28 13:03:47,606] [IoT-Analytics] INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} - Mgt Console URL : https://192.x.x.x:9445/carbon/
[2017-05-28 13:03:47,686] [IoT-Analytics] INFO {org.wso2.carbon.event.receiver.core.EventReceiverDeployer} - Event Receiver undeployed successfully: Geo-Receiver-HTTP-LocationStream.xml
[2017-05-28 13:03:47,721] [IoT-Analytics] INFO {org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime} - Connecting receiver Geo-Receiver-HTTP-LocationStream
[2017-05-28 13:03:47,725] [IoT-Analytics] INFO {org.wso2.carbon.event.stream.core.internal.EventJunction} - Producer added to the junction. Stream:org.wso2.geo.LocationStream:1.0.0
[2017-05-28 13:03:47,725] [IoT-Analytics] INFO {org.wso2.carbon.event.receiver.core.EventReceiverDeployer} - Event Receiver configuration successfully deployed and in active state: Geo-Receiver-HTTP-LocationStream
[2017-05-28 13:03:58,761] [IoT-Analytics] INFO {org.wso2.carbon.event.processor.manager.core.internal.CarbonEventManagementService} - Starting polling event receivers
[2017-05-28 13:26:31,353] [IoT-Analytics] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected
[2017-05-28 13:26:31,468] [IoT-Analytics] ERROR {org.wso2.carbon.databridge.core.internal.queue.QueueWorker} - Dropping wrongly formatted event sent for -1234
org.wso2.carbon.databridge.core.exception.EventConversionException: Error when converting org.wso2.android.agent.Stream:1.0.0 of event bundle with events 1
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:181)
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.toEventList(ThriftEventConverter.java:90)
at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.core.exception.EventConversionException: No StreamDefinition for streamId org.wso2.android.agent.Stream:1.0.0 present in cache
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:166)
... 7 more
[2017-05-28 13:32:32,042] [IoT-Analytics] ERROR {org.wso2.carbon.databridge.core.internal.queue.QueueWorker} - Dropping wrongly formatted event sent for -1234
org.wso2.carbon.databridge.core.exception.EventConversionException: Error when converting org.wso2.android.agent.Stream:1.0.0 of event bundle with events 1
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:181)
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.toEventList(ThriftEventConverter.java:90)
at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.core.exception.EventConversionException: No StreamDefinition for streamId org.wso2.android.agent.Stream:1.0.0 present in cache
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:166)
... 7 more
When I try to define a geofencing query and shape on geo dashboard, after pressing save, it throws follwing error:
[2017-05-28 13:47:35,301] [IoT-Analytics] ERROR {org.wso2.carbon.databridge.core.internal.queue.QueueWorker} - Dropping wrongly formatted event sent for -1234
org.wso2.carbon.databridge.core.exception.EventConversionException: Error when converting org.wso2.android.agent.Stream:1.0.0 of event bundle with events 1
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:181)
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.toEventList(ThriftEventConverter.java:90)
at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.core.exception.EventConversionException: No StreamDefinition for streamId org.wso2.android.agent.Stream:1.0.0 present in cache
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:166)
... 7 more
[2017-05-28 13:48:14,089] [IoT-Analytics] INFO {JAGGERY.store.carbon.super.fs.gadget.geo-dashboard.controllers.set_alerts:jag} - /* Enter a unique ExecutionPlan */
#Plan:name('Geo-ExecutionPlan-Within_home---_4e42e50120311aea_alert')
/* Enter a unique description for ExecutionPlan */
-- #Plan:description('ExecutionPlan')
/* define streams/tables and write queries here ... */
#Import('org.wso2.geo.StandardSpatialEvents:1.0.0')
define stream dataIn (id string, latitude double, longitude double, timeStamp long, type string ,speed float, heading float, eventId string);
#Export('org.wso2.geo.ProcessedSpatialEvents:1.0.0')
define stream dataOut (id string, latitude double, longitude double, timeStamp long, type string ,speed float, heading float, eventId string, state string, information string);
from dataIn[geo:within(longitude,latitude,"{'type':'Polygon','coordinates':[[[32.79895305633545,39.98135815980835],[32.799296379089355,39.98124306559073],[32.79925346374511,39.98077857624252],[32.799028158187866,39.98035107883081],[32.79843807220459,39.97984547747823],[32.79794454574585,39.98008800217112],[32.79849708080292,39.980803239473126],[32.79895305633545,39.98135815980835]]]}")==true and id == "4e42e50120311aea"]#geodashboard:subscribe()
select id , latitude, longitude,timeStamp, type, speed, heading ,eventId , "ALERTED" as state, "This device is in home restricted area!!!" as information
insert into dataOut;
from dataIn[geo:within(longitude,latitude,"{'type':'Polygon','coordinates':[[[32.79895305633545,39.98135815980835],[32.799296379089355,39.98124306559073],[32.79925346374511,39.98077857624252],[32.799028158187866,39.98035107883081],[32.79843807220459,39.97984547747823],[32.79794454574585,39.98008800217112],[32.79849708080292,39.980803239473126],[32.79895305633545,39.98135815980835]]]}")!=true and id == "4e42e50120311aea"]
select id , latitude, longitude,timeStamp, type, speed, heading ,eventId , "NORMAL" as state, "" as information
insert into dataOut;
[2017-05-28 13:48:14,139] [IoT-Analytics] WARN {org.wso2.carbon.server.admin.module.handler.AuthenticationHandler} - Illegal access attempt at [2017-05-28 13:48:14,0139] from IP address 127.0.0.1 while trying to authenticate access to service EventProcessorAdminService
[2017-05-28 13:48:14,144] [IoT-Analytics] ERROR {org.jaggeryjs.jaggery.core.manager.WebAppManager} - org.mozilla.javascript.WrappedException: Wrapped org.apache.axis2.AxisFault: Transport error: 401 Error: Unauthorized (/portal/store/carbon.super/fs/gadget/geo-dashboard/controllers/set_alerts.jag#117)
org.jaggeryjs.scriptengine.exceptions.ScriptException: org.mozilla.javascript.WrappedException: Wrapped org.apache.axis2.AxisFault: Transport error: 401 Error: Unauthorized (/portal/store/carbon.super/fs/gadget/geo-dashboard/controllers/set_alerts.jag#117)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:571)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:588)
at org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:508)
at org.jaggeryjs.jaggery.core.JaggeryServlet.doPost(JaggeryServlet.java:29)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:747)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:377)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.jaggeryjs.jaggery.core.JaggeryFilter.doFilter(JaggeryFilter.java:21)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.dashboard.store.filter.TenantStoreFilter.doFilter(TenantStoreFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:120)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:958)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1756)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1715)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.mozilla.javascript.WrappedException: Wrapped org.apache.axis2.AxisFault: Transport error: 401 Error: Unauthorized (/portal/store/carbon.super/fs/gadget/geo-dashboard/controllers/set_alerts.jag#117)
at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:1754)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:148)
at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:225)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.jaggeryjs.rhino.portal.store.carbon.super.fs.gadget.geo_dashboard.controllers.c6._c_script_0(/portal/store/carbon.super/fs/gadget/geo-dashboard/controllers/set_alerts.jag:117)
at org.jaggeryjs.rhino.portal.store.carbon.super.fs.gadget.geo_dashboard.controllers.c6.call(/portal/store/carbon.super/fs/gadget/geo-dashboard/controllers/set_alerts.jag)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
at org.jaggeryjs.rhino.portal.store.carbon.super.fs.gadget.geo_dashboard.controllers.c6.call(/portal/store/carbon.super/fs/gadget/geo-dashboard/controllers/set_alerts.jag)
at org.jaggeryjs.rhino.portal.store.carbon.super.fs.gadget.geo_dashboard.controllers.c6.exec(/portal/store/carbon.super/fs/gadget/geo-dashboard/controllers/set_alerts.jag)
at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:567)
... 45 more
Caused by: org.apache.axis2.AxisFault: Transport error: 401 Error: Unauthorized
at org.apache.axis2.transport.http.HTTPSender.handleResponse(HTTPSender.java:326)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:196)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.wso2.carbon.event.processor.stub.EventProcessorAdminServiceStub.validateExecutionPlan(EventProcessorAdminServiceStub.java:2207)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
... 54 more
I searched all issues related to these exceptions and made additional configurations mentioned in older product documentations, but couldn't find a solution. Is there any additional configuration that I could miss?
Thanks in advance
First issue is because of that org.wso2.android.agent.Stream definition is not available with vanila distribution pack.
You can download it from the this link or build this. Extract the archive and copy the android_agent_analytics-1.0.0.car into $IOT_HOME/analytics/repository/deployments/server/carbonapps path (this issue is fixed in the latest).
For the second issue; There's an authentication issue. Did you try login into the /portal with a private browser window? It might fix the issue.
WSO2 IoT 3.1.0 is released, and we have fixed those issues. And now it includes the geo-fencing alert for exit and entry too.

Cannot borrow client for ssl://das_server:7712

I´m trying to configure DAS server with my Gateway wso2 API Management.
In my log always appear the messages bellow:
TID: [-1] [] [2017-04-24 19:55:11,133] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker} - Error while trying to connect to the endpoint. Cannot borrow client for ssl://das_server:7712 {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://das_server:7712
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:100)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift exception
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:50)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:94)
... 6 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.recv_connect(ThriftSecureEventTransmissionService.java:110)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:96)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:46)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
... 15 more
Does someone have an idea about this error?
My architecture is distributed components
Please make sure to add the GW certificate to the client-trustore.jks of the DAS server. This can be the issue which is not trusting your GW from the DAS server.

Change default admin password on WSO2 DAS

After installing WSO2 Data Analytics Server and changing the default admin password i am getting the following ERROR
ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
- Error while trying to connect to the endpoint. Cannot borrow client for ssl://localhost:7711 {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://localhost:7711
Does anybody knows why this is happening. What conf file i have to change and how?
stack trace:
TID: [-1234] [] [2016-08-01 09:06:19,010] ERROR {org.wso2.carbon.databridge.core.internal.authentication.Authenticator}
- wrong userName or password {org.wso2.carbon.databridge.core.internal.authentication.Authenticator}
TID: [-1] [] [2016-08-01 09:06:19,013] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
- Thrift Authentication Exception {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift Authentication Exception
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:48)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:94)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ThriftAuthenticationException(message:wrong userName or password)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$connect_result.read(ThriftSecureEventTransmissionService.java:1394)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.recv_connect(ThriftSecureEventTransmissionService.java:88)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:74)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:46)
... 7 more
TID: [-1] [] [2016-08-01 09:06:19,015] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
- Error while trying to connect to the endpoint. Cannot borrow client for ssl://localhost:7711 {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://localhost:7711
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:100)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift Authentication Exception
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:48)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:94)
... 6 more
Caused by: ThriftAuthenticationException(message:wrong userName or password)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$connect_result.read(ThriftSecureEventTransmissionService.java:1394)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.recv_connect(ThriftSecureEventTransmissionService.java:88)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:74)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:46)
... 7 more
TID: [-1234] [] [2016-08-01 09:06:29,070] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2016-08-01 09:06:29,070+0300] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [-1] [] [2016-08-01 09:06:48,893] INFO {org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup} - No receiver is reachable at reconnection, will try to reconnect every 30 sec {org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup}
TID: [-1] [] [2016-08-01 09:06:48,927] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected {org.wso2.carbon.databridge.core.DataBridge}
TID: [-1234] [] [2016-08-01 09:06:48,932] ERROR {org.wso2.carbon.databridge.core.internal.authentication.Authenticator}
- wrong userName or password {org.wso2.carbon.databridge.core.internal.authentication.Authenticator}
TID: [-1] [] [2016-08-01 09:06:48,933] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
- Thrift Authentication Exception {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift Authentication Exception
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:48)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:94)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ThriftAuthenticationException(message:wrong userName or password)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$connect_result.read(ThriftSecureEventTransmissionService.java:1394)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.recv_connect(ThriftSecureEventTransmissionService.java:88)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:74)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:46)
... 7 more
TID: [-1] [] [2016-08-01 09:06:48,935] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
- Error while trying to connect to the endpoint. Cannot borrow client for ssl://localhost:7711 {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker}
org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Cannot borrow client for ssl://localhost:7711
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:100)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:43)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointAuthenticationException: Thrift Authentication Exception
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:48)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:94)
... 6 more
Caused by: ThriftAuthenticationException(message:wrong userName or password)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$connect_result.read(ThriftSecureEventTransmissionService.java:1394)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.recv_connect(ThriftSecureEventTransmissionService.java:88)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:74)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:46)
... 7 more
Thanks for sharing the stack trace. The logs show ThriftAuthenticationException(message:wrong userName or password) this means that even though the admin password is changed via the UI that some places are still referring to the default password.
If we are publishing events into DAS from ex: API Manager 1.10.0. If the admin's password has been changed on DAS, then on the other products(ex:api manager) we need to specify the new password where ever the as well as the admin
and admin are given such as in api-manager.xml
When we grep for port 7711 within the DAS pack the matrix.xml is also having admin as the password as a result could you kindly change the password on the relevant areas and check again.
regards,shavantha
there are two ways to change the password. However,
1.We can change the password by updating the admin-password via user-mgt.xml or via the web interface as mentioned by you
The below document has the guidelines that will help you change the password
https://docs.wso2.com/display/DAS301/Adding+and+Managing+Users+and+Roles
However, appreciate if you can provide the following answers to try and reproduce your scenario
1.Did this error prevent you from logging in after the attempt to change the default password
2.Can you please share the full stack trace for the error.
3.What is the Operating system and JDK version are you using
I have resolved the same error on EI 6.1.0 and EI 6.1.1 (Integrator and Analytics profile activated)
After the change of the admin password must be update this files for databridge comunications:
EI_HOME/repository/deployment/server/eventpublishers/MessageFlowConfigurationPublisher.xml
EI_HOME/repository/deployment/server/eventpublishers/MessageFlowStatisticsPublisher.xml
Set property encrypted to false and replace the encrypted old password with not encrypted new password
<property encrypted="false" name="password">NOT-ENCRYPTED-NEW-PASSWORD</property>
/conf/log4j.properties
Change the password for log4j.appender.LOGEVENT.password
[SOLVED] Finally the problem was a sample carbon application that was deployed and was trying to connect with the default credentials. Thank you all
user-mgt.xml
UserManager->Password
repository/conf/log4j.properties
# DAS_AGENT uses PatternLayout.
log4j.appender.DAS_AGENT.layout=org.wso2.carbon.analytics.shared.data.agents.log4j.util.TenantAwarePatternLayout
log4j.appender.DAS_AGENT.columnList=%D,%S,%A,%d,%c,%p,%m,%H,%I,%Stacktrace
log4j.appender.DAS_AGENT.userName=admin
log4j.appender.DAS_AGENT.password=xxxxxxx
log4j.appender.DAS_AGENT.url=tcp://localhost:7612
log4j.appender.DAS_AGENT.maxTolerableConsecutiveFailure=5
log4j.appender.DAS_AGENT.streamDef=loganalyzer:1.0.0
log4j.logger.trace.messages=TRACE,CARBON_TRACE_LOGFILE