CloudHUB Notification - Fails connections from workers in VPC - mule

Question

I developed an API which will trigger email alerts based on some business requirements. To raise email alerts, I used cloudHub create-notification connector. The connector configuration is below:
<cloudhub:config name="CloudHub_Config" doc:name="CloudHub Config">
<cloudhub:connection
username="${ch.username}"
password="${ch.password}"
environment="${environment.id}"/>
</cloudhub:config>
Username and password are configured in Runtime manager > Application > properties, and the user has required permissions which I verified in Access Management. A Custom Application Notification alert is configured on correct environment in Runtime manager.
I am using below code to generate alerts from mule flows.
<cloudhub:create-notification doc:name="Success" config-ref="CloudHub_Config" domain="${domain}" priority="INFO">
<cloudhub:message><![CDATA[#["
Success scenario
Hello user"]]]>
</cloudhub:message>
<cloudhub:custom-properties><![CDATA[#[output application/java
---
{
"category" : "Info",
"status" : "Success"
}]]]>
</cloudhub:custom-properties>
</cloudhub:create-notification>
I tested this flow on 4.4.0 (latest) runtime on Studio pointing to AnyPoint platform TEST environment, and it was success. I deployed the same code to cloudHub worker (within VPC) and it failed test-connection during application start-up. The deployment went through and started successfully, but it fails when executing the component from flows. The error details is:
Message : Unexpected Error Occurred
Error type : MULE:UNKNOWN
Payload Type : org.mule.runtime.core.internal.streaming.bytes.ManagedCursorStreamProvider
Root Exception stack trace:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:289)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1675)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:1015)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:1012)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1504)
at org.glassfish.grizzly.ssl.SSLUtils.executeDelegatedTask(SSLUtils.java:274)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:735)
at org.glassfish.grizzly.ssl.SSLFilter.doHandshakeStep(SSLFilter.java:345)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:648)
at org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:349)
at com.ning.http.client.providers.grizzly.SwitchingSSLFilter.handleRead(SwitchingSSLFilter.java:74)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:540)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:151)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
CloudHub Notification component looks like not able to connect to platform APIs. The below logs shows the error while start-up of application:
08:45:46.011 07/26/2022 Worker-0 [MuleRuntime].uber.02: [myCompany-api-v1].uber#org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationInstance.testConnectivity:189 #5176f3d4 INFO
Connectivity test failed for config 'CloudHub_Config'. Application deployment will continue. Error was: Unexpected Error occurred trying to validate the connection.
java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:111)
at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:42)
at org.mule.runtime.module.extension.internal.runtime.config.ClassLoaderConnectionProviderWrapper.validate(ClassLoaderConnectionProviderWrapper.java:72)
at org.mule.runtime.core.internal.connection.ReconnectableConnectionProviderWrapper.validate(ReconnectableConnectionProviderWrapper.java:50)
at org.mule.runtime.core.internal.connection.ErrorTypeHandlerConnectionProviderWrapper.validate(ErrorTypeHandlerConnectionProviderWrapper.java:85)
at org.mule.runtime.core.internal.connection.DefaultConnectionManager.doTestConnectivity(DefaultConnectionManager.java:186)
at org.mule.runtime.core.internal.connection.DefaultConnectionManager.lambda$testConnectivity$1(DefaultConnectionManager.java:169)
at org.mule.runtime.core.internal.connection.DefaultConnectionManager.doTestConnectivity(DefaultConnectionManager.java:176)
at org.mule.runtime.core.internal.connection.DefaultConnectionManager.testConnectivity(DefaultConnectionManager.java:152)
at org.mule.runtime.core.internal.connection.DelegateConnectionManagerAdapter$EagerConnectionManagerAdapter.testConnectivity(DelegateConnectionManagerAdapter.java:176)
at org.mule.runtime.core.internal.connection.DelegateConnectionManagerAdapter.testConnectivity(DelegateConnectionManagerAdapter.java:98)
at org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationInstance$1.doWork(LifecycleAwareConfigurationInstance.java:204)
at org.mule.runtime.core.api.retry.policy.AbstractPolicyTemplate.execute(AbstractPolicyTemplate.java:62)
at org.mule.runtime.core.internal.retry.async.RetryWorker.run(RetryWorker.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:152)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at com.mulesoft.connectors.cloudhub.internal.CloudHubConnectionProvider.validate(CloudHubConnectionProvider.java:93)
... 20 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1566)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:545)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1217)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1185)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:471)
at org.glassfish.grizzly.ssl.SSLUtils.sslEngineWrap(SSLUtils.java:451)
at org.glassfish.grizzly.ssl.SSLConnectionContext.wrap(SSLConnectionContext.java:360)
at org.glassfish.grizzly.ssl.SSLUtils.handshakeWrap(SSLUtils.java:327)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:724)
at org.glassfish.grizzly.ssl.SSLFilter.doHandshakeStep(SSLFilter.java:345)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:648)
at org.glassfish.grizzly.ssl.SSLBaseFilter.handleRead(SSLBaseFilter.java:349)
at com.ning.http.client.providers.grizzly.SwitchingSSLFilter.handleRead(SwitchingSSLFilter.java:74)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:540)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
... 7 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1729)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:333)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1688)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:1015)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:1012)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1504)
at org.glassfish.grizzly.ssl.SSLUtils.executeDelegatedTask(SSLUtils.java:274)
at org.glassfish.grizzly.ssl.SSLBaseFilter.doHandshakeStep(SSLBaseFilter.java:735)
... 25 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:450)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:289)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1675)
... 33 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
... 39 more
Scenarios I tried:
Running the same code on 4.3.0 runtime on same cloudHub
worker within VPC - IT FAILED.
Running the same code on 4.4.0 (latest) runtime on another
AnyPoint platform's cloudHub worker which is not on VPC - IT WORKED FINE.
Seems like connectivity of cloudHub-notification component running on worker within VPC is causing SSL handshake error. Strange that this component doesn't provide any settings to configure SSL.
Has anyone faced this issue before, if yes - please help me here?