When using wiremock to (ty to) intercept and mock calls via okhttp3, it seems that it either doesn't get the call from okhttp or simply makes a passthrough
I followed the basic setup from the wiremock website and have the following test code:
#WireMockTest
class DummyTest {
#Test
fun `Some other test`(wireMockRuntimeInfo: WireMockRuntimeInfo) {
configureFor("localhost", 8080)
stubFor(get(urlEqualTo("/dummy/path"))
.willReturn(ok("test example")))
val client = OkHttpClient()
val request = Request.Builder()
.url("http://localhost:8080/dummy/path")
.build()
client.newCall(request).execute().let {
val result = it.body()!!.string()
assertEquals("test example", result)
}
}
}
My dependendies looks like this:
dependencies {
implementation("com.squareup.okhttp3:okhttp:3.14.9")
testImplementation(kotlin("test"))
testImplementation("com.github.tomakehurst:wiremock-jre8:2.33.2")
}
And the result I get is this:
Connect to http://localhost:8080 [localhost/127.0.0.1] failed: Connection refused
org.apache.hc.client5.http.HttpHostConnectException: Connect to http://localhost:8080 [localhost/127.0.0.1] failed: Connection refused
at java.base#17.0.4/sun.nio.ch.Net.pollConnect(Native Method)
at java.base#17.0.4/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base#17.0.4/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
at java.base#17.0.4/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
at java.base#17.0.4/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base#17.0.4/java.net.Socket.connect(Socket.java:633)
at app//org.apache.hc.client5.http.socket.PlainConnectionSocketFactory$1.run(PlainConnectionSocketFactory.java:87)
at java.base#17.0.4/java.security.AccessController.doPrivileged(AccessController.java:569)
at app//org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:84)
at app//org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:148)
at app//org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:396)
at app//org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:158)
at app//org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:168)
at app//org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:136)
at app//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at app//org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57)
at app//org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:175)
at app//org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
at app//org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
at app//org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:75)
at app//org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:89)
at app//com.github.tomakehurst.wiremock.client.HttpAdminClient.safelyExecuteRequest(HttpAdminClient.java:506)
at app//com.github.tomakehurst.wiremock.client.HttpAdminClient.executeRequest(HttpAdminClient.java:489)
at app//com.github.tomakehurst.wiremock.client.HttpAdminClient.executeRequest(HttpAdminClient.java:466)
at app//com.github.tomakehurst.wiremock.client.HttpAdminClient.addStubMapping(HttpAdminClient.java:146)
at app//com.github.tomakehurst.wiremock.client.WireMock.register(WireMock.java:414)
at app//com.github.tomakehurst.wiremock.client.WireMock.register(WireMock.java:409)
at app//com.github.tomakehurst.wiremock.client.WireMock.givenThat(WireMock.java:115)
at app//com.github.tomakehurst.wiremock.client.WireMock.stubFor(WireMock.java:119)
at app//DummyTest.Some other test(DummyTest.kt:21)
at java.base#17.0.4/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base#17.0.4/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base#17.0.4/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base#17.0.4/java.lang.reflect.Method.invoke(Method.java:568)
at app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:686)
at app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
at app//org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at app//org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at app//org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at app//org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:205)
at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:201)
at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:137)
at app//org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:71)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base#17.0.4/java.util.ArrayList.forEach(ArrayList.java:1511)
at app//org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base#17.0.4/java.util.ArrayList.forEach(ArrayList.java:1511)
at app//org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at app//org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at app//org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at app//org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at app//org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at app//org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at app//org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
at java.base#17.0.4/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base#17.0.4/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base#17.0.4/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base#17.0.4/java.lang.reflect.Method.invoke(Method.java:568)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at jdk.proxy1/jdk.proxy1.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
For a complete example see the: minimal sample project
It occurs to me that it might be some basic instruction missing on their site, however, I have not been able to find any such missing instructions.
How can I proceed from here?
I'm trying to use client side Encryption with the S3 service, here is my code :
KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");
keyGenerator.init(256);
SecretKey secretKey = keyGenerator.generateKey();
String encodedKey = Base64.getEncoder().encodeToString(secretKey.getEncoded());
System.out.println("Secret Key == " + encodedKey);
return AmazonS3EncryptionClientV2Builder.standard()
.disableChunkedEncoding()
.withKmsClient(AWSKMSClientBuilder.standard().withRegion(Regions.EU_WEST_3).build())
.withEndpointConfiguration(endpointConfiguration)
.withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
.withClientConfiguration(new ClientConfiguration())
.withCryptoConfiguration(new CryptoConfigurationV2().withCryptoMode(CryptoMode.AuthenticatedEncryption))
.withEncryptionMaterialsProvider(new StaticEncryptionMaterialsProvider(new EncryptionMaterials(secretKey)))
.build();
I'm getting this error while trying to download the file that has been encrypted in the S3 bucket.
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
at java.base/com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doFinal(GaloisCounterMode.java:1395) ~[na:na]
at java.base/com.sun.crypto.provider.GaloisCounterMode.engineDoFinal(GaloisCounterMode.java:406) ~[na:na]
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2205) ~[na:na]
at com.amazonaws.services.s3.internal.crypto.keywrap.AesGcmKeyWrapper.unwrapCek(AesGcmKeyWrapper.java:83) ~[aws-java-sdk-s3-1.12.217.jar:na]
... 84 common frames omitted
Any help please ?
I have simple Java client to MQTT Mosquitto server:
public class TestMQTT3 {
public static void main(String[] args) {
System.out.println("Starting");
String serverUrl = "ssl://192.168.1.8:8887";
String path= "C:\\projects\\certs\\CA4\\";
String caFilePath =path+"cacert.pem";
String mqttUserName = "b";
String mqttPassword = "b";
MqttClient client;
try {
client = new MqttClient(serverUrl, "2");
MqttConnectOptions options = new MqttConnectOptions();
options.setUserName(mqttUserName);
options.setPassword(mqttPassword.toCharArray());
options.setConnectionTimeout(60);
options.setKeepAliveInterval(60);
options.setMqttVersion(MqttConnectOptions.MQTT_VERSION_3_1);
SSLSocketFactory socketFactory = getSocketFactory3(caFilePath);
options.setSocketFactory(socketFactory);
System.out.println("starting connect the server...");
client.connect(options);
System.out.println("connected!");
Thread.sleep(1000);
client.subscribe(
"/u/56ca327d17531d08e76bddd4a215e37f5fd6082f7442151c4d3f1d100a0ffd4e",
0);
client.disconnect();
System.out.println("disconnected!");
} catch (MqttException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
private static SSLSocketFactory getSocketFactory3(final String caCrtFile) throws Exception
{
Security.addProvider(new BouncyCastleProvider());
// load CA certificate
X509Certificate caCert = null;
FileInputStream fis = new FileInputStream(caCrtFile);
BufferedInputStream bis = new BufferedInputStream(fis);
CertificateFactory cf = CertificateFactory.getInstance("X.509");
while (bis.available() > 0)
{
caCert = (X509Certificate) cf.generateCertificate(bis);
System.out.println(caCert.toString());
}
// CA certificate is used to authenticate server
KeyStore caKs = KeyStore.getInstance(KeyStore.getDefaultType());
caKs.load(null, null);
caKs.setCertificateEntry("ca-certificate", caCert);
TrustManagerFactory tmf = TrustManagerFactory.getInstance("X509");
tmf.init(caKs);
// finally, create SSL socket factory
SSLContext context = SSLContext.getInstance("TLSv1.2");
context.init(null, tmf.getTrustManagers(), null);
return context.getSocketFactory();
}
}
Trying to connect by using CA certificate:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
f6:7e:ef:1c:70:ef:30:64
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=California, L=Hawthorne, O=PhilNet, OU=UN, CN=CN
Validity
Not Before: Jan 15 17:36:08 2020 GMT
Not After : Jan 14 17:36:08 2021 GMT
Subject: C=US, ST=California, L=Hawthorne, O=PhilNet, OU=UN, CN=CN
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:d2:65:49:db:fc:87:dd:36:8a:e0:27:e6:bb:66:
66:78:35:af:86:ae:d3:e5:e2:07:db:8e:51:f2:67:
06:22:78:99:b3:2f:81:14:a9:e7:28:7e:2a:96:8d:
fb:ec:29:64:39:5b:b7:d4:3c:22:b0:30:18:0d:e4:
c1:38:57:c2:ba:c5:09:11:12:46:8b:cc:06:08:0d:
e4:86:3f:98:e7:61:2d:d8:a9:40:34:e6:87:d7:7d:
c4:7c:62:51:78:b3:fd:d8:a6:1d:15:0b:80:fb:78:
29:59:9c:b7:30:ad:7e:92:f0:bc:94:3c:03:30:c1:
83:28:92:de:34:0b:68:8c:19:6e:d7:29:de:75:23:
59:8c:11:51:a8:84:69:32:d3:96:3d:eb:df:44:7b:
b6:85:2e:f4:af:98:a7:28:84:7b:7d:c9:56:89:66:
e0:e0:3d:63:ae:59:46:23:98:13:bc:af:72:7b:6f:
d5:a6:ec:3e:4e:56:2c:87:f6:70:ab:47:05:a3:4a:
1b:9e:2c:ec:52:7f:3e:7f:b9:a3:33:59:8a:1d:28:
cc:d4:39:0c:8b:f2:12:2d:82:09:63:bd:ae:b3:51:
c6:a4:ac:d3:ab:e4:31:de:b6:b0:11:85:90:36:33:
70:15:94:d0:54:ab:07:bb:a9:6a:63:3e:84:cc:5c:
c3:13
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
75:AF:4E:B9:E0:49:FE:62:C1:21:28:B0:77:38:36:02:22:2E:1B:8F
X509v3 Authority Key Identifier:
keyid:75:AF:4E:B9:E0:49:FE:62:C1:21:28:B0:77:38:36:02:22:2E:1B:8F
X509v3 Subject Alternative Name:
IP Address:192.168.1.8, DNS:glass
Signature Algorithm: sha1WithRSAEncryption
7b:af:79:85:a6:69:3f:00:3c:d3:f9:b7:9f:f6:31:8a:3d:fd:
7f:9e:63:8b:86:de:4e:34:34:11:b3:e1:73:3d:24:8d:06:e5:
ba:a6:91:ba:bc:0a:2f:b1:95:34:4b:c8:cd:cf:6b:31:14:17:
5f:a4:4e:74:8d:07:01:fb:7f:2b:0d:fc:6a:35:85:46:ea:ba:
fc:98:92:de:69:4d:53:f2:c8:99:e9:bd:fa:df:a7:cf:aa:48:
c0:6f:93:ba:cd:79:43:8a:89:8b:e7:bd:99:dd:11:31:c8:5f:
76:a8:fd:99:13:1c:a0:8b:e3:86:72:62:cf:09:7c:de:9d:cf:
db:0b:fc:81:62:1c:32:b0:81:52:58:d9:2f:0b:44:fa:8a:59:
5f:23:b2:01:f0:8e:53:c3:8d:36:1c:25:0b:b5:80:67:95:85:
66:74:8a:08:cc:9c:dc:c1:c6:f6:4b:b7:4b:a3:1a:2d:41:19:
20:7b:54:f0:f1:fe:22:e6:55:7e:14:07:66:77:36:2c:17:ee:
31:33:40:c8:b9:6e:fa:c4:98:86:3a:6f:ba:c0:72:22:75:6c:
5f:4f:94:07:c9:cf:6d:67:61:ff:e9:da:88:95:68:72:78:43:
59:f2:e9:5b:0f:95:d6:7e:33:19:70:23:b4:1e:dd:c9:db:d4:
bc:16:d6:9e
Got error:
starting connect the server...
MqttException (0) - javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:38)
at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:736)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1640)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at org.eclipse.paho.client.mqttv3.internal.SSLNetworkModule.start(SSLNetworkModule.java:149)
at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:722)
... 1 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1622)
... 10 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:159)
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:85)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357)
... 16 more
What I do wrong and how to solve problem?
One cassandra cluster doesn't have SSL enabled and another cassandra cluster has SSL enabled. How to interact with both the cassandra cluster from a single spark job. I have to copy the table from one server(without SSL) and put into another server(with SSL).
Spark job:-
object TwoClusterExample extends App {
val conf = new SparkConf(true).setAppName("SparkCassandraTwoClusterExample")
println("Starting the SparkCassandraLocalJob....")
val sc = new SparkContext(conf)
val connectorToClusterOne = CassandraConnector(sc.getConf.set("spark.cassandra.connection.host", "localhost"))
val connectorToClusterTwo = CassandraConnector(sc.getConf.set("spark.cassandra.connection.host", "remote"))
val rddFromClusterOne = {
implicit val c = connectorToClusterOne
sc.cassandraTable("test","one")
}
{
implicit val c = connectorToClusterTwo
rddFromClusterOne.saveToCassandra("test","one")
}
}
Cassandra conf:-
spark.master spark://ip:6066
spark.executor.memory 1g
spark.cassandra.connection.host remote
spark.cassandra.auth.username iccassandra
spark.cassandra.auth.password pwd1
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.eventLog.enabled true
spark.eventLog.dir /Users/test/logs/spark
spark.cassandra.connection.ssl.enabled true
spark.cassandra.connection.ssl.trustStore.password pwd2
spark.cassandra.connection.ssl.trustStore.path truststore.jks
Submitting the job:-
spark-submit --deploy-mode cluster --master spark://ip:6066 --properties-file cassandra-count.conf --class TwoClusterExample target/scala-2.10/cassandra-table-assembly-1.0.jar
Below error i am getting:-
17/10/26 16:27:20 DEBUG STATES: [/remote:9042] preventing new connections for the next 1000 ms
17/10/26 16:27:20 DEBUG STATES: [/remote:9042] Connection[/remote:9042-1, inFlight=0, closed=true] failed, remaining = 0
17/10/26 16:27:20 DEBUG ControlConnection: [Control connection] error on /remote:9042 connection, no more host to try
com.datastax.driver.core.exceptions.TransportException: [/remote] Cannot connect
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:157)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:140)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:222)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /remote:9042
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:220)
... 6 more
17/10/26 16:27:20 DEBUG Cluster: Shutting down
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.io.IOException: Failed to open native connection to Cassandra at {remote}:9042
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56)
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
at com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120)
at com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:304)
at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.tableDef(CassandraTableRowReaderProvider.scala:51)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.verify(CassandraTableRowReaderProvider.scala:146)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:143)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD.count(RDD.scala:1143)
at cassandraCount$.runJob(cassandraCount.scala:27)
at cassandraCount$delayedInit$body.apply(cassandraCount.scala:22)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at cassandraCount$.main(cassandraCount.scala:10)
at cassandraCount.main(cassandraCount.scala)
... 6 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /remote:9042 (com.datastax.driver.core.exceptions.TransportException: [/remote] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:231)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1414)
at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:393)
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
... 37 more
17/10/26 16:27:23 INFO SparkContext: Invoking stop() from shutdown hook
17/10/26 16:27:23 INFO SparkUI: Stopped Spark web UI at http://10.7.10.138:4040
Working code:-
object ClusterSSLTest extends App{
val conf = new SparkConf(true).setAppName("sparkCassandraLocalJob")
println("Starting the ClusterSSLTest....")
val sc = new SparkContext(conf)
val sourceCluster = CassandraConnector(
sc.getConf.set("spark.cassandra.connection.host", "localhost"))
val destinationCluster = CassandraConnector(
sc.getConf.set("spark.cassandra.connection.host", "remoteip1,remoteip2")
.set("spark.cassandra.auth.username","uname")
.set("spark.cassandra.auth.password","pwd")
.set("spark.cassandra.connection.ssl.enabled","true")
.set("spark.cassandra.connection.timeout_ms","10000")
.set("spark.cassandra.connection.ssl.trustStore.path", "../truststore.jks")
.set("spark.cassandra.connection.ssl.trustStore.password", "pwd")
.set("spark.cassandra.connection.ssl.trustStore.type", "JKS")
.set("spark.cassandra.connection.ssl.protocol", "TLS")
.set("spark.cassandra.connection.ssl.enabledAlgorithms", "TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA")
)
val rddFromSourceCluster = {
implicit val c = sourceCluster
val tbRdd = sc.cassandraTable("analytics","products")
println(s"no of rows ${tbRdd.count()}")
tbRdd
}
val rddToDestinationCluster = {
implicit val c = destinationCluster // connect to source cluster in this code block.
rddFromSourceCluster.saveToCassandra("analytics","products")
}
sc.stop()
}
I have read though many different examples but I am currently having difficulties trying to communicate via a proxy using HTTPS. I have a wrapper to create a Apache HttpClient as seen in the code below.
Currently if I make my call without setting up a proxy it will use my truststore from the SSLSocketFactory and correctly allow the communication via SSL. The only certificate required is a verisign server certificate which does not require authentication.
When I setup a proxy I get an error saying:
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
I feel that I must be missing some type of proxy setup which makes the proxy connection use the same SSLSocketFactory?
I tested with -Djavax.net.debug=ssl and I can see a lot more SSL activity when going direct. When I use direct I can see all the keys loaded and sent with the request, when I use the proxy I only see:
httpConnector.receiver.3, setSoTimeout(30000) called
%% No cached client session
*** ClientHello, TLSv1
RandomCookie: GMT: 1307565311 bytes = { 184, 216, 5, 151, 154, 212, 232, 96, 69, 73, 240, 54, 236, 26, 8, 45, 109, 9, 192,
227, 193, 58, 129, 212, 57, 249, 205, 56 }
Session ID: {}
Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_C
BC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH
_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH
_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
]
Compression Methods: { 0 }
***
httpConnector.receiver.3, WRITE: TLSv1 Handshake, length = 73
httpConnector.receiver.3, WRITE: SSLv2 client hello message, length = 98
httpConnector.receiver.3, handling exception: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
httpConnector.receiver.3, SEND TLSv1 ALERT: fatal, description = unexpected_message
httpConnector.receiver.3, WRITE: TLSv1 Alert, length = 2
httpConnector.receiver.3, called closeSocket()
httpConnector.receiver.3, IOException in getSession(): javax.net.ssl.SSLException: Unrecognized SSL message, plaintext conn
ection?
httpConnector.receiver.3, called close()
httpConnector.receiver.3, called closeInternal(true)
httpConnector.receiver.3, called close()
httpConnector.receiver.3, called closeInternal(true)
2011-12-20 11:11:59,401 [httpConnector.receiver.3] INFO - The JavaScript method AddEvent threw an exception of type class co
m.alarmpoint.integrationagent.soap.exception.SOAPRequestException with message "javax.net.ssl.SSLPeerUnverifiedException: pe
er not authenticated". The exception will be propogated up the call stack.
Can anyone help out please. Here is my code for setting up the proxy and SSLSocketFactory.
var client = httpClientWrapper.getHttpClient();
var proxy = new HttpHost(PROXY_HOST, PROXY_PORT, "https");
client.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
var authpref = new ArrayList();
authpref.add(AuthPolicy.BASIC);
client.getParams().setParameter(AuthPNames.PROXY_AUTH_PREF, authpref);
ServiceAPI.getLogger().debug("KeyStore.getDefaultType() " + KeyStore.getDefaultType());
var trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
var instream = new FileInputStream(new File("conf/my.truststore"));
try {
ServiceAPI.getLogger().debug("getting trustore");
trustStore.load(instream, "changeit".split(''));
} finally {
instream.close();
}
var socketFactory = new SSLSocketFactory(trustStore);
var sch = new Scheme("https", socketFactory, 443);
client.getConnectionManager().getSchemeRegistry().register(sch);
Stack trace:
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(Unknown Source)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:390)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:488)
at org.apache.http.conn.scheme.SchemeSocketFactoryAdaptor.connectSocket(SchemeSocketFactoryAdaptor.java:62)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:561)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
Here's a variant on aaron's solution, in Java (vs Groovy). This solution also avoids the HttpClientWrapper class (where does that come from?), and loads the proxy's certificate directly. It is written against HttpClient 4.2 (but I think it should work with 4.0). As an added bonus it includes an example of proxy authentication for a Windows proxy such as Microsoft ForeFront TMG.
It took me long enough to piece this together that I figured I should share it:
HttpParams params = new BasicHttpParams();
DefaultHttpClient.setDefaultHttpParams( params ); // Add the default parameters to the parameter set we're building
DefaultHttpClient client = new DefaultHttpClient( params );
KeyStore trustStore = KeyStore.getInstance( KeyStore.getDefaultType() );
trustStore.load( null );
InputStream certStream = new FileInputStream( "cert-file" );
CertificateFactory cf = CertificateFactory.getInstance("X.509");
X509Certificate cert = (X509Certificate)cf.generateCertificate(certStream);
certStream.close();
trustStore.setCertificateEntry( "proxy-cert", cert );
SSLSocketFactory socketFactory = new SSLSocketFactory(trustStore);
client.getConnectionManager().getSchemeRegistry().register( new Scheme( "https", 443, socketFactory ));
client.getParams().setParameter( ConnRoutePNames.DEFAULT_PROXY,
new HttpHost( "my-proxy", 8080 ));
// These 3 lines are only needed if your proxy is Windows based & requires authentication
AuthScope scope = new AuthScope( "myproxy", 8080, null, AuthPolicy.NTLM );
Credentials credentials = new NTCredentials( "username", "changeit", "WORKSTATION", "MY-DOMAIN" );
client.getCredentialsProvider().setCredentials( scope, credentials );
HttpGet get = new HttpGet( "https://mysite.com/resource" );
String result = client.execute( get, new BasicResponseHandler() );
System.out.println( result );
I solved this. The problem which I found once debugging into HttpClients code was the way my proxy was configured and the scheme's available.
HttpRoute[{tls}->https://someproxy->https://some_endpoint:443]
The problem was that the proxy was setup for https scheme but it was actually running on http. This became a problem when the wrapper did not configure a http scheme. In the end I created the SSLSocketFactory for my truststore and a default http scheme and setup my proxy correctly.
// Setup the Keystore and Schemes for the HttpClient and Proxy
var trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
var instream = new FileInputStream(new File("conf/my.truststore"));
try {
trustStore.load(instream, "changeit".split(''));
} finally {
instream.close();
}
var socketFactory = new SSLSocketFactory(trustStore);
var schHttp = new Scheme("http", PROXY_PORT, PlainSocketFactory.getSocketFactory());
// Create the HttpClient wrapper which will have the truststore SSLSocketFactory and a default http scheme and proxy setup
httpClientWrapper = new HttpClientWrapper("some_endpoint", 443, "/", socketFactory);
var client = httpClientWrapper.getHttpClient();
var proxy = new HttpHost(PROXY_HOST, PROXY_PORT, "http");
client.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
client.getConnectionManager().getSchemeRegistry().register(schHttp);
Have you tried using the global -Dhttp.proxyHost=proxy.host.com -Dhttp.proxyPort=8080 when launching your java process to verify that the SSLSocketFactory isn't falling back to proxyless communications.