AEADBadTagException: Tag mismatch using AWS SDK - amazon-s3

I'm trying to use client side Encryption with the S3 service, here is my code :
KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");
keyGenerator.init(256);
SecretKey secretKey = keyGenerator.generateKey();
String encodedKey = Base64.getEncoder().encodeToString(secretKey.getEncoded());
System.out.println("Secret Key == " + encodedKey);
return AmazonS3EncryptionClientV2Builder.standard()
.disableChunkedEncoding()
.withKmsClient(AWSKMSClientBuilder.standard().withRegion(Regions.EU_WEST_3).build())
.withEndpointConfiguration(endpointConfiguration)
.withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
.withClientConfiguration(new ClientConfiguration())
.withCryptoConfiguration(new CryptoConfigurationV2().withCryptoMode(CryptoMode.AuthenticatedEncryption))
.withEncryptionMaterialsProvider(new StaticEncryptionMaterialsProvider(new EncryptionMaterials(secretKey)))
.build();
I'm getting this error while trying to download the file that has been encrypted in the S3 bucket.
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
at java.base/com.sun.crypto.provider.GaloisCounterMode$GCMDecrypt.doFinal(GaloisCounterMode.java:1395) ~[na:na]
at java.base/com.sun.crypto.provider.GaloisCounterMode.engineDoFinal(GaloisCounterMode.java:406) ~[na:na]
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2205) ~[na:na]
at com.amazonaws.services.s3.internal.crypto.keywrap.AesGcmKeyWrapper.unwrapCek(AesGcmKeyWrapper.java:83) ~[aws-java-sdk-s3-1.12.217.jar:na]
... 84 common frames omitted
Any help please ?

Related

OkHttp3 - unable to find valid certification path to requested target

In my app I use retrofit2(2.9.0) with OkHttp3(3.14.4). I want to add tls client certificate to all of my requests to some api. I've got the certificate in a .p12 file. I read the file, loaded into X509Certificate class then I used the .addTrustedCertificate(certificate) method. The certificate is correct I tried it using curl. Unfortunately when I run the code I get an exception.
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:641)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:460)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:177)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:213)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
at okhttp3.RealCall.execute(RealCall.java:81)
at retrofit2.OkHttpCall.execute(OkHttpCall.java:204)
at com.jakewharton.retrofit2.adapter.reactor.ExecuteSinkConsumer.accept(ExecuteSinkConsumer.java:39)
at com.jakewharton.retrofit2.adapter.reactor.ExecuteSinkConsumer.accept(ExecuteSinkConsumer.java:24)
at reactor.core.publisher.FluxCreate.subscribe(FluxCreate.java:94)
at reactor.core.publisher.Flux.subscribe(Flux.java:8143)
at com.jakewharton.retrofit2.adapter.reactor.BodyFlux.subscribe(BodyFlux.java:36)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:625)
... 47 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
... 53 more
Process finished with exit code 1
my code looks like this
val fileName = "certyficate.p12"
val file = File(fileName)
val inputStream = file.inputStream()
val p12 = KeyStore.getInstance("pkcs12")
val password = "password"
p12.load(inputStream, password.toCharArray())
val alias = p12.aliases().nextElement()
val certificate = p12.getCertificate(alias) as X509Certificate
val ob = ObjectMapper().registerModule(KotlinModule())
val hostname = "https://myapi.com"
val timeout = 5L
val okHttpClient = OkHttpClient.Builder()
.connectTimeout(timeout, TimeUnit.SECONDS)
.readTimeout(timeout, TimeUnit.SECONDS)
.writeTimeout(timeout, TimeUnit.SECONDS)
.apply {
addInterceptor(HttpLoggingInterceptor().apply {
level = HttpLoggingInterceptor.Level.BODY
})
}
.apply {
val clientCertificate = HandshakeCertificates
.Builder()
.addTrustedCertificate(certificate)
.build()
sslSocketFactory(clientCertificate.sslSocketFactory(), clientCertificate.trustManager())
}
.build()
val retrofit = Retrofit.Builder()
.baseUrl(hostname)
.addCallAdapterFactory(RxJava2CallAdapterFactory.createWithScheduler(
io.reactivex.schedulers.Schedulers.io()
))
.addCallAdapterFactory(ReactorCallAdapterFactory.createWithScheduler(Schedulers.newElastic(
MyApi::class.java.name
)))
.addConverterFactory(JacksonConverterFactory.create(ob))
.client(okHttpClient)
.build()
val p = retrofit.create(MyApi::class.java)
val res = p.doRequest().block()
println(res)
What's a possible reason? how to correctly use client certificates when using OkHttp3 ?
EDIT after I added .addPlatformTrustedCertificates() I get
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:180)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:336)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.logging.HttpLoggingInterceptor.intercept(HttpLoggingInterceptor.java:213)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
at okhttp3.RealCall.execute(RealCall.java:81)
at retrofit2.OkHttpCall.execute(OkHttpCall.java:204)
at com.jakewharton.retrofit2.adapter.reactor.ExecuteSinkConsumer.accept(ExecuteSinkConsumer.java:39)
at com.jakewharton.retrofit2.adapter.reactor.ExecuteSinkConsumer.accept(ExecuteSinkConsumer.java:24)
at reactor.core.publisher.FluxCreate.subscribe(FluxCreate.java:94)
at reactor.core.publisher.Flux.subscribe(Flux.java:8143)
at com.jakewharton.retrofit2.adapter.reactor.BodyFlux.subscribe(BodyFlux.java:36)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
I run with javax.net.debug=ssl,handshake and found in my logs
javax.net.ssl|DEBUG|12|my.app.MyApi-2|2021-03-03 14:00:09.656 CET|CertificateMessage.java:290|No X.509 certificate for client authentication, use empty Certificate message instead
javax.net.ssl|DEBUG|12|my.app.MyApi-2|2021-03-03 14:00:09.656 CET|CertificateMessage.java:321|Produced client Certificate handshake message (
"Certificates": <empty list>
)
also my build.gradle
compile group: 'com.squareup.retrofit2', name: 'retrofit', version: '2.9.0'
compile group: 'com.squareup.retrofit2', name: 'converter-jackson', version: '2.9.0'
compile group: 'com.squareup.retrofit2', name: 'adapter-rxjava2', version: '2.9.0'
compile group: 'com.jakewharton.retrofit', name: 'retrofit2-reactor-adapter', version: '2.1.0'
compile group: 'com.squareup.okhttp3', name: 'okhttp-tls', version: '3.14.4'
val p12 = KeyStore.getInstance("pkcs12").apply {
load(inputStream, password)
}
val clientCert = p12.getCertificateChain(alias)[0] as X509Certificate
val rootCert = p12.getCertificateChain(alias)[1] as X509Certificate
val clientPrivateKey = p12.getKey(alias, password) as PrivateKey
val clientTLSCredentials = HeldCertificate(KeyPair(clientCert.publicKey, clientPrivateKey), clientCert)
val clientTlsHandShake = HandshakeCertificates.Builder()
.heldCertificate(clientTLSCredentials, rootCert)
.addPlatformTrustedCertificates()
.build()
val client = OkHttpClient.Builder()
.sslSocketFactory(clientTlsHandShake.sslSocketFactory(), clientTlsHandShake.trustManager())
.build()
This should be working, see for an example https://github.com/square/okhttp/blob/083315a473d8b2fc972a70449272b296683ac38c/android-test/src/androidTest/java/okhttp/android/test/letsencrypt/LetsEncryptClientTest.kt
val handshakeCertificates = HandshakeCertificates.Builder()
.addPlatformTrustedCertificates()
.addTrustedCertificate(cert)
.build()
clientBuilder
.sslSocketFactory(handshakeCertificates.sslSocketFactory(),
handshakeCertificates.trustManager)
You should can look at the certificate chain in SunCertPathBuilderException in a debugger.
This is a subclass of the generic CertPathBuilderException. It contains an adjacency list with information regarding the unsuccessful paths that the SunCertPathBuilder tried.
Ultimately you may need to enable SSL debugging in JSSE to see why it is failing.
Was able to get this working using Clyde Barrow's solution. Here is my pure Java solution (not much different than Clyde's). This is using a client certificate signed by Entrust and an intermediate certificate.
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
keyStore.load(new FileInputStream(new File("myKeyStore.jks")), "mypassword".toCharArray());
Certificate[] certificates = keyStore.getCertificateChain("myalias");
X509Certificate clientCertificate = (X509Certificate) certificates[0];
X509Certificate intermediateCertificate = (X509Certificate) certificates[1];
PrivateKey privateKey = (PrivateKey) keyStore.getKey("myalias", "mypassword".toCharArray());
PublicKey publicKey = clientCertificate.getPublicKey();
KeyPair keyPair = new KeyPair(publicKey, privateKey);
HeldCertificate heldCertificate = new HeldCertificate(keyPair, clientCertificate);
HandshakeCertificates.Builder builder =
new HandshakeCertificates.Builder()
.heldCertificate(heldCertificate, intermediateCertificate)
.addPlatformTrustedCertificates();
HandshakeCertificates handshakeCertificates = builder.build();
OkHttpClient okClient =
new OkHttpClient().newBuilder().readTimeout(30, TimeUnit.SECONDS)
.sslSocketFactory(handshakeCertificates.sslSocketFactory(), handshakeCertificates.trustManager())
.build();

Client authentifcation for EJB to Wildfly12

I want to try connect from a client to an ejb. I use WildFly12 and Java8.
Before with Wildfly8 all works fine.
I remove to local user authentification of standalone.xml to get the same error at server side.
Error from server.log
Caused by: org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried:
JBOSS-LOCAL-USER: org.wildfly.security.auth.AuthenticationException: JBREM000300: Authentication failed due to I/O error
at org.jboss.remoting3.ConnectionPeerIdentityContext.doAuthenticate(ConnectionPeerIdentityContext.java:368)
at org.jboss.remoting3.ConnectionPeerIdentityContext.authenticate(ConnectionPeerIdentityContext.java:174)
at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:505)
at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:494)
at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
at org.xnio.IoUtils$2.execute(IoUtils.java:71)
at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693)
at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132)
at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570)
at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:494)
at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:433)
at org.jboss.remoting3.UncloseableEndpoint.getConnectedIdentity(UncloseableEndpoint.java:51)
at org.jboss.remoting3.Endpoint.getConnectedIdentity(Endpoint.java:122)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver.lambda$getConnection$2(RemoteEJBReceiver.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver.getConnection(RemoteEJBReceiver.java:185)
I create InitialContextas follow:
clientProperties.put( Context.INITIAL_CONTEXT_FACTORY, "org.wildfly.naming.client.WildFlyInitialContextFactory" );
clientProperties.put( Context.PROVIDER_URL, "remote+http://" + m_strAppServerHostname + ":" + m_iAppServerPort );
clientProperties.put( Context.SECURITY_PRINCIPAL, getAppServerUserName() );
clientProperties.put( Context.SECURITY_CREDENTIALS, getAppServerPassword() );
return new InitialContext( clientProperties );

Lookup Hbase Tbl from UDF (Beeline , Hbase,Delegation Tokens)

I have a requirement to write Custom UDF for data lookup from Hbase Table .
NOTE : I have done Unit Testing with HIVE . It seems to be working .
But when I use the same UDF Beeline, Its failed . By default Cloudera restricts impersonation and allows only hive user for running queries in Beeline. On Job startup , YarnChild is setting the following delegations tokens.
I want to add token (Kind : HBASE_AUTH_TOKEN ) for dealing with Hbase.
Kind: mapreduce.job
Kind: HDFS_DELEGATION_TOKEN
Kind: kms-dt
I researched and found out how HbaseStorageHandler is using Delegation Token ( i.e HBASE_AUTH_TOKEN ) for Hbase . So I used the same set of functions and its not working either .
Functions from HbasestorageHandler (to obtain tokens to Job ) :
private void addHBaseDelegationToken(Configuration conf, JobConf jconf) throws IOException {
if (User.isHBaseSecurityEnabled(conf)) {
try {
logger.info("isHbaseSecurityEnabled :True ");
User e = User.getCurrent();
logger.info("isHbaseSecurityEnabled :User ==> " + e.toString());
Token authToken = getAuthToken(conf, e);
logger.info("isHbaseSecurityEnabled :AuthToken==> "+authToken.toString());
Job job = new Job(conf);
if(authToken == null) {
UserGroupInformation ugi = UserGroupInformation.getLoginUser();
ugi.setAuthenticationMethod(UserGroupInformation.AuthenticationMethod.KERBEROS);
e.obtainAuthTokenForJob(jconf);
} else {
logger.info("authToken is not null"+authToken.toString());
job.getCredentials().addToken(authToken.getService(), authToken);
}
logger.info("obtained Token /....");
} catch (InterruptedException var5) {
throw new IOException("Error while obtaining hbase delegation token", var5);
}
}
}
private static Token<AuthenticationTokenIdentifier> getAuthToken(Configuration conf, User user) throws IOException, InterruptedException {
ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, "mr-init-credentials", (Abortable) null);
Token var4;
try {
String e = ZKClusterId.readClusterIdZNode(zkw);
logger.info("====== clusterID : " + e);
var4 = (new AuthenticationTokenSelector()).selectToken(new Text(e), user.getUGI().getTokens());
if (var4 == null) {
logger.info("var4 is null===========================");
} else {
logger.info("====== Hbase Token : " + var4.toString());
}
} catch (KeeperException var8) {
throw new IOException(var8);
} catch (NullPointerException np) {
return null;
} finally {
zkw.close();
}
return var4;
}
After calling addHBaseDelegationToken() in configure() of UDF. I am getting the following exception .I am not sure How I can make hvie user to talk with Hbase as hive.keytab is handled by Cloudera and its secured.
Any Inputs might be helpful. Thanks !
Exception StackTrace :
2018-10-11 04:48:07,625 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hive (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2018-10-11 04:48:07,627 WARN [main] org.apache.hadoop.hbase.ipc.RpcClientImpl: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2018-10-11 04:48:07,628 FATAL [main] org.apache.hadoop.hbase.ipc.RpcClientImpl: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:181)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:618)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:163)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:744)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:741)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:741)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:907)
at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:874)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1246)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:34118)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1633)
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:104)
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:94)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:107)
at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)
at org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
at org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:111)
at org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:108)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:340)
at org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:108)
at com.barclaycardus.hadoop.utils.udfs.HbaseTblLookupUDF.configure(HbaseTblLookupUDF.java:131)
at org.apache.hadoop.hive.ql.exec.MapredContext.setup(MapredContext.java:120)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:143)
at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:980)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:63)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:196)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 66 more
Already Tried below options :
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java
https://github.com/ibm-research-ireland/sparkoscope/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HBaseCredentialProvider.scala

Path has type Object rather than String

I am implementing SSL connection for my web application. I have keystore which i am passing while running SBT. But i am getting error "PATH HAS TYPE OBJECT RATHER THAN STRING".
This way i am passing keystore :-
run -Dhttp.port=disabled -Dhttps.port=9448 -Dhttps.keyStore.path="certs\example.com.jks" -Dhttps.keyStore.type=JKS -Dhttps.keyStore.password=changeit
I am getting below error :-
error] p.c.s.AkkaHttpServer - Cannot load SSL context
ava.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at play.core.server.ssl.ServerSSLEngine$.createScalaSSLEngineProvider(ServerSSLEngine.scala:103)
at play.core.server.ssl.ServerSSLEngine$.createSSLEngineProvider(ServerSSLEngine.scala:35)
at play.core.server.AkkaHttpServer.$anonfun$httpsServerBinding$1(AkkaHttpServer.scala:126)
at play.core.server.AkkaHttpServer.$anonfun$httpsServerBinding$1$adapted(AkkaHttpServer.scala:124)
at scala.Option.map(Option.scala:146)
at play.core.server.AkkaHttpServer.<init>(AkkaHttpServer.scala:124)
aused by: com.typesafe.config.ConfigException$WrongType: system properties: path has type OBJECT rather than STRI
at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:159)
at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:170)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:184)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:189)
at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:246)
at play.core.server.ssl.DefaultSSLEngineProvider.createSSLContext(DefaultSSLEngineProvider.scala:34)
at play.core.server.ssl.DefaultSSLEngineProvider.<init>(DefaultSSLEngineProvider.scala:24)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Please tell me if anything wrong in the above.
The Exception suggest, that one of your path is wrong.
Here is the code, the exception happened:
def createSSLContext(applicationProvider: ApplicationProvider): SSLContext = {
val httpsConfig = serverConfig.configuration.underlying.getConfig("play.server.https")
val keyStoreConfig = httpsConfig.getConfig("keyStore")
val keyManagerFactory: KeyManagerFactory = if (keyStoreConfig.hasPath("path")) {
val path = keyStoreConfig.getString("path") // HERE EXACTLY
As you can see it expects a path, called: play.server.https.keyStore.path.
It seems that you have configured that path wrong: Check your application.conf.
Or adjust your run command.

How to handle Spark with multiple cassandra server with different ssl policy

One cassandra cluster doesn't have SSL enabled and another cassandra cluster has SSL enabled. How to interact with both the cassandra cluster from a single spark job. I have to copy the table from one server(without SSL) and put into another server(with SSL).
Spark job:-
object TwoClusterExample extends App {
val conf = new SparkConf(true).setAppName("SparkCassandraTwoClusterExample")
println("Starting the SparkCassandraLocalJob....")
val sc = new SparkContext(conf)
val connectorToClusterOne = CassandraConnector(sc.getConf.set("spark.cassandra.connection.host", "localhost"))
val connectorToClusterTwo = CassandraConnector(sc.getConf.set("spark.cassandra.connection.host", "remote"))
val rddFromClusterOne = {
implicit val c = connectorToClusterOne
sc.cassandraTable("test","one")
}
{
implicit val c = connectorToClusterTwo
rddFromClusterOne.saveToCassandra("test","one")
}
}
Cassandra conf:-
spark.master spark://ip:6066
spark.executor.memory 1g
spark.cassandra.connection.host remote
spark.cassandra.auth.username iccassandra
spark.cassandra.auth.password pwd1
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.eventLog.enabled true
spark.eventLog.dir /Users/test/logs/spark
spark.cassandra.connection.ssl.enabled true
spark.cassandra.connection.ssl.trustStore.password pwd2
spark.cassandra.connection.ssl.trustStore.path truststore.jks
Submitting the job:-
spark-submit --deploy-mode cluster --master spark://ip:6066 --properties-file cassandra-count.conf --class TwoClusterExample target/scala-2.10/cassandra-table-assembly-1.0.jar
Below error i am getting:-
17/10/26 16:27:20 DEBUG STATES: [/remote:9042] preventing new connections for the next 1000 ms
17/10/26 16:27:20 DEBUG STATES: [/remote:9042] Connection[/remote:9042-1, inFlight=0, closed=true] failed, remaining = 0
17/10/26 16:27:20 DEBUG ControlConnection: [Control connection] error on /remote:9042 connection, no more host to try
com.datastax.driver.core.exceptions.TransportException: [/remote] Cannot connect
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:157)
at com.datastax.driver.core.Connection$1.operationComplete(Connection.java:140)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:222)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /remote:9042
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:220)
... 6 more
17/10/26 16:27:20 DEBUG Cluster: Shutting down
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.io.IOException: Failed to open native connection to Cassandra at {remote}:9042
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)
at com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.scala:56)
at com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraConnector.scala:81)
at com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:109)
at com.datastax.spark.connector.cql.CassandraConnector.withClusterDo(CassandraConnector.scala:120)
at com.datastax.spark.connector.cql.Schema$.fromCassandra(Schema.scala:304)
at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.tableDef(CassandraTableRowReaderProvider.scala:51)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef$lzycompute(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.tableDef(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableRowReaderProvider$class.verify(CassandraTableRowReaderProvider.scala:146)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.verify(CassandraTableScanRDD.scala:59)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.getPartitions(CassandraTableScanRDD.scala:143)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD.count(RDD.scala:1143)
at cassandraCount$.runJob(cassandraCount.scala:27)
at cassandraCount$delayedInit$body.apply(cassandraCount.scala:22)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at cassandraCount$.main(cassandraCount.scala:10)
at cassandraCount.main(cassandraCount.scala)
... 6 more
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /remote:9042 (com.datastax.driver.core.exceptions.TransportException: [/remote] Cannot connect))
at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:231)
at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:77)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1414)
at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:393)
at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
... 37 more
17/10/26 16:27:23 INFO SparkContext: Invoking stop() from shutdown hook
17/10/26 16:27:23 INFO SparkUI: Stopped Spark web UI at http://10.7.10.138:4040
Working code:-
object ClusterSSLTest extends App{
val conf = new SparkConf(true).setAppName("sparkCassandraLocalJob")
println("Starting the ClusterSSLTest....")
val sc = new SparkContext(conf)
val sourceCluster = CassandraConnector(
sc.getConf.set("spark.cassandra.connection.host", "localhost"))
val destinationCluster = CassandraConnector(
sc.getConf.set("spark.cassandra.connection.host", "remoteip1,remoteip2")
.set("spark.cassandra.auth.username","uname")
.set("spark.cassandra.auth.password","pwd")
.set("spark.cassandra.connection.ssl.enabled","true")
.set("spark.cassandra.connection.timeout_ms","10000")
.set("spark.cassandra.connection.ssl.trustStore.path", "../truststore.jks")
.set("spark.cassandra.connection.ssl.trustStore.password", "pwd")
.set("spark.cassandra.connection.ssl.trustStore.type", "JKS")
.set("spark.cassandra.connection.ssl.protocol", "TLS")
.set("spark.cassandra.connection.ssl.enabledAlgorithms", "TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA")
)
val rddFromSourceCluster = {
implicit val c = sourceCluster
val tbRdd = sc.cassandraTable("analytics","products")
println(s"no of rows ${tbRdd.count()}")
tbRdd
}
val rddToDestinationCluster = {
implicit val c = destinationCluster // connect to source cluster in this code block.
rddFromSourceCluster.saveToCassandra("analytics","products")
}
sc.stop()
}