Apache VFS - FTPS support for TLSv1 - ssl

Please can someone tell me if Apache VFS 2.6.0 will support FTPS at TLSv1 ?
The reason I ask is because we have written an application using VFS and developed it against VsFTPD 3.0.2
Now we have moved into testing we are seeing a lot of old machines using VsFTPD 2.2.2 with only TLSv1 support.
Exceptions (don't give much away)
org.apache.commons.vfs2.FileSystemException: Could not connect to FTP server on "100.200.150.25".
at org.apache.commons.vfs2.provider.ftp.FtpClientFactory$ConnectionFactory.createConnection(FtpClientFactory.java:220)
at org.apache.commons.vfs2.provider.ftps.FtpsClientFactory.createConnection(FtpsClientFactory.java:57)
at org.apache.commons.vfs2.provider.ftps.FtpsClientWrapper.createClient(FtpsClientWrapper.java:47)
at org.apache.commons.vfs2.provider.ftp.FTPClientWrapper.createClient(FTPClientWrapper.java:97)
at org.apache.commons.vfs2.provider.ftp.FTPClientWrapper.getFtpClient(FTPClientWrapper.java:146)
at org.apache.commons.vfs2.provider.ftp.FTPClientWrapper.<init>(FTPClientWrapper.java:52)
at org.apache.commons.vfs2.provider.ftps.FtpsClientWrapper.<init>(FtpsClientWrapper.java:41)
at org.apache.commons.vfs2.provider.ftps.FtpsFileProvider.doCreateFileSystem(FtpsFileProvider.java:49)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:93)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:72)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:56)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:717)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:654)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:750)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at org.springframework.cloud.sleuth.instrument.async.TraceAsyncAspect.traceBackgroundThread(TraceAsyncAspect.java:67)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:70)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:180)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
at org.apache.commons.net.ftp.FTPSClient.sslNegotiation(FTPSClient.java:289)
at org.apache.commons.net.ftp.FTPSClient._connectAction_(FTPSClient.java:226)
at org.apache.commons.net.SocketClient._connect(SocketClient.java:244)
at org.apache.commons.net.SocketClient.connect(SocketClient.java:202)
at org.apache.commons.vfs2.provider.ftp.FtpClientFactory$ConnectionFactory.createConnection(FtpClientFactory.java:163)

Apache VFS does support TLSv1
The issue here was in the commonly seen and equally vague Exception
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
Java 11 has a set of disabled Ciphers which are used to encrypt for TLS
The one I found was being used by VSFTPD 2.2.2 is DES-CBC3-SHA
The one I found disabled by Java 11 in conf/java.security was 3DES_EDE_CBC
I removed it from the disabled list and now the connection works fine.
PLEASE NOTE
I will be advising services to upgrade their VSFTPD servers. I don't like downgrading security but money talks and I suspect this smaller/cheaper change will be implemented.

Related

Sync settings over proxy - Failed to Sync IDE settings

I can't sync my IntelliJ IDEA settings over a proxy.
When I tried to sync, I have an error :
My logs show an exception with signature :
2021-09-02 10:00:15,784 [1760431] INFO - cloudConfig.CloudConfigManager - === Start.updateIO ===
2021-09-02 10:00:15,784 [1760431] INFO - cloudConfig.CloudConfigManager - === updateIO ===
2021-09-02 10:00:15,904 [1760551] INFO - cloudConfig.CloudConfigManager - java.security.SignatureException: Signature does not match.
javax.net.ssl.SSLHandshakeException: java.security.SignatureException: Signature does not match.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1974)
at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1969)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1968)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
at com.jetbrains.cloudconfig.AbstractHttpClient.download(AbstractHttpClient.java:94)
at com.jetbrains.cloudconfig.CloudConfigFileClient.list(CloudConfigFileClient.java:192)
at com.intellij.idea.cloudConfig.ETagCloudConfigFileClient.list(ETagCloudConfigFileClient.java:36)
at com.intellij.cloudConfig.CloudConfigManager.listFiles(CloudConfigManager.java:2659)
at com.intellij.cloudConfig.CloudConfigManager.lambda$updateIO$15(CloudConfigManager.java:764)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.SignatureException: Signature does not match.
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:197)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
at com.jetbrains.cloudconfig.AbstractHttpClient.download(AbstractHttpClient.java:92)
... 11 more
Caused by: java.security.cert.CertificateException: java.security.SignatureException: Signature does not match.
at com.jetbrains.b.a.L.a(L.java:496)
at com.jetbrains.b.a.L.checkServerTrusted(L.java:483)
at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1510)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
... 28 more
Caused by: java.security.SignatureException: Signature does not match.
at java.base/sun.security.x509.X509CertImpl.verify(X509CertImpl.java:422)
at java.base/sun.security.x509.X509CertImpl.verify(X509CertImpl.java:390)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.jetbrains.b.a.L.a(L.java:489)
... 31 more
2021-09-02 10:00:15,906 [1760553] INFO - cloudConfig.CloudConfigManager - === End.updateIO ===
My settings
I use IntelliJ-IDEA 2021.2 (Ultimate Edition) over an enterprise proxy.
My enterprise proxy is defined in settings and the check connection is successful :
My enterprise self-signed certificate is accepted :
I can successfully display and install plugins from marketplace.
This is my first question, feel free to comment if I can improve it.
Your proxy server substitutes the remote ssl certificate. You should either exclude the proxy for JetBrains account or add your custom certificate that your proxy uses into the IDE JDK which IDE uses to run. This bundled JDK is located in <IDE HOME>/jbr directory.
See Import the Certificate as a Trusted Certificate about steps how to import the certificate into the JDK.
Please note that you must be sure what you are doing when importing custom certificate. You may want to contact your IT department about this.

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure in WAS 8.5.5.9

We have a single installation of WAS(8.5.5.9),with java of 1.8_64 and TLSv1.2 enabled on JVM. But when I run the code sslSocket.getEnabledProtocols(), it gives me `supported protocols=[TLSv1], which is V1.0.
When I check in Security > SSL certificate and key management, and under Related Items, click SSL configurations. ( such as CellDefaultSSLsetting , NodedefaultSSLsetting and any other SSLConfig), it shows 'TLSv1.2' as the protocol, under QoP settings.
To give you some background, am trying to connect to a dropwizard application which is configured with supported protocols as TLSv1.2 from an application deployed on WAS.
Since it tries to connect via TLS1, it gives javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure.
Error:
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at com.ibm.jsse2.j.a(j.java:31)
at com.ibm.jsse2.j.a(j.java:43)
at com.ibm.jsse2.as.b(as.java:816)
at com.ibm.jsse2.as.a(as.java:752)
at com.ibm.jsse2.as.i(as.java:130)
at com.ibm.jsse2.as.a(as.java:483)
at com.ibm.jsse2.as.startHandshake(as.java:160)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:415)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.camel.component.http4.HttpProducer.executeMethod(HttpProducer.java:334)
at org.apache.camel.component.http4.HttpProducer.process(HttpProducer.java:193)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:695)
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:623)
at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:247)
at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:172)
at org.apache.camel.processor.RecipientList.process(RecipientList.java:132)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
at org.apache.camel.impl.ProducerCache$1.doInProducer(ProducerCache.java:529)
at org.apache.camel.impl.ProducerCache$1.doInProducer(ProducerCache.java:497)
at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:365)
at org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:497)
at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:242)
at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:148)
at org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:135)
at org.apache.camel.impl.DefaultProducerTemplate.request(DefaultProducerTemplate.java:301)
Please guide me on how to fix the issue.
Thanks,
Sravan Kumar

Jmeter 3.1- javax.net.ssl.SSLException: Received fatal alert: protocol_version

We are trying to run https request in Jmeter 3.1.
But while executing getting following error.
javax.net.ssl.SSLException: Received fatal alert: protocol_version
at sun.security.ssl.Alerts.getSSLException(Unknown Source)
at sun.security.ssl.Alerts.getSSLException(Unknown Source)
at sun.security.ssl.SSLSocketImpl.recvAlert(Unknown Source)
at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
at sun.security.ssl.AppOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at org.apache.jmeter.protocol.http.sampler.HTTPHC3Impl.sample(HTTPHC3Impl.java:269)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1166)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1155)
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:475)
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:418)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:249)
at java.lang.Thread.run(Unknown Source)
We tried following solutions but did't solved the issue:
added "https.default.protocol=TLSv1" in "jmeter.properties" file
Tried solution provided in
http://rajanmanoj.blogspot.in/2011/02/how-to-test-ssl-using-jmeter.html
this link
We are Using Jmeter 3.1, Java Jdk 1.7, Windows 7
Https default protocol may vary depending on the version of JVM.
You can change it. Go to jmeter home directory. Open jmeter.properties file. Uncomment line:
https.default.protocol=SSLv3
May be you need to restart jmeter after change. Not sure about that.
This is highly due to protocol version mismatch. There is mismatch between SSL protocol version used by the client and the server.
Make sure that you have latest version of Java and JMeter installed in your system. Also check for the security certificate in JMETER_HOME\bin folder.
You can check this question for more details.

How to configure Apache NiFi for a Kerberized Hadoop Cluster

I have Apache NiFi running standalone and its working fine. But, when I am trying to setup Apache NiFi to access Hive or HDFS Kerberized Cloudera Hadoop Cluster. I am getting issues.
Can someone guide me on the documentation for Setting HDFS/Hive/HBase (with Kerberos)
Here is the configuration I gave in nifi.properties
# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf
nifi.kerberos.service.principal=pseeram#JUNIPER.COM
nifi.kerberos.keytab.location=/uhome/pseeram/learning/pseeram.keytab
nifi.kerberos.authentication.expiration=10 hours
I referenced various links like, but none of those are helpful.
(Since the below link said it had issues in NiFi 0.7.1 version, I tried NiFi 1.1.0 version. I had the same bitter experience)
https://community.hortonworks.com/questions/62014/nifi-hive-connection-pool-error.html
https://community.hortonworks.com/articles/4103/hiveserver2-jdbc-connection-url-examples.html
Here are the errors I am getting logs:
ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.hive.SelectHiveQL
org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://ddas1106a:10000/innovate: Peer indicated failure: Unsupported mechanism type PLAIN)
at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:292) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
at sun.reflect.GeneratedMethodAccessor191.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]
at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177) ~[na:na]
at com.sun.proxy.$Proxy83.getConnection(Unknown Source) ~[na:na]
at org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.jar:1.1.0]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.jar:1.1.0]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.jar:1.1.0]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.jar:1.1.0]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.jar:1.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_51]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_51]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://ddas1106a:10000/innovate: Peer indicated failure: Unsupported mechanism type PLAIN)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:288) ~[nifi-hive-processors-1.1.0.jar:1.1.0]
... 18 common frames omitted
Caused by: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://ddas1106a:10000/innovate: Peer indicated failure: Unsupported mechanism type PLAIN
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:231) ~[hive-jdbc-1.2.1.jar:1.2.1]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) ~[hive-jdbc-1.2.1.jar:1.2.1]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[hive-jdbc-1.2.1.jar:1.2.1]
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545) ~[commons-dbcp-1.4.jar:1.4]
... 21 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: Unsupported mechanism type PLAIN
at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199) ~[hive-exec-1.2.1.jar:1.2.1]
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307) ~[hive-exec-1.2.1.jar:1.2.1]
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[hive-exec-1.2.1.jar:1.2.1]
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204) ~[hive-jdbc-1.2.1.jar:1.2.1]
... 27 common frames omitted
WARN [NiFi Web Server-29] o.a.nifi.dbcp.hive.HiveConnectionPool HiveConnectionPool[id=278beb67-0159-1000-cffa-8c8534c285c8] Configuration does not have security enabled, Keytab and Principal will be ignored
What you've added in nifi.properties file is useful for Kerberizing nifi cluster. In order to access kerberized hadoop cluster, you need to provide appropriate config files and keytabs in NiFi's HDFS processor.
For example, if you are using putHDFS to write to a Hadoop cluster:
Hadoop Configuration Resources : paths to core-site.xml and hdfs-site.xml
Kerberos Principal: Your principal to access hadoop cluster
kerberos keytab: Path to keytab generated using krb5.conf of hadoop cluster. nifi.kerberos.krb5.file in nifi.properties must be pointed to appropriate krb5.conf file.
Immaterial of whether NiFi is inside kerberized hadoop cluster or not, this post might be useful.
https://community.hortonworks.com/questions/84659/how-to-use-apache-nifi-on-kerberized-hdp-cluster-n.html

inter-Solr-node communication with SSL

Again, I have some problems with Solr. I'm trying to set up a SolrCloud cluster with three ZooKeeper instances and two Solr instances. Communication should be secured with SSL. I used the official Solr wiki as manual: https://cwiki.apache.org/confluence/display/solr/Enabling+SSL
When I start the first Solr node everything works fine. I can access the Web UI via HTTPS and communication between Solr and ZooKeeper also works. But when I start the second Solr node in my cluster I get the following exception:
Error while trying to recover. core=core1_shard2_replica2:java.util.concurrent.ExecutionException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://ec2-xx-xx-xxx-xx.eu-central-1.compute.amazonaws.com:8983/solr
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:593)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:350)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://ec2-xx-xx-xxx-xx.eu-central-1.compute.amazonaws.com:8983/solr
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:604)
at org.apache.solr.client.solrj.impl.HttpSolrClient.lambda$httpUriRequest$13(HttpSolrClient.java:299)
... 5 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
... 6 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:352)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
... 24 more
Caused by: java.security.cert.CertPathValidatorException: signature check failed
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:219)
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:347)
... 30 more
Caused by: java.security.SignatureException: Signature does not match.
at sun.security.x509.X509CertImpl.verify(X509CertImpl.java:424)
at sun.security.provider.certpath.BasicChecker.verifySignature(BasicChecker.java:166)
at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:147)
at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
So it seems that Solr does not accept the self-signed certificate. The option -Dsolr.ssl.checkPeerName=false as stated in the Solr Wiki has no effect (I put this parameter in the solr.in.sh script). I have to admit I'm pretty clueless right now :(
Does anyone have any idea what I could do to make the inter-Solr-node communication work?
BTW: I'm on Solr 6.1.0, and before I activated SSL everything worked like a charm.
Thanks a lot!
You can also add second node's certificate to the first node's keystore and also add first node's certificate to the second node's keystore via following command:
keytool -import -alias ca -file other_node_cert -keystore solr-ssl.keystore.jks
Then you can use self-signed certificate.
Looking at exception details it appears to be a handshake issue between java client code and the ssl enabled server. The JRE involved here probably doesn't trust the certificate. You can try to mark the cert as a trusted one (include in JRE truststore). Check JRE version specific documentation for steps to update truststore.