NiFi ListSFTP connector cannot connect to sftp server - authentication

I have problems getting ListSFTP processor to work.
First I have tested accessing the sftp server using the commandline on the NiFi server. This works but I have to provide the pass phrase of the private ssh key file.
Then I made sure that I configured the ListSFTP processor using the same settings as the sftp commandline:
Hostname: ip address of the server
Port: 22
Username: Username with access to the server and bound to the key
Private Key Path: set to the full path of the file containing
Private Key Phrase: password for the key file
Remote path: .
Strict Host Key Checking: false
Host Key File: File hat contains the output of ssh-keyscan for the
given server
With these settings I get an error in the logfile. Any idea why NiFi cannot access the key file and access the sftp server. Cheers
2020-03-19 15:43:19,736 ERROR [Timer-Driven Process Thread-2]
o.a.nifi.processors.standard.ListSFTP
ListSFTP[id=d343fa19-0170-1000-3352-999829a448cc] Failed to perform
listing on remote host due to java.io.IOException: Failed to obtain c
onnection to remote host due to com.jcraft.jsch.JSchException: invalid
privatekey: [B#6d9b662: {} java.io.IOException: Failed to obtain
connection to remote host due to com.jcraft.jsch.JSchException:
invalid privatekey: [B#6d9b662
at org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:515)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:212)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:175)
at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:106)
at org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:146)
at org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:472)
at org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:414)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by: com.jcraft.jsch.JSchException: invalid privatekey: [B#6d9b662
at com.jcraft.jsch.KeyPair.load(KeyPair.java:664)
at com.jcraft.jsch.KeyPair.load(KeyPair.java:561)
at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:40)
at com.jcraft.jsch.JSch.addIdentity(JSch.java:407)
at com.jcraft.jsch.JSch.addIdentity(JSch.java:388)
at org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:485)
... 18 common frames omitted

Related

Key exchange was not finished, connection is closed with ganymed-ssh-2 and Cisco Confd

I am using JNC 1.0 & ganymed-ssh-2 v262 to establish a Netconf session over SSH to a Cisco Confd node and getting the following error. What could be the reason?
java.io.IOException: Key exchange was not finished, connection is closed.
at ch.ethz.ssh2.transport.KexManager.getOrWaitForConnectionInfo(KexManager.java:75)
at ch.ethz.ssh2.transport.TransportManager.getConnectionInfo(TransportManager.java:169)
at ch.ethz.ssh2.Connection.connect(Connection.java:759)
at com.tailf.jnc.SSHConnection.(Unknown Source)
at com.tailf.jnc.Device.connect(Unknown Source)
at com.tailf.jnc.Device.connect(Unknown Source)
....
java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: java.io.IOException: Cannot negotiate, proposals do not match.
at ch.ethz.ssh2.transport.ClientKexManager.handleMessage(ClientKexManager.java:123)
at ch.ethz.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:572)
at ch.ethz.ssh2.transport.TransportManager$1.run(TransportManager.java:261)
at java.base/java.lang.Thread.run(Thread.java:833)
java.io.IOException: Key exchange was not finished, connection is closed.
...
...
Caused by: java.io.IOException: Cannot negotiate, proposals do not match.
It means that there is a mismatch between the supported key exchange algorithms on the client and those supported by the server.
I'd advice to update ganymed-ssh-2 to v263. Apparently the latest version has added support for the following kex algorithms:
diffie-hellman-group14-sha256
diffie-hellman-group16-sha512
diffie-hellman-group18-sha512
You can check what the server supports by using ssh's debug flag on a terminal:
ssh -vvv user#host

How to enable multiple listeners in kafka cluster

I have a 3 nodes Kafka cluster. I have enabled SASL_PLAINTEXT and it is working fine with Port 6667. Now I want o enable SSL for different Port in the same cluster. I have enabled the trustore and Keystore certificates. and I did below configuration from the broker side.
listeners : SSL://localhost:6668
security.inter.broker.protocol : SSL
ssl.key.password : xxxx
ssl.keystore.location : /root/kafka.server.keystore.jks
ssl.keystore.password : xxxxx
ssl.truststore.location : /root/kafka.server.truststore.jks
ssl.truststore.password : xxxxxx
ssl.keystore.type : JKS
ssl.truststore.type : JKS
I Have given permission also. I am getting below errors
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /root/kafka.server.keystore.jks of type JKS
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /root/kafka.server.keystore.jks of type JKS
Caused by: java.io.FileNotFoundException: /root/kafka.server.keystore.jks (Permission denied)
Caused by: java.io.FileNotFoundException:
/root/kafka.server.keystore.jks (Permission denied)
The error trace is fairly clear. /root/kafka.server.keystore.jks cannot be accessed by the process. Note that the process typically runs on a different user and I suspect that the keystore has been created by a different user.
Make sure that the user that is running the process has sufficient access rights for reading /root/kafka.server.keystore.jks. One way of achieving this is to change the ownership of the file:
sudo chown -R userWhoRunsTheProcess:userGroup /root/kafka.server.keystore.jks
Regarding the question, listeners takes a list of addresses,
listeners : SSL://0.0.0.0:6668,SASL_PLAINTEXT://0.0.0.0;6667
You already have sasl, so I would suggest using sasl_ssl

Try to run the azure-iot-sdk-java provisioning-x509-sample with an own x509-certificate

I'm trying to run the azure-iot-sdk-java provisioning-X509-sample.
What I do jet is, I have create a certificate-chain with openssl and generate a X509-certificat. I uploaded this certificate to the azure DPS.
I check the fingerprint of the uploaded cert and all looks fine.
In a next step I configure the ProvisioningX509Sample.java. Enter the idScope, globalEndpoint and as the leafPublicPem I enter the Base64 String
"-----BEGIN CERTIFICATE-----" + "\n" +
************ + "\n" +
"-----END CERTIFICATE-----"
Also as the leafPrivateKey I enter the Private Key of my intermediate-ca which create/sign my X509-Device-certifivate
"-----BEGIN RSA PRIVATE KEY-----" + "\n" +
************+ "\n" +
"-----END RSA PRIVATE KEY-----";
As transport-protocol i will use MQTT
If I run this sample I get the following error(s)
Starting...
Beginning setup.
Waiting for Provisioning Service to register
com.microsoft.azure.sdk.iot.provisioning.device.internal.exceptions.ProvisioningDeviceConnectionException: Exception opening connection amqp
at com.microsoft.azure.sdk.iot.provisioning.device.internal.contract.mqtt.ContractAPIMqtt.open(ContractAPIMqtt.java:142)
at com.microsoft.azure.sdk.iot.provisioning.device.internal.task.ProvisioningTask.call(ProvisioningTask.java:267)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to connect to mqtt service
at com.microsoft.azure.sdk.iot.deps.transport.mqtt.MqttConnection.connect(MqttConnection.java:151)
at com.microsoft.azure.sdk.iot.provisioning.device.internal.contract.mqtt.ContractAPIMqtt.open(ContractAPIMqtt.java:135)
... 5 more
Caused by: MqttException (0) - javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:38)
at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:715)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1779)
at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:124)
at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1156)
at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1267)
at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1179)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:348)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
at org.eclipse.paho.client.mqttv3.internal.SSLNetworkModule.start(SSLNetworkModule.java:108)
at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:701)
... 7 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
... 21 more
Registration error, bailing out
Press any key to exit...
I don't understand this error. Is there another step missing?
thanks for help and best regards
JensV
DPS only supports TLS 1.2. TLS connections from older versions will be rejected.

LDAP- Error opening connection : xx.xx.xx.xx:389

I'm trying to connect the LDAP server available on cloud. The web interface for the server is : xx.xx.xx.xx/phpldapadmin where I can login and do whatever operation I want to do.
But when I'm trying to bind the ldap from my local machine through JXplorer, I'm getting connection timeout error. Here are the screenshots :
Connection Wizard
Error wizard:
Stack Trace
javax.naming.CommunicationException: XX.XX.XX.XX:389 [Root exception is java.net.ConnectException: Connection timed out: connect]
at com.sun.jndi.ldap.Connection.<init>(Connection.java:223)
at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:136)
at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1600)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2698)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:316)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:193)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:211)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:154)
at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:84)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:153)
at com.ca.commons.jndi.JNDIOps.openContext(JNDIOps.java:529)
at com.ca.commons.jndi.JNDIOps.<init>(JNDIOps.java:123)
at com.ca.commons.jndi.BasicOps.<init>(BasicOps.java:55)
at com.ca.commons.jndi.AdvancedOps.<init>(AdvancedOps.java:59)
at com.ca.commons.naming.DXOps.<init>(DXOps.java:41)
at com.ca.directory.jxplorer.broker.CBGraphicsOps.<init>(CBGraphicsOps.java:46)
at com.ca.directory.jxplorer.broker.JNDIDataBroker.openConnection(JNDIDataBroker.java:477)
at com.ca.directory.jxplorer.broker.JNDIDataBroker.openConnection(JNDIDataBroker.java:422)
at com.ca.directory.jxplorer.broker.JNDIDataBroker.processRequest(JNDIDataBroker.java:396)
at com.ca.directory.jxplorer.broker.DataBroker.processQueue(DataBroker.java:200)
at com.ca.directory.jxplorer.broker.JNDIDataBroker.processQueue(JNDIDataBroker.java:913)
at com.ca.directory.jxplorer.broker.DataBroker.run(DataBroker.java:165)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:69)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:337)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at com.sun.jndi.ldap.Connection.createSocket(Connection.java:365)
at com.sun.jndi.ldap.Connection.<init>(Connection.java:200)
... 25 more
Please help !
Connection timed out indicates that the port 389 is not accessible from your network. Make sure the firewall in cloud allows your ip to access port 389 of the LDAP host. phpldapadmin will work since it will listen on port 80/443 and it in turn connects with LDAP locally.
You can test the connectivity by running telnet <LDAP Host> 389. If the port is accessible from you host, you'll be able to establish a connection.

Setting up secure cassandra cluster (java.lang.RuntimeException: Failed to setup secure pipeline at )

I have followed the steps mentioned on https://github.com/PatrickCallaghan/datastax-ssl-secure-cluster/blob/master/README.md for setting up a secure SSL cassandra cluster. I receive the same error as you "Failed to setup secure pipeline". I overrode my cassandra.yaml cipher suites as mentioned by the website and I still get the same error.
My cassandra.yaml looks like this:
client_encryption_options:
enabled: true
# If enabled and optional is set to true encrypted and unencrypted connections are handled.
optional: false
keystore: ***/ssl/cassandra3_keystore.jks
keystore_password: ****
# require_client_auth: false
# Set trustore and truststore_password if require_client_auth is true
# truststore: conf/.truststore
# truststore_password: cassandra
# More advanced defaults below:
# protocol: TLS
# algorithm: SunX509
# store_type: JKS
cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA]
Could someone guide me on what I could do?
Here is the full error trace:
Exception (java.lang.RuntimeException) encountered during startup: Failed to setup secure pipeline
java.lang.RuntimeException: Failed to setup secure pipeline
at org.apache.cassandra.transport.Server$AbstractSecureIntializer.<init>(Server.java:354)
at org.apache.cassandra.transport.Server$SecureInitializer.<init>(Server.java:411)
at org.apache.cassandra.transport.Server.start(Server.java:152)
at org.apache.cassandra.service.NativeTransportService$$Lambda$203.0000000040E88830.accept(Unknown Source)
at java.util.Collections$SingletonSet.forEach(Collections.java:4778)
at org.apache.cassandra.service.NativeTransportService.start(NativeTransportService.java:128)
at org.apache.cassandra.service.CassandraDaemon.startNativeTransport(CassandraDaemon.java:633)
at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:495)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714)
Caused by: java.io.IOException: Error creating the initializing the SSL Context
at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:170)
at org.apache.cassandra.transport.Server$AbstractSecureIntializer.<init>(Server.java:350)
... 9 more
Caused by: java.security.NoSuchAlgorithmException: SunX509 KeyManagerFactory not available
at sun.security.jca.GetInstance.getInstance(GetInstance.java:171)
at javax.net.ssl.KeyManagerFactory.getInstance(KeyManagerFactory.java:12)
at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:146)
... 10 more
ERROR 15:36:01 Exception encountered during startup
java.lang.RuntimeException: Failed to setup secure pipeline
at org.apache.cassandra.transport.Server$AbstractSecureIntializer.<init>(Server.java:354) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.transport.Server$SecureInitializer.<init>(Server.java:411) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.transport.Server.start(Server.java:152) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.NativeTransportService$$Lambda$203.0000000040E88830.accept(Unknown Source) ~[na:na]
at java.util.Collections$SingletonSet.forEach(Collections.java:4778) ~[na:1.8.0-internal]
at org.apache.cassandra.service.NativeTransportService.start(NativeTransportService.java:128) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.startNativeTransport(CassandraDaemon.java:633) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:495) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.jar:3.7]
Caused by: java.io.IOException: Error creating the initializing the SSL Context
at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:170) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.transport.Server$AbstractSecureIntializer.<init>(Server.java:350) ~[apache-cassandra-3.7.jar:3.7]
... 9 common frames omitted
Caused by: java.security.NoSuchAlgorithmException: SunX509 KeyManagerFactory not available
at sun.security.jca.GetInstance.getInstance(GetInstance.java:171) ~[na:1.8.0-internal]
at javax.net.ssl.KeyManagerFactory.getInstance(KeyManagerFactory.java:12) ~[na:8.0 build_20150122]
at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:146) ~[apache-cassandra-3.7.jar:3.7]
... 10 common frames omitted
You can get round it by overriding the cipher suites for both node-to-node and client-node properties e.g.
cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA]
This is because of the following problem in Oracle Java. http://www.pathin.org/tutorials/java-cassandra-cannot-support-tls_rsa_with_aes_256_cbc_sha-with-currently-installed-providers/
Once downloaded you can copy the files to the correct library on your server.
e.g.
scp * root#server:/usr/lib/jvm/java-7-oracle/jre/lib/security/