I am trying to setup inter-broker SSL (not client) authentication and keep seeing the following errors:
[2019-05-17 06:33:47,151] INFO [Controller id=1004, targetBrokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] INFO [SocketServer brokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] ERROR [Controller id=1004, targetBrokerId=1004] Connection to node 1004 (/$IP:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
My server.properties is:
listeners=PLAINTEXT://$IP:9092,SSL://$IP:9093
security.inter.broker.protocol=SSL
ssl.truststore.password=$PASS
ssl.keystore.password=$PASS
ssl.key.password=$PASS
ssl.endpoint.identification.algorithm=""
ssl.keystore.location=/etc/kafka/kafka.server.keystore.jks
ssl.truststore.location=/etc/kafka/kafka.server.truststore.jks
``
When I run `openssl s_client -debug -connect $IP:9093 -tls1` I get back a list of certificates and `Secure Renegotiation IS supported`
Despite adding `-Djavax.net.debug=all` there's not anything in the logs which points to the problem.
Kafka version 2.2
Any ideas?
I had incorrectly set the value of ssl.endpoint.identification.algorithm="" instead of ssl.endpoint.identification.algorithm", this fixed it.
This value was changed in 2.2 to default to https so setting it to nothing worked.
Related
Getting SSL errors in a cluster of three Kafka servers that communicate over SSL (only).
Why is this happening / how can I fix it?
server.properties
listeners=SSL://some_host_name.corp.com:9092
inter.broker.listener.name=SSL
ssl.keystore.location=/some/path/to/keystore.p12
ssl.keystore.password=***
ssl.key.password=***
ssl.truststore.location=/some/path/to/server.truststore.jks
ssl.truststore.password=***
ssl.enabled.protocols=TLSv1.2
ssl.client.auth=required
ssl.keystore.type=PKCS12
ssl.truststore.type=JKS
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# Kafka is running with -Djavax.net.debug=ssl
# Assume localhost IP address is 1.2.3.4
18:31:29.479 [data-plane-kafka-socket-acceptor-ListenerName(SSL)-SSL-9092] DEBUG kafka.network.Acceptor - Accepted connection from /1.2.3.4:46732 on /1.2.3.4:9092 and assigned it to processor 1, sendBufferSize [actual|requested]: [212992|1048576] recvBufferSize [actual|requested]: [212992|1048576]
18:31:29.479 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] DEBUG kafka.network.Processor - Processor 1 listening to new connection from /1.2.3.4:46732
Using SSLEngineImpl.
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, fatal error: 80: problem unwrapping net record
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, SEND TLSv1.2 ALERT: fatal, description = internal_error
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, WRITE: TLSv1.2 Alert, length = 2
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeOutbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, closeOutboundInternal()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeInbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
18:31:29.481 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] DEBUG o.a.k.c.network.SslTransportLayer - [SslTransportLayer channelId=1.2.3.4:9092-1.2.3.4:46732-108 key=sun.nio.ch.SelectionKeyImpl#33a4cffe] SSLEngine.closeInBound() raised an exception.
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561) ~[na:1.8.0_181]
at org.apache.kafka.common.network.SslTransportLayer.handshakeFailure(SslTransportLayer.java:871) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.SslTransportLayer.maybeProcessHandshakeFailure(SslTransportLayer.java:909) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:295) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [kafka-clients-2.7.0.jar:na]
at kafka.network.Processor.poll(SocketServer.scala:923) [kafka_2.12-2.7.0.jar:na]
at kafka.network.Processor.run(SocketServer.scala:826) [kafka_2.12-2.7.0.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
18:31:29.481 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] INFO o.a.kafka.common.network.Selector - [SocketServer brokerId=1] Failed authentication with /1.2.3.4 (SSL handshake failed)
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeOutbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, closeOutboundInternal()
Elsewhere in the log of another cluster member, authentication succeeds. So why does it work between different nodes, but not from one node to itself?
20:07:39.479 [data-plane-kafka-network-thread-3-ListenerName(SSL)-SSL-1] DEBUG o.a.k.c.network.SslTransportLayer - [SslTransportLayer channelId=1.2.3.6:9092-1.2.3.5:41182-0 key=sun.nio.ch.SelectionKeyImpl#69864b7d] SSL handshake completed successfully with peerHost '1.2.3.5' peerPort 41182 peerPrincipal '...' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384'
20:07:39.480 [data-plane-kafka-network-thread-3-ListenerName(SSL)-SSL-1] DEBUG o.a.kafka.common.network.Selector - [SocketServer brokerId=3] Successfully authenticated with /1.2.3.5
I have solved the issue by moving the SSL listener away from the default port of 9092.
This leaves me with a different question: if I have configured only one listener, with SSL, on port 9092, why is there any PLAINTEXT activity on that same port?
The errors shown above were caused by each broker trying to connect to itself on port 9092, probably using PLAINTEXT, even though I had configured only SSL.
Also: what is the difference between settings (1) and (2) below?
# (1) The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
listeners=SSL://$auto_host:9092
# (2) The port the socket server listens on
port=9092
I'm trying to publish message to a Tibco Queue on a SSL Tibco Server through JMeter 5.4.1 using JMS Point-to-Point Logic Controller.
JMS Point To Point Controller Config
But I'm getting the following error message:
2021-06-13 12:25:46,278 ERROR o.a.j.p.j.s.JMSSampler: Not permitted:
Failed to connect to any server at: ssl://[server-name]:7352,
ssl://[server-name]:7352 [Error: Failed to connect via SSL to
[ssl://[server-name]:7352]: Received fatal alert:
protocol_version: url that returned this exception =
SSL://[server-name]:7352 ]
javax.naming.AuthenticationException: Not permitted: Failed to connect
to any server at: ssl://[server-name]:7352,
ssl://[server-name]:7352 [Error: Failed to connect via SSL to
[ssl://[server-name]:7352]: Received fatal alert:
protocol_version: url that returned this exception =
SSL://[server-name] ] at
com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:670)
~[tibjms.jar:8.0.0] at
com.tibco.tibjms.naming.TibjmsContext.lookup(TibjmsContext.java:491)
~[tibjms.jar:8.0.0] at
javax.naming.InitialContext.lookup(InitialContext.java:417)
~[?:1.8.0_291] at
org.apache.jmeter.protocol.jms.sampler.JMSSampler.threadStarted(JMSSampler.java:638)
[ApacheJMeter_jms.jar:5.4.1] at
org.apache.jmeter.threads.JMeterThread$ThreadListenerTraverser.addNode(JMeterThread.java:784)
[ApacheJMeter_core.jar:5.4.1] at
org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:993)
[jorphan.jar:5.4.1] at
org.apache.jorphan.collections.HashTree.traverse(HashTree.java:976)
[jorphan.jar:5.4.1] at
org.apache.jmeter.threads.JMeterThread.threadStarted(JMeterThread.java:752)
[ApacheJMeter_core.jar:5.4.1] at
org.apache.jmeter.threads.JMeterThread.initRun(JMeterThread.java:740)
[ApacheJMeter_core.jar:5.4.1] at
org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:252)
[ApacheJMeter_core.jar:5.4.1]
I tried:
openssl s_client -connect [server-name]:7352
It gave the following output:
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID:
Session-ID-ctx:
So added the following line in jmeter.properties file.
https.default.protocol=TLSv1.2
Also commented jdk.tls.disabledAlgorithms from java.security file for JDK (I'm using jdk1.8.0_291)
# jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
# DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
# include jdk.disabled.namedCurves
But still I'm getting the same error. Someone please help.
I think you're using the wrong property (not only the wrong property but the wrong place as well), you're setting default protocol for HTTPS, while you need to set it for TLS, i.e. add the next line to system.properties file
jdk.tls.client.protocols=TLSv1.2
JMeter restart will be required to apply this property.
If it won't help or you will get different errors - consider adding the next line there as well:
javax.net.debug=all
and then check jmeter.log file and stdout for any suspicious entries
More information:
Configuring JMeter
Apache JMeter Properties Customization Guide
I resolved it by using the latest tibjms.jar in the lib directory in JMeter as the Tibco server was upgraded some hours before I raised this issue.
I have a distributed kafka with 3 brokers that have port numbers 9093, 9094, 9095 and I added the SSL with port numbers 9096, 9097, 9098.
I am getting the following error when I run the producer client:
[2020-06-15 10:08:07,892] ERROR [Producer clientId=console-producer] Connection to node -1 (/myip-address:9096) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2020-06-15 10:08:07,893] WARN [Producer clientId=console-producer] Bootstrap broker myip-address:9096 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
I have tried to use: ssl.endpoint.identification.algorithm= but that did not solve the problem for me.
I am using Kafka 2.5
I can share my config files if need be.
What else could I try to troubleshoot this issue?
Thank you.
I was able to solve this issue by simply using my
domain-name:9096,domain-name:9097,domain-name:9098
instead of:
my-ip-address:9096,my-ip-address:9097,my-ip-address:9098
So, using the actual domain name is important due to the certificate being created with the domain name.
Have RabbitMQ configured to enable TLS with certificates. Key, Cert, and CA defined in .conf file. Upon service startup, error is thrown. Cannot find the cause for this to be thrown and logging isn't giving any more information at the debug level.
Get a client alert failure and am not certain of cause.
2019-03-22 10:04:18.690 [info] <0.7.0> Server startup complete; 4 plugins started.
* rabbitmq_amqp1_0
* rabbitmq_management
* rabbitmq_management_agent
* rabbitmq_web_dispatch
2019-03-22 10:04:24.831 [debug] <0.689.0> Supervisor {<0.689.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.690.0>
2019-03-22 10:04:24.831 [debug] <0.689.0> Supervisor {<0.689.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.690.0>, {acceptor,{0,0,0,0},5671}) at pid <0.691.0>
2019-03-22 10:04:24.909 [info] <0.688.0> TLS server: In state certify received CLIENT ALERT: Fatal - Certificate Unknown
Our certs didn't have the correct type of X509v3 Extended Key Usage on the cert.
For x509 Auth, you'll need to assign client web auth when creating the certificate.
X509v3 Extended Key Usage:
TLS Web Client Authentication
This won't fix the issue if your certificate CA is broken and can't be verified, but for my issue, this was the resolution.
I have done broker and client configuration on same node.
When ssl.client.auth=none it works fine but whenever I change that property to "required", ssl.client.auth=required and enabled security.inter.broker.protocol=SSL then it gives me an issue on producer side.
[2017-12-13 11:06:56,106] WARN Failed to send SSL Close message (org.apache.kafka.common.network.SslTransportLayer)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
at org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:194)
at org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:161)
at org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:45)
at org.apache.kafka.common.network.Selector.close(Selector.java:442)
at org.apache.kafka.common.network.Selector.poll(Selector.java:310)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
at java.lang.Thread.run(Thread.java:745)
Any solution for this?
As mentioned in the comments, the answer was found by the asker:
Its resolved.This issue comes when signed certificate of the server
did not match with client keystore. We need to generate client
keystore and import the the signed certificate of the server to client
keystore.