Getting SSL errors in a cluster of three Kafka servers that communicate over SSL (only).
Why is this happening / how can I fix it?
server.properties
listeners=SSL://some_host_name.corp.com:9092
inter.broker.listener.name=SSL
ssl.keystore.location=/some/path/to/keystore.p12
ssl.keystore.password=***
ssl.key.password=***
ssl.truststore.location=/some/path/to/server.truststore.jks
ssl.truststore.password=***
ssl.enabled.protocols=TLSv1.2
ssl.client.auth=required
ssl.keystore.type=PKCS12
ssl.truststore.type=JKS
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# Kafka is running with -Djavax.net.debug=ssl
# Assume localhost IP address is 1.2.3.4
18:31:29.479 [data-plane-kafka-socket-acceptor-ListenerName(SSL)-SSL-9092] DEBUG kafka.network.Acceptor - Accepted connection from /1.2.3.4:46732 on /1.2.3.4:9092 and assigned it to processor 1, sendBufferSize [actual|requested]: [212992|1048576] recvBufferSize [actual|requested]: [212992|1048576]
18:31:29.479 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] DEBUG kafka.network.Processor - Processor 1 listening to new connection from /1.2.3.4:46732
Using SSLEngineImpl.
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, fatal error: 80: problem unwrapping net record
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, SEND TLSv1.2 ALERT: fatal, description = internal_error
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, WRITE: TLSv1.2 Alert, length = 2
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeOutbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, closeOutboundInternal()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeInbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
18:31:29.481 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] DEBUG o.a.k.c.network.SslTransportLayer - [SslTransportLayer channelId=1.2.3.4:9092-1.2.3.4:46732-108 key=sun.nio.ch.SelectionKeyImpl#33a4cffe] SSLEngine.closeInBound() raised an exception.
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561) ~[na:1.8.0_181]
at org.apache.kafka.common.network.SslTransportLayer.handshakeFailure(SslTransportLayer.java:871) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.SslTransportLayer.maybeProcessHandshakeFailure(SslTransportLayer.java:909) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:295) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [kafka-clients-2.7.0.jar:na]
at kafka.network.Processor.poll(SocketServer.scala:923) [kafka_2.12-2.7.0.jar:na]
at kafka.network.Processor.run(SocketServer.scala:826) [kafka_2.12-2.7.0.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
18:31:29.481 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] INFO o.a.kafka.common.network.Selector - [SocketServer brokerId=1] Failed authentication with /1.2.3.4 (SSL handshake failed)
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeOutbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, closeOutboundInternal()
Elsewhere in the log of another cluster member, authentication succeeds. So why does it work between different nodes, but not from one node to itself?
20:07:39.479 [data-plane-kafka-network-thread-3-ListenerName(SSL)-SSL-1] DEBUG o.a.k.c.network.SslTransportLayer - [SslTransportLayer channelId=1.2.3.6:9092-1.2.3.5:41182-0 key=sun.nio.ch.SelectionKeyImpl#69864b7d] SSL handshake completed successfully with peerHost '1.2.3.5' peerPort 41182 peerPrincipal '...' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384'
20:07:39.480 [data-plane-kafka-network-thread-3-ListenerName(SSL)-SSL-1] DEBUG o.a.kafka.common.network.Selector - [SocketServer brokerId=3] Successfully authenticated with /1.2.3.5
I have solved the issue by moving the SSL listener away from the default port of 9092.
This leaves me with a different question: if I have configured only one listener, with SSL, on port 9092, why is there any PLAINTEXT activity on that same port?
The errors shown above were caused by each broker trying to connect to itself on port 9092, probably using PLAINTEXT, even though I had configured only SSL.
Also: what is the difference between settings (1) and (2) below?
# (1) The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
listeners=SSL://$auto_host:9092
# (2) The port the socket server listens on
port=9092
Problem summary:
I am trying to run symmetric ds on tomcat with ssl. Setup details
1). MasterNode M1: running on tomcat.
2). Cleint node C1: running on tomcat.
I am able to run symmetricds on tomcat without ssl. This setup works fine all sync happens.
Now when I am trying to run tomcat with following setting for ssl, it gives bad_certificate error on client node:
connection setting on Master node:
<Connector SSLEnabled="true" acceptCount="10"
algorithm="PKIX" allowTrace="false"
ciphers=""
clientAuth="true"
enableLookups="true"
keystoreFile="/root/symmetric-server-3.8.28/security/keystore.p12"
keystorePass="changeit#1"
keystoreType="pkcs12"
port="31415"
redirectPort="-1"
scheme="https"
secure="true"
sslEnabledProtocols="TLSv1.2"
sslProtocol="TLSv1.2"
truststoreFile="/root/symmetric-server-3.8.28/security/keystore.p12"
truststorePass="changeit#1"
truststoreType="pkcs12"/>
connector setting on Client node:
<Connector SSLEnabled="true" acceptCount="10"
algorithm="PKIX" allowTrace="false"
ciphers=""
clientAuth="true"
enableLookups="true"
keystoreFile="/root/symmetric-server-3.8.28/security/keystore2.p12"
keystorePass="changeit#1"
keystoreType="pkcs12"
port="31415"
redirectPort="-1"
scheme="https"
secure="true"
sslEnabledProtocols="TLSv1.2"
sslProtocol="TLSv1.2"
truststoreFile="/root/symmetric-server-3.8.28/security/keystore2.p12"
truststorePass="changeit#1"
truststoreType="pkcs12"/>
Note: If I run Master node tomcat with clientAuth=false then everything works fine. On further debugging I found that client is not providing its certificate during TLS handshake. Following is error: Experts please help what config I need to do so that client provide certificate:
Error:
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
Upgrade to SymmetricDS 3.8.31. There was an issue that was fixed that allows for client TLS authentication now:
http://www.symmetricds.org/issues/view.php?id=2886
Given the new requirement that browsers no longer support voice/video over insecure origine(which is taking us we the developers to redo/undo/goback since none of the test environment workarounds is practical in A distributed environment),I have tried to switch to https and wss and I have configured MMS connector with the following.I am running a converged app with two tomcat instances:
1-Apache Tomcat v8.0 at localhost for the app embeding WebRTComm framework( WebRTComm.js and jain-sip.js etc..).https is working perfect in browser and in code for the app.
2-Mobicent Apache Tomcat v7.0 at localhost for the sipservlet for signalling and other need pre-session requirements.WSS is causing trouble at the server with no return from handshake.
this is the connector in Mobicent Apache Tomcat v7.0.
URIEncoding="UTF-8"
acceptCount="200"
clientAuth="false"
compressableMimeType="text/html,text/xml,text/plain"
compression="off"
compressionMinSize="2048"
connectionUploadTimeout="120000"
disableUploadTimeout="true"
enableLookups="false"
keystoreFile=":\Developement\.....\ServerKeyStore\server.jks" // wonder why relative path(conf/ServerTrustStore\truststore.jks) is resolved but throw FileNoFoundExc in particular case.Any idea?
keystorePass="xxxxxx"
keyAlias="clientselfsigned"
maxKeepAliveRequests="200"
maxThreads="250"
maxSpareThreads="75"
minSpareThreads="25"
maxHttpHeaderSize="8192"
port="8443"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="https"
secure="true"
sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1,SSLv3,SSLv2Hello"
sslProtocol="TLS"
truststorePass="xxxxxx"
truststoreType="jks"
truststorefile="D:\Developement\.....\ServerKeyStore\server.jks" // wonder why relative path(conf/ServerTrustStore\truststore.jks) is resolved but throw FileNoFoundExc in particular case.Any idea?
/>
and this is the connector in Apache Tomcat v8.0 for the web app:
URIEncoding="UTF-8"
acceptCount="200"
clientAuth="false"
compressableMimeType="text/html,text/xml,text/plain"
compression="off"
connectionUploadTimeout="120000"
disableUploadTimeout="true"
enableLookups="false"
keystoreFile="D:\Developement\....\ServerTrustStore\truststore.jks"
keystorePass="xxxxx"
keyAlias="clientselfsigned"
maxKeepAliveRequests="200"
maxThreads="250"
maxSpareThreads="75"
minSpareThreads="25"
maxHttpHeaderSize="8192"
port="8443"
protocol="org.apache.coyote.http11.Http11NioProtocol"
scheme="https"
secure="true"
sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1,SSLv3,SSLv2Hello"
sslProtocol="TLS"
truststorePass="xxxxxx"
truststoreType="jks"
truststorefile="D:\Developement\....\ServerTrustStore\truststore.jks"
/>
Both configured my sip-stack and eclipse lunch configuration(run/debug) with :
gov.nist.javax.sip.TLS_CLIENT_AUTH_TYPE=Disabled
javax.net.ssl.keyStore="D:\Developement\Projects\Sources\Pending\security\ServerKeyStore\server.jks"
javax.net.ssl.trustStore="D:\Developement\Projects\Sources\Pending\security\ServerKeyStore\server.jks"
javax.net.ssl.trustStorePassword=xxx
javax.net.ssl.keyStorePassword=xxx
javax.net.ssl.trustStoreType=JKS
gov.nist.javax.sip.TLS_CLIENT_PROTOCOLS=TLSv1.2,TLSv1.1,TLSv1,SSLv3,SSLv2Hello
gov.nist.javax.sip.MAX_MESSAGE_SIZE=1048576
javax.net.debug=ssl
.
I have also created the necessary keystore and clien truststore as can be seen above.
I have tried a variation of TLSv1.2,TLSv1.1,TLSv1 and see that the client(jain-sip.js) and MMS are both hapy with TLSv1.2. see ClientHello, TLSv1.2 and
ServerHello, TLSv1.2 as below:
%% Initialized: [Session-13, SSL_NULL_WITH_NULL_NULL]
%% Negotiating: [Session-13, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
*** ServerHello, TLSv1.2
and the TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 was selected.
So when I click client.hml=>>TelScaleRTMController.js(onClickConnectButtonViewEventHandler)==>>this.webRTCommClient.open(this.webRTCommClientConfiguration);==>RegistrarSIPServlet.java, TLS handshakes succeeds as I can get to my RegistrarSIPServlet.doRegister():
It is here that things go bad:As soon as it executes SipServletResponse.send();,it says:
NioSelector-WSS-172.62.2.10/5082, RECV TLSv1.2 ALERT: fatal, protocol_version
NioSelector-WSS-172.62.2.10/5082, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Received fatal alert: protocol_version
NioSelector-WSS-172.62.2.10/5082, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Received fatal alert: protocol_version
javax.net.ssl.SSLException: Received fatal alert: protocol_version
.....
Caused by: java.lang.NullPointerException
at gov.nist.javax.sip.stack.SSLStateMachine.unwrap(SSLStateMachine.java:263)
at gov.nist.javax.sip.stack.SSLStateMachine.unwrap(SSLStateMachine.java:198)
at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel.addBytes(NioTlsWebSocketMessageChannel.java:215)
at gov.nist.javax.sip.stack.NioTcpMessageChannel.readChannel(NioTcpMessageChannel.java:117)
.....
the sip msg generated and received by the registrar is:
SIP message sent: REGISTER sip:pbx.server.com SIP/2.0
Call-ID: 1451477183409
CSeq: 1 REGISTER
From: sip:admin#pbx.server.com;tag=1451477183429
To: sip:admin#pbx.server.com
Via: SIP/2.0/WSS XNCrSBbscUch.invalid;branch=z9hG4bK-333430-45dd017fdcbb8f884bf0a78789902ae7;rport
Max-Forwards: 70
User-Agent: RoooterUAv1.0[admin-MOHAMMAD-125-1]-0
Expires: 3600
Allow: INVITE,ACK,BYE,CANCEL,UPDATE,INFO,SUBSCRIBE,NOTIFY,REFER,MESSAGE,OPTIONS
Contact: sip:admin#XNCrSBbscUch.invalid;transport=wss
Content-Length: 0
As you can see no SDP is generated as now and this is where we stop.
The client waits to timeout ....
One more,I have restriced to the same domain of my PBX, in my dar as:
INVITE=("project","DAR:From","ORIGINATING","","NO_ROUTE","0","REGEX=From:.sip:.#pbx.domain.com"),("project","DAR:From","TERMINATING","","NO_ROUTE","0","REGEX=From:.sip:.#pbx.domain.com")
MESSAGE=("project","DAR:From","ORIGINATING","","NO_ROUTE","0")
PUBLISH=("project","DAR:From","ORIGINATING","","NO_ROUTE","0")
NOTIFY=("project","DAR:From","ORIGINATING","","NO_ROUTE","0")
REGISTER=("project","DAR:From","ORIGINATING","","NO_ROUTE","0","REGEX=From:.sip:.#pbx.domain.com",("project","DAR:From","TERMINATING","","NO_ROUTE","0","REGEX=From:.sip:.#pbx.domain.com")
REFER=("project","DAR:From","ORIGINATING","","NO_ROUTE","0")
SUBSCRIBE=("project","DAR:From","ORIGINATING","","NO_ROUTE","0")
OPTIONS=("project","DAR:From","ORIGINATING","","NO_ROUTE","0")
I will very much applreciate your ideas on understanding the problem and the solution.
Thank you !!
I write a demo to access a server with java HttpsUrlConnection.
I implement the X509TrustManager interface with MyX509TrustManager to trust all certificates.
It works fine with JDK7 and JDK8, but when use JDK6 and JDK5, error happens:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1215)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1199)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1014)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
For server side, running on Tomcat 8 and JDK8.
EDIT I add -Djavax.net.debug=ssl,handshake and find following:
*** main, WRITE: TLSv1 Handshake, length = 177 main, WRITE: SSLv2 client hello message, length = 173
main, received EOFException: error main, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
main, SEND TLSv1 ALERT: fatal, description = handshake_failure
main, WRITE: TLSv1 Alert, length = 2 main, called closeSocket()
main, called close() main, called closeInternal(true)
How to disable SSLv3 in Wildfly app server. Following configuration
<server-identities>
<ssl protocol="TLSv1">
<keystore path="https.keystore" relative-to="keystore.home" keystore-password="secret" alias="https" key-password="secret" />
</ssl>
</server-identities>
falls back to SSLv3, if SSLv3 requested. Looks like associated bug https://bugzilla.redhat.com/show_bug.cgi?id=1153853 in redhat is closed as "WONTFIX"
Add [enabled-protocols="TLSv1,TLSv1.1,TLSv1.2"] to element. It works with Wildfly 8.2