Kafka Broker Failed authentication - SSL handshake failed - ssl

Getting SSL errors in a cluster of three Kafka servers that communicate over SSL (only).
Why is this happening / how can I fix it?
server.properties
listeners=SSL://some_host_name.corp.com:9092
inter.broker.listener.name=SSL
ssl.keystore.location=/some/path/to/keystore.p12
ssl.keystore.password=***
ssl.key.password=***
ssl.truststore.location=/some/path/to/server.truststore.jks
ssl.truststore.password=***
ssl.enabled.protocols=TLSv1.2
ssl.client.auth=required
ssl.keystore.type=PKCS12
ssl.truststore.type=JKS
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# Kafka is running with -Djavax.net.debug=ssl
# Assume localhost IP address is 1.2.3.4
18:31:29.479 [data-plane-kafka-socket-acceptor-ListenerName(SSL)-SSL-9092] DEBUG kafka.network.Acceptor - Accepted connection from /1.2.3.4:46732 on /1.2.3.4:9092 and assigned it to processor 1, sendBufferSize [actual|requested]: [212992|1048576] recvBufferSize [actual|requested]: [212992|1048576]
18:31:29.479 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] DEBUG kafka.network.Processor - Processor 1 listening to new connection from /1.2.3.4:46732
Using SSLEngineImpl.
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, fatal error: 80: problem unwrapping net record
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, SEND TLSv1.2 ALERT: fatal, description = internal_error
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, WRITE: TLSv1.2 Alert, length = 2
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeOutbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, closeOutboundInternal()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeInbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
18:31:29.481 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] DEBUG o.a.k.c.network.SslTransportLayer - [SslTransportLayer channelId=1.2.3.4:9092-1.2.3.4:46732-108 key=sun.nio.ch.SelectionKeyImpl#33a4cffe] SSLEngine.closeInBound() raised an exception.
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[na:1.8.0_181]
at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561) ~[na:1.8.0_181]
at org.apache.kafka.common.network.SslTransportLayer.handshakeFailure(SslTransportLayer.java:871) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.SslTransportLayer.maybeProcessHandshakeFailure(SslTransportLayer.java:909) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:295) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) [kafka-clients-2.7.0.jar:na]
at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [kafka-clients-2.7.0.jar:na]
at kafka.network.Processor.poll(SocketServer.scala:923) [kafka_2.12-2.7.0.jar:na]
at kafka.network.Processor.run(SocketServer.scala:826) [kafka_2.12-2.7.0.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
18:31:29.481 [data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1] INFO o.a.kafka.common.network.Selector - [SocketServer brokerId=1] Failed authentication with /1.2.3.4 (SSL handshake failed)
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, called closeOutbound()
data-plane-kafka-network-thread-1-ListenerName(SSL)-SSL-1, closeOutboundInternal()
Elsewhere in the log of another cluster member, authentication succeeds. So why does it work between different nodes, but not from one node to itself?
20:07:39.479 [data-plane-kafka-network-thread-3-ListenerName(SSL)-SSL-1] DEBUG o.a.k.c.network.SslTransportLayer - [SslTransportLayer channelId=1.2.3.6:9092-1.2.3.5:41182-0 key=sun.nio.ch.SelectionKeyImpl#69864b7d] SSL handshake completed successfully with peerHost '1.2.3.5' peerPort 41182 peerPrincipal '...' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384'
20:07:39.480 [data-plane-kafka-network-thread-3-ListenerName(SSL)-SSL-1] DEBUG o.a.kafka.common.network.Selector - [SocketServer brokerId=3] Successfully authenticated with /1.2.3.5

I have solved the issue by moving the SSL listener away from the default port of 9092.
This leaves me with a different question: if I have configured only one listener, with SSL, on port 9092, why is there any PLAINTEXT activity on that same port?
The errors shown above were caused by each broker trying to connect to itself on port 9092, probably using PLAINTEXT, even though I had configured only SSL.
Also: what is the difference between settings (1) and (2) below?
# (1) The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
listeners=SSL://$auto_host:9092
# (2) The port the socket server listens on
port=9092

Related

ktor how to hide javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

I'm developing a web app, and the app uses navigator.mediaDevices.getUserMedia, so it needs HTTPS.
So I created a dummy certificate and localhost KTOR loads it.
Which is great! Except it litters the logs with
2022-10-05 10:26:10.046 [eventLoopGroupProxy-3-4] WARN i.n.h.s.ApplicationProtocolNegotiationHandler - [id: 0x96c37772, L:0.0.0.0/0.0.0.0:8443] TLS handshake failed:
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
Is there any way to keep those from spamming the log?
(either through an improved self-signed cert, or through a filter, or a ktor config of "I get it, stop bothering me...")

Cannot connect to Kafka over SSL from Internet, but works from within the Kafka pod

I have deployed Kafka strimzi to my Kubernetes cluster.
My goal is to be able to connect to Kafka topic over SSL listener(port 9094) and be able to produce/consume messages.
Symptoms
[2022-05-05 10:20:11,252] ERROR Error when sending message to topic public.kafka.incoming.events with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Topic public.kafka.incoming.events not present in metadata after 60000 ms.
The message is not effectively sent by the producer or received by the consumer.
logs of strimzi-cluster-operator pod when selecting loadbalancer or ingress as type (i.e.: kafka.listeners.external.type).
failed to reconcile ZookeeperScalingException: Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.my-kafka-project.svc:2181. Connection was not ready in 300000 ms.
Not sure if this is related with my Kubernetes cluster configuration problem. When I select NodePort - it works as expected and kafka + zookeper are started normally.
To make this story short, I have:
1. Activated the debugging flag:
log4j.logger.org.apache.kafka=INFO -> log4j.logger.org.apache.kafka=DEBUG
and I am able to read the logs from Kafka server.
2. Configured/generated the signed certificate, keystore/truststore JKs and the configuration I use looks like:
cat <<EOT > ssl.properties
security.protocol=SSL
ssl.protocol=TLSv1.2
bootstrap.servers=my-cluster-kafka-external-bootstrap:9094
ssl.truststore.location=/home/kafka/user-truststore.jks
ssl.truststore.password=<password>
ssl.keystore.location=/home/kafka/user-keystore.jks
ssl.keystore.password=<password>
ssl.key.password=<password>
ssl.endpoint.identification.algorithm=
EOT
3. Distinctions between case #1 which works and case #2 which does not work and is why I am asking this question.
Case 1: This is the case which works normally, I am connecting to Kafka topics from within the pod which is running Kafka server.
Case 2: In the second case, which is not working as expected, the connecting goes to a load-balancer(i.e which is exposed to the internet).
The load-balancer connection is then established through a private network (10.20.10.0/24) which communicates with the Kubernetes cluster.
Case 1 (which works, and is why I have put here to help solve this problem)
Testing from a Kafka server pod (within the cluster).
Producer
[kafka#my-cluster-kafka-0 ~]$ /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-external-bootstrap:9094 --topic public.kafka.incoming.events --producer.config consumer-ssl.properties
>ttesttt
>
Consumer
[kafka#my-cluster-kafka-0 ~]$ /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-0.my-cluster-kafka-brokers.my-kafka-project.svc.cluster.local:9094 --topic public.kafka.incoming.events --consumer.config consumer-ssl.properties
ttesttt
Confirmation
Confirmation based on the Kafka server logs.
When a Producer is connecting successfully
2022-05-05 13:40:07,988 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-xx.xxx.xx4.xx7:45150-42 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/xx.xxx.xx4.xx7:45150], selector=sun.nio.ch.EPollSelectorImpl#5c2628ee, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost 'xx.xxx.xx4.xx7' peerPort 45150 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
2022-05-05 13:40:07,989 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /xx.xxx.xx4.xx7 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
When a Consumer is connecting successfully
2022-05-05 13:43:56,053 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-10.15.19.76:40094-43 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/10.15.19.76:40094], selector=sun.nio.ch.EPollSelectorImpl#7df0ddcc, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost '10.15.19.76' peerPort 40094 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-10]
2022-05-05 13:43:56,053 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /10.15.19.76 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-10]
2022-05-05 13:43:56,323 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-xx.xxx.xx4.xx7:1270-43 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/xx.xxx.xx4.xx7:1270], selector=sun.nio.ch.EPollSelectorImpl#323eac4e, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost 'xx.xxx.xx4.xx7' peerPort 1270 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-11]
2022-05-05 13:43:56,323 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /xx.xxx.xx4.xx7 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-11]
2022-05-05 13:43:56,335 INFO [GroupCoordinator 0]: Dynamic member with unknown member id joins group console-consumer-27420 in Empty state. Created a new member id console-consumer-f9cb320c-a5cf-4e40-a686-6dbd1d37cdd6 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-3]
2022-05-05 13:43:56,340 INFO [GroupCoordinator 0]: Preparing to rebalance group console-consumer-27420 in state PreparingRebalance with old generation 0 (__consumer_offsets-26) (reason: Adding new member console-consumer-f9cb320c-a5cf-4e40-a686-6dbd1d37cdd6 with group instance id None) (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-6]
2022-05-05 13:43:59,342 INFO [GroupCoordinator 0]: Stabilized group console-consumer-27420 generation 1 (__consumer_offsets-26) with 1 members (kafka.coordinator.group.GroupCoordinator) [executor-Rebalance]
2022-05-05 13:43:59,359 INFO [GroupCoordinator 0]: Assignment received from leader console-consumer-f9cb320c-a5cf-4e40-a686-6dbd1d37cdd6 for group console-consumer-27420 for generation 1. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-7]
2022-05-05 13:43:59,468 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-xx.xxx.xx4.xx7:1584-43 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/xx.xxx.xx4.xx7:1584], selector=sun.nio.ch.EPollSelectorImpl#5c2628ee, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost 'xx.xxx.xx4.xx7' peerPort 1584 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
2022-05-05 13:43:59,468 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /xx.xxx.xx4.xx7 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
Case 2 (which does not work, and is why I am asking this question)
Testing from my local machine (connecting through Internet).
Producer
$ /opt/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --bootstrap-server internet-reachable-domain-name:9094 --topic public.kafka.incoming.events --producer.config consumer-ssl.properties
....long SSL/TLS hand-shake....
)
ttesttttt
javax.net.ssl|ALL|16|kafka-producer-network-thread | console-producer|2022-05-05 10:05:04.469 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
Notice how the character '>' is not getting displayed on the first line.
Consumer
$ /opt/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server internet-reachable-domain-name:9094 --topic public.kafka.incoming.events --consumer.config consumer-ssl.properties
javax.net.ssl|DEBUG|01|main|2022-05-05 10:08:29.034 EDT|Finished.java:535|Consuming server Finished handshake message (
"Finished": {
"verify data": {
0000: 6A 3A 3D F6 87 81 A0 36 FF 08 77 CB
}'}
)
javax.net.ssl|ALL|01|main|2022-05-05 10:08:37.815 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 10:08:48.159 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 10:09:06.713 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 10:09:37.193 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 10:10:11.936 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
Note how the message is not reaching, and even if I type a message from a Producer of the Case #1, its not showing on this consumer.
Confirmation
Confirmation based on the Kafka server logs.
When a Producer is connecting
2022-05-05 14:14:43,507 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-10.15.30.64:19952-46 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/10.15.30.64:19952], selector=sun.nio.ch.EPollSelectorImpl#5c2628ee, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost '10.15.30.64' peerPort 19952 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
2022-05-05 14:14:43,507 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /10.15.30.64 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-12]
When a Consumer is connecting
2022-05-05 14:16:02,620 DEBUG [SslTransportLayer channelId=10.15.19.76:9094-10.15.219.64:42153-47 key=channel=java.nio.channels.SocketChannel[connected local=/10.15.19.76:9094 remote=/10.15.219.64:42153], selector=sun.nio.ch.EPollSelectorImpl#7df0ddcc, interestOps=1, readyOps=0] SSL handshake completed successfully with peerHost '10.15.219.64' peerPort 42153 peerPrincipal 'CN=my-user' cipherSuite 'TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384' (org.apache.kafka.common.network.SslTransportLayer) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-10]
2022-05-05 14:16:02,621 DEBUG [SocketServer listenerType=ZK_BROKER, nodeId=0] Successfully authenticated with /10.15.219.64 (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(EXTERNAL-9094)-SSL-10]
Other information
I am using the option:
export KAFKA_OPTS="-Djavax.net.debug=ssl:handshake:verbose"
to get more information from Kafka clients.
This is what I can see from the producer in case #2.
javax.net.ssl|ALL|01|main|2022-05-05 09:54:50.374 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:55:23.285 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:55:48.477 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:56:13.718 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:56:50.532 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:57:25.793 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:58:01.707 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:58:30.806 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|ALL|01|main|2022-05-05 09:59:07.867 EDT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
Kafka version
Both clients are identical:
# Local device
$ /opt/kafka_2.13-3.1.0/bin/kafka-topics.sh --version
3.1.0 (Commit:37edeed0777bacb3)
# Kafka server pod
$ /opt/kafka/bin/kafka-topics.sh --version
3.1.0 (Commit:37edeed0777bacb3)
Thanks #OneCricketeer for the information.
The links you provided were the exact one I had to look. The problem was that Ingress option or Load-Balancer did not work on my Kubernetes Cluster, Kafka was not starting. Only nodePort option worked, but that was not sufficient to address the problem I had.
I ended up using bitnami instead of strimzi, https://bitnami.com/stack/kafka/helm and it worked like charm.
I relied on the example provided by #Rubber Duck on Accessing bitnami/kafka outside the kubernetes cluster
I just had to make sure to forward the ports by the load-balancer.
command:
helm install kafka-bitnami --values bitnami-values-external.yaml bitnami/kafka
file: bitnami-values-external.yaml
replicaCount: 3
externalAccess:
enabled: true
autoDiscovery:
enabled: false
image:
registry: docker.io
repository: bitnami/kubectl
tag: 1.23.4-debian-10-r17
pullPolicy: IfNotPresent
pullSecrets: []
resources:
limits: {}
requests: {}
service:
type: NodePort
port: 9094
loadBalancerIPs: []
loadBalancerSourceRanges: []
nodePorts:
- 30000
- 30001
- 30002
useHostIPs: false
annotations: {}
domain: <domain-name which points to my load-balancer public IP>
Note that replicaCount must much the number of nodePorts items.
load balancer configs
The load balancer config will have something like:
frontend-1 port: 30000 -> backend-1 port: 30000, server ips: [set of private worker node ips]
frontend-1 port: 30001 -> backend-1 port: 30001, server ips: [set of private worker node ips]
frontend-1 port: 30002 -> backend-1 port: 30002, server ips: [set of private worker node ips]
Maybe from here, I could come back to strimzi and try to adjust/fix it because I understand little bit better how it works.
Well, Strimzi is a great platform to run Kafka on Kubernetes. As you haven't shared the kafka CR file, I am assuming you have done the following setup to access Kafka from outside the k8s cluster-
# ...
listeners:
# ...
- name: external
port: 9094
type: loadbalancer
tls: true
# ...
Make sure you have resources like the below-
With this configuration and proper Authn/Authz, you would be able to access Kafka from outside the k8s cluster.
Check this article (PS - written by me), which takes a dig on how to enable Authn/Authz in Strimzi and it also discusses Accessing Kafka from inside the k8s cluster and from outside of it.

Kafka inter broker SSL handshake failed

I am trying to setup inter-broker SSL (not client) authentication and keep seeing the following errors:
[2019-05-17 06:33:47,151] INFO [Controller id=1004, targetBrokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] INFO [SocketServer brokerId=1004] Failed authentication with /$IP (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2019-05-17 06:33:47,151] ERROR [Controller id=1004, targetBrokerId=1004] Connection to node 1004 (/$IP:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
My server.properties is:
listeners=PLAINTEXT://$IP:9092,SSL://$IP:9093
security.inter.broker.protocol=SSL
ssl.truststore.password=$PASS
ssl.keystore.password=$PASS
ssl.key.password=$PASS
ssl.endpoint.identification.algorithm=""
ssl.keystore.location=/etc/kafka/kafka.server.keystore.jks
ssl.truststore.location=/etc/kafka/kafka.server.truststore.jks
``
When I run `openssl s_client -debug -connect $IP:9093 -tls1` I get back a list of certificates and `Secure Renegotiation IS supported`
Despite adding `-Djavax.net.debug=all` there's not anything in the logs which points to the problem.
Kafka version 2.2
Any ideas?
I had incorrectly set the value of ssl.endpoint.identification.algorithm="" instead of ssl.endpoint.identification.algorithm", this fixed it.
This value was changed in 2.2 to default to https so setting it to nothing worked.

Incoming HTTP post over HTTPS causing exception

I have made a simple RESTlet that I have hosted on an Amazon EC2 instance.
I can reach this when i run it over HTTP.
I needed to convert this to working with HTTPS so followed the RESTlet guide at https://restlet.com/open-source/documentation/user-guide/2.2/core/security/https
When I connect to it over HTTP I get this error..
Restlet-917142466, READ: TLSv1.2 Alert, length = 2
Restlet-917142466, RECV TLSv1.2 ALERT: fatal, certificate_unknown
Restlet-917142466, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
Restlet-917142466, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
Restlet-917142466, called closeInbound()
Restlet-917142466, fatal: engine already closed. Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
Restlet-917142466, called closeOutbound()
Restlet-917142466, closeOutboundInternal()
The Java code is very simple and mirrors the sample RESTlet server in the guide.
Any suggestions on what I could try to fix this?

java6 HttpsUrlConnection error:javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

I write a demo to access a server with java HttpsUrlConnection.
I implement the X509TrustManager interface with MyX509TrustManager to trust all certificates.
It works fine with JDK7 and JDK8, but when use JDK6 and JDK5, error happens:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:882)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1215)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1199)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:434)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1014)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
For server side, running on Tomcat 8 and JDK8.
EDIT I add -Djavax.net.debug=ssl,handshake and find following:
*** main, WRITE: TLSv1 Handshake, length = 177 main, WRITE: SSLv2 client hello message, length = 173
main, received EOFException: error main, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
main, SEND TLSv1 ALERT: fatal, description = handshake_failure
main, WRITE: TLSv1 Alert, length = 2 main, called closeSocket()
main, called close() main, called closeInternal(true)