Apache Kafka with SSL is working but SSL errors against localhost in kafka log (drives me nuts) - ssl

Question

i have a Kafka problem, that drives me nuts.
We have a 4 nodes cluster. We used to work without SSL in our development stage. --> no problem.
For release, we switched on SSL for both listeners. --> Everything is working fine (application + kafka manager CMAK + Monitoring)
But we get one an error in our kafka broker serverlog in all environements (test, release, prod). Something is polling, and i don´t know what it is, or where to look:
It starts with:
[2020-10-16 10:50:27,866] INFO AdminClientConfig values:
bootstrap.servers = [127.0.0.1:10092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 120000
retries = 5
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
(org.apache.kafka.clients.admin.AdminClientConfig)
Then massive SSL error polling:
[2020-10-16 10:48:11,799] INFO [SocketServer brokerId=2] Failed authentication with /127.0.0.1 (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2020-10-16 10:48:13,141] INFO [SocketServer brokerId=2] Failed authentication with /127.0.0.1 (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2020-10-16 10:48:14,476] INFO [SocketServer brokerId=2] Failed authentication with /127.0.0.1 (SSL handshake failed) (org.apache.kafka.common.network.Selector)
Then Timeout:
[2020-10-16 10:48:20,890] INFO [AdminClient clientId=adminclient-25] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[2020-10-16 10:48:20,892] INFO [AdminClient clientId=adminclient-25] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
After 1-2 minutes, it starts again.
Our Broker config:
# Maintained by Ansible
zookeeper.connect=ZOOKEEPER1:2181,ZOOKEEPER2:2181,ZOOKEEPER3:2181
log.dirs=KAFKKALOGDIR
broker.id=2
confluent.license.topic.replication.factor=3
log.segment.bytes=1073741824
socket.receive.buffer.bytes=102400
socket.send.buffer.bytes=102400
offsets.topic.replication.factor=3
num.network.threads=8
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
confluent.support.metrics.enable=False
zookeeper.connection.timeout.ms=18000
num.io.threads=16
socket.request.max.bytes=104857600
log.retention.check.interval.ms=300000
group.initial.rebalance.delay.ms=0
confluent.metadata.topic.replication.factor=3
num.recovery.threads.per.data.dir=2
default.replication.factor=3
num.partitions=10
log.retention.hours=168
confluent.support.customer.id=anonymous
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
listeners=INTERNAL://:10091,EXTERNAL://:10092
advertised.listeners=INTERNAL://BROKERURL:10091,EXTERNAL://BROKERURL:10092
## Inter Broker Listener Configuration
inter.broker.listener.name=INTERNAL
listener.name.internal.ssl.truststore.location=LOCATION
listener.name.internal.ssl.truststore.password=PASSWORD
listener.name.internal.ssl.keystore.location=LOCATION
listener.name.internal.ssl.keystore.password=PASSWORD
listener.name.internal.ssl.key.password=PASSWORD
listener.name.external.ssl.truststore.location=LOCATION
listener.name.external.ssl.truststore.password=PASSWORD
listener.name.external.ssl.keystore.location=LOCATION
listener.name.external.ssl.keystore.password=PASSWORD
listener.name.external.ssl.key.password=PASSWORD
## Metrics Reporter Configuration
confluent.metrics.reporter.security.protocol=SSL
confluent.metrics.reporter.ssl.truststore.location=LOCATION
confluent.metrics.reporter.ssl.truststore.password=PASSWORD
What i did:
-disabled our monitoring agent (thought the agent is polling without SSL) --> nothing
-Add an additional localhost listener with PLAINTEXT 127.0.0.1 --> got massive problems with error "no matching leader for topic XY"
So, i don´t know how to continue - maybe someone has an idea
many thanks

Answer 1

Your AdminClientConfig specifies security.protocol=PLAINTEXT - that doesn't seem right given you want to enable SSL. https://kafka.apache.org/11/javadoc/org/apache/kafka/common/security/auth/SecurityProtocol.html shows the possible options for that variable.
You also have sasl.jaas.config=null, which I don't believe is correct either.
https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/ provides a good walk-through of how to setup security with Kafka.
Edited to add (in response to followup question): AdminClient is a Java class which is instantiated from your connect-distributed.properties file. When you search your service logfile for AdminClient you will see something like this:
[2020-09-16 02:58:14,180] INFO AdminClientConfig values:
bootstrap.servers = [servernames-elided:9092]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 20000
retries = 2147483647
retry.backoff.ms = 500
sasl.client.callback.handler.class = null
sasl.jaas.config = [hidden]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = PLAIN
security.protocol = SASL_SSL
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
(org.apache.kafka.clients.admin.AdminClientConfig:347)
Note that sasl.jaas.config = [hidden] - that's because the username and password for accessing the cluster are stored in the properties file directly, like so:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"someUsernameThisIs\" password=\"notMyRealPassword\";
Note that the escaping for the doublequotes is necessary for the configuration parser.