Getting Unrecognized SSL message, plaintext connection ERROR in docker image of kafka - ssl

Question

I'm trying to setup SSL in Kafka using docker image. I'm getting below weird ERROR after configuring. Since I'm unable to understand what's the issue and still figuring out what happening here. Any help highly appreciated.
I have generate kafka1_keystore.jks and truststore.jks keystore and added kafka1.crt (server certificate ) and root.crt (root certificate ) inside keystore of both jks.
Still SSL handshake is failing.
Error
I'm not sure which plaintext kafka is trying to connect using SSL.
Below is my zookeeper and kafka docker compose yml files.
zookeeper.yml
version: '3.7'
services:
zoo1:
image: confluentinc/cp-zookeeper:7.0.0
hostname: zoo1
ports:
- 2191:2181
networks:
- cms-overlay
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_SERVERS: zoo1:2888:3888
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 5
ZOOKEEPER_SYNC_LIMIT: 2
deploy:
replicas: 1
restart_policy:
condition: any
placement:
constraints:
- node.labels.kafka==kafka1
volumes:
- /var/lib/zookeeper/1/data:/var/lib/zookeeper/data
- /var/lib/zookeeper/1/logs:/var/lib/zookeeper/logs
- /var/lib/zookeeper/1/datalog:/var/lib/zookeeper/datalog
- /var/lib/zookeeper/1/log:/var/lib/zookeeper/log
networks:
cms-overlay:
external: true
kafka.yml file
version: '3.7'
services:
kafka1:
image: confluentinc/cp-enterprise-kafka:7.0.0
ports:
- 9192:9192
deploy:
replicas: 1
restart_policy:
condition: any
placement:
constraints:
- node.labels.kafka==kafka1
networks:
- cms-overlay
environment:
JAVA_OPTS : "-Xms6g -Xmx6g -XX:MetaspaceSize=96m -XX:+UseG1GC -
XX:MaxGCPauseMillis=20
-XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M
-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 "
KAFKA_BROKER_ID: 1
KAFKA_LISTENERS: ISSL://:29092,OSSL://:9192
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: ISSL:SSL,OSSL:SSL
KAFKA_INTER_BROKER_LISTENER_NAME: ISSL
KAFKA_ADVERTISED_LISTENERS:
ISSL://kafka1:29092,OSSL://172.16.41.11:9192
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_NUM_PARTITIONS: 3
KAFKA_METRIC_REPORTERS:
io.confluent.metrics.reporter.ConfluentMetricsReporter
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_DEFAULT_REPLICATION_FACTOR: 1
KAFKA_MIN_INSYNC_REPLICAS: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100
KAFKA_LOG4J_ROOT_LOGLEVEL: INFO
CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka1:29092
CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zoo1:2181
CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
CONFLUENT_METRICS_ENABLE: 'true'
CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'
KAFKA_SSL_KEYSTORE_FILENAME: kafka1_keystore.jks
KAFKA_SSL_KEYSTORE_CREDENTIALS: kafka_secret.txt
KAFKA_SSL_KEY_CREDENTIALS: kafka_secret.txt
KAFKA_SSL_TRUSTSTORE_FILENAME: truststore.jks
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: trust_secret.txt
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
KAFKA_LISTENER_NAME_INTERNAL_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
KAFKA_SSL_CLIENT_AUTH: required
KAFKA_INTER_BROKER_PROTOCOL: SSL
KAFKA_JMX_OPTS: "
-Djavax.net.ssl.trustStore=/etc/kafka/secrets/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.keyStore=/etc/kafka/secrets/kafka1_keystore.jks -Djavax.net.ssl.keyStorePassword=kafka123 -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true -Djavax.net.debug=all"
volumes:
- /var/lib/kafka/1/data:/var/lib/kafka/data
- /home/certs/san:/etc/kafka/secrets
networks:
cms-overlay:
external: true
In above docker compose yml files I have tried setting up multiple SSL listeners.