Databricks <-> Kafka - SSL handshake failed - ssl

Question

Below is the error we have received when trying to read the stream
Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore /dbfs/FileStore/Certs/client.keystore.jks
Caused by: java.nio.file.NoSuchFileException: /dbfs/FileStore/Certs/client.keyst
When trying to read a stream from Kafka, Databricks is unable to find keystore files.
df = spark.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers","kafka server with port") \
.option("kafka.security.protocol", "SSL") \
.option("kafka.ssl.truststore.location",'/dbfs/FileStore/Certs/client.truststore.jks' ) \
.option("kafka.ssl.keystore.location", '/dbfs/FileStore/Certs/client.keystore.jks') \
.option("kafka.ssl.keystore.password", keystore_pass) \
.option("kafka.ssl.truststore.password", truststore_pass) \
.option("kafka.ssl.keystore.type", "JKS") \
.option("kafka.ssl.truststore.type", "JKS") \
.option("subscribe","sports") \
.option("startingOffsets", "earliest") \
.load()
The file exists in the dbfs and also able to read the file.
We have also mounted the blob storage in datrbicks and tried to read the files from ADLS gen2.
The driver logs also has additional error: 22/11/04 12:18:07 ERROR DefaultSslEngineFactory: Modification time of key store could not be obtained.
We are trying to read a kafka stream by authenticating it using SSL keystores.
The connection doesn't seem to work as databricks is unable to view the keystores

Answer 1

I was able to access the key-store files by adding dbfs prefix to the original path.
so, instead of using the path /dbfs/FileStore/Certs/client.truststore.jks, I used /dbfs/dbfs/FileStore/Certs/client.truststore.jks. enter code here
But I am now receiving SSL handshake error even though the trust-store I have created is based on server certificate and the fingerprint in the certificate matches the trust-store fingerprint.
kafkashaded.org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed Caused by: java.security.cert.CertPathValidatorException: signature check failed Caused by: java.security.SignatureException: Signature does not match.