Error Reading Kafka message into Azure Databricks

[SreeG]

Team

I am trying to test the connection to Kafka broker from Azure Databricks. Telnet and IP is successful.

When I am trying to read the data, I am getting "Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" error.

For reference below is the code:

connString ="kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"uid\" password=\"pwd\";"

df = (spark
.readStream
.format("kafka")
.option("subscribe", "NOTIFICATIONS")
.option("kafka.bootstrap.servers", "brkip:9093")
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.mechanism", "PLAIN")
.option("kafka.sasl.jaas.config", connString)
.option("kafka.request.timeout.ms", "60000")
.option("kafka.session.timeout.ms", "30000")
.load())

display(df)

Am I missing any thing? Any suggestions?

Regards

Sree

[feiyun0112]

because you use SASL_SSL connect to Kafka, it need certificate file

https://docs.databricks.com/en/connect/streaming/kafka.html 

[SreeG]

Had to hold on this testing. But, when I get a chance to work on this, I will update my findings. Thank you!