Error Reading Kafka message into Azure Databricks

SreeG — Wed, 08 May 2024 20:34:57 GMT

Team

I am trying to test the connection to Kafka broker from Azure Databricks. Telnet and IP is successful.

When I am trying to read the data, I am getting "Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" error.

For reference below is the code:

connString ="kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"uid\" password=\"pwd\";"

df = (spark
.readStream
.format("kafka")
.option("subscribe", "NOTIFICATIONS")
.option("kafka.bootstrap.servers", "brkip:9093")
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.mechanism", "PLAIN")
.option("kafka.sasl.jaas.config", connString)
.option("kafka.request.timeout.ms", "60000")
.option("kafka.session.timeout.ms", "30000")
.load())

display(df)

Am I missing any thing? Any suggestions?

Regards

Sree

Re: Error Reading Kafka message into Azure Databricks

feiyun0112 — Thu, 09 May 2024 01:41:57 GMT

because you use SASL_SSL connect to Kafka, it need certificate file

https://docs.databricks.com/en/connect/streaming/kafka.html

Re: Error Reading Kafka message into Azure Databricks

SreeG — Thu, 16 May 2024 16:20:11 GMT

Had to hold on this testing. But, when I get a chance to work on this, I will update my findings. Thank you!

topic Error Reading Kafka message into Azure Databricks in Data Engineering

Error Reading Kafka message into Azure Databricks

Re: Error Reading Kafka message into Azure Databricks

Re: Error Reading Kafka message into Azure Databricks