cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Error Reading Kafka message into Azure Databricks

SreeG
New Contributor II

Team

I am trying to test the connection to Kafka broker from Azure Databricks. Telnet and IP is successful.

When I am trying to read the data, I am getting "Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target" error.

For reference below is the code:

connString ="kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"uid\" password=\"pwd\";"

df = (spark
.readStream
.format("kafka")
.option("subscribe", "NOTIFICATIONS")
.option("kafka.bootstrap.servers", "brkip:9093")
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.mechanism", "PLAIN")
.option("kafka.sasl.jaas.config", connString)
.option("kafka.request.timeout.ms", "60000")
.option("kafka.session.timeout.ms", "30000")
.load())

display(df)

Am I missing any thing? Any suggestions?

Regards

Sree

2 REPLIES 2

feiyun0112
Contributor III

because you use SASL_SSL connect to Kafka, it need certificate file

https://docs.databricks.com/en/connect/streaming/kafka.html 

SreeG
New Contributor II

Had to hold on this testing. But, when I get a chance to work on this, I will update my findings. Thank you!

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!