Description:
On one POC, we are trying to connect to Heroku Kafka from the Databricks Workspace.
On the Heroku Kafka, the configuration details for connection are given in the form of 4 variables as below:
-------Connection Variables————
KAFKA_CLIENT_CERT (in .pem format)
KAFKA_CLIENT_CERT_KEY (in .pem format)
KAFKA_TRUSTED_CERT (in .pem format)
KAFKA_URL (kafka+ssl:// <<broker1:port>>, kafka+ssl:// <<broker2:port>>..........)
-------Connection Variables—————
On the Databricks side, we are trying to connect using the below scala code.
--Code--
import org.apache.spark.sql.functions.{get_json_object, json_tuple}
var streamingInputDF =
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "kafka+ssl://host1:port1,kafka+ssl://host2:port2")
.option("kafka.security.protocol","ssl")
.option("kafka.ssl.truststore.location", "<<truststore.jks location on dbfs>>")
.option("kafka.ssl.truststore.password", "<<password>>")
.option("kafka.ssl.keystore.location", "<<keystore.jks location on dbfs>>")
.option("kafka.ssl.keystore.password", "<<password>>")
.option("subscribe", "<<topic_name>>")
.option("startingOffsets", "latest")
.option("minPartitions", "10")
.option("failOnDataLoss", "true")
.load()
display(streamingInputDF)
--Code—
Error that I am getting is given below:
---Error-----
---Error——
What have I tried so far?
I have created the jks file using the blog https://blogs.mulesoft.com/dev-guides/api-connectors-templates/how-to-connect-to-apache-kafka-on-her....
I have tried executing the notebook placing the jks file on both the S3 bucket with mount point on DBFS as well as placing the certificates in the dbfs locations like -
- /FileStore/tables/certificates/keystore.jks
- /FileStore/tables/certificates/truststore.jks
But even after placing the jks files on both storage options, I am getting the error “Failed to load SSL keystore /FileStore/tables/certificates/keystore.jks of type JKS.
Kindly help in resolving this issue.