Connecting Databricks to Heroku Kafka

[Kaniz]

Description:

On one POC, we are trying to connect to Heroku Kafka from the Databricks Workspace. 

 On the Heroku Kafka, the configuration details for connection are given in the form of 4 variables as below: 

-------Connection Variables———— 

KAFKA_CLIENT_CERT (in .pem format) 

KAFKA_CLIENT_CERT_KEY (in .pem format) 

KAFKA_TRUSTED_CERT (in .pem format) 

KAFKA_URL (kafka+ssl:// <<broker1:port>>, kafka+ssl:// <<broker2:port>>..........) 

-------Connection Variables————— 

On the Databricks side, we are trying to connect using the below scala code.

--Code--

import org.apache.spark.sql.functions.{get_json_object, json_tuple}
 
 
 
var streamingInputDF = 
 
 spark.readStream
 
  .format("kafka")
 
  .option("kafka.bootstrap.servers", "kafka+ssl://host1:port1,kafka+ssl://host2:port2")
 
  .option("kafka.security.protocol","ssl")
 
  .option("kafka.ssl.truststore.location", "<<truststore.jks location on dbfs>>")
 
  .option("kafka.ssl.truststore.password", "<<password>>")
 
  .option("kafka.ssl.keystore.location", "<<keystore.jks location on dbfs>>")
 
  .option("kafka.ssl.keystore.password", "<<password>>")
 
  .option("subscribe", "<<topic_name>>")   
 
  .option("startingOffsets", "latest") 
 
  .option("minPartitions", "10") 
 
  .option("failOnDataLoss", "true")
 
  .load()
 
 
 
display(streamingInputDF)

--Code—

Error that I am getting is given below:

---Error-----

---Error——

What have I tried so far?

I have created the jks file using the blog https://blogs.mulesoft.com/dev-guides/api-connectors-templates/how-to-connect-to-apache-kafka-on-her....

I have tried executing the notebook placing the jks file on both the S3 bucket with mount point on DBFS as well as placing the certificates in the dbfs locations like - 

1. /FileStore/tables/certificates/keystore.jks
2. /FileStore/tables/certificates/truststore.jks

But even after placing the jks files on both storage options, I am getting the error “Failed to load SSL keystore /FileStore/tables/certificates/keystore.jks of type JKS.

Kindly help in resolving this issue.