Kafka unable to read client.keystore.jks.

[Jayanth746]

Below is the error we have received when trying to read the stream

Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore /dbfs/FileStore/Certs/client.keystore.jks

Caused by: java.nio.file.NoSuchFileException: /dbfs/FileStore/Certs/client.keyst

When trying to read a stream from Kafka, databricks is unable to find keystore files.

df = spark.readStream \

 .format("kafka") \

 .option("kafka.bootstrap.servers","kafka server with port") \

 .option("kafka.security.protocol", "SSL") \

 .option("kafka.ssl.truststore.location",'/dbfs/FileStore/Certs/client.truststore.jks' ) \

 .option("kafka.ssl.keystore.location", '/dbfs/FileStore/Certs/client.keystore.jks') \

 .option("kafka.ssl.keystore.password", keystore_pass) \

 .option("kafka.ssl.truststore.password", truststore_pass) \

 .option("kafka.ssl.keystore.type", "JKS") \

 .option("kafka.ssl.truststore.type", "JKS") \

 .option("subscribe","sports")   \

 .option("startingOffsets", "earliest") \

 .load()

The file exists in the dbfs and also able to read the file.

[Debayan]

Hi @Jayanth Goulla​ , Does this works: kafka.ssl.keystore.type = PEM ?

Reference: https://docs.databricks.com/structured-streaming/kafka.html#use-ssl

[Jayanth746]

Hi @Debayan Mukherjee​ , Please see the results after using PEM as the keystore type.

Caused by: kafkashaded.org.apache.kafka.common.errors.InvalidConfigurationException: SSL key store password cannot be specified with PEM format, only key password may be specified

I have use the document posted in the chat to get this working.

Also if I use SASL_SSL as protocol I get the below error

Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set

The files are present in the dbfs

[Debayan]

You’ll have to construct JAAS file and pass with JVM option. Or, you can pass the content of JAAS as Kafka source option, say, dynamic JAAS config.https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+cli...

[Jayanth746]

Do I need to use JAAS even if I already have certificates for SSL connection?

I am only looking to establish SSL connection and not SASL.

[Jayanth746]

Hi @Debayan Mukherjee​ , This worked after using the absolute path

/dbfs/dbfs/FileStore/Certs/client.truststore.jks instead of just dbfs/FileStore/Certs/client.truststore.jks.

However, I need this to be working for a ADLS gen2 path.

[mwoods]

@Jayanth746did you have any luck with this eventually? Hitting the same issue - appears that spark isn't able to read from adls directly, but the docs are vague as to whether it should be possible. Looks like will probably have to copy them to a local path first.

[Jayanth746]

Hi @mwoods , I was unable to refer to ADLS path directly.

This is what I have done to get this working 

val keystore_location = adls_path + "/" + operator + "/certs/client.keystore.jks"

val dbfs_ks_location = "dbfs:/FileStore/"+ operator +"/Certs/client.keystore.jks"

dbutils.fs.cp(keystore_location,dbfs_ks_location)

.option("kafka.ssl.keystore.location","/"+dbfs_ks_location.replace(":",""))

[mwoods]

@Jayanth746- FYI, as of today, reading the keystore/truststore from abfss paths directly is now working for me, so may be worth a retry on your end.

Not sure whether it was fixed on the DataBricks side, or if it was down to a change of setup on my side. If you find it still doesn't work for you, assuming you have used an external location to access, double-check that the principal/grant mapping in there is correct.

[mwoods]

Ok, scrub that - the problem in my case was that I was using the 14.0 databricks runtime, which appears to have a bug relating to abfss paths here. Switching back to the 13.3 LTS release resolved it for me. So if you're in the same boat finding abfss paths in kafka.ssl.keystore.location and kafka.ssl.truststore.location are failing, try switching back to 13.3 LTS.