Spark readStream kafka.ssl.keystore.location abfss path

mwoods
New Contributor III

‎09-18-2023 09:29 AM

Similar to https://community.databricks.com/t5/data-engineering/kafka-unable-to-read-client-keystore-jks/td-p/2...- the documentation (https://learn.microsoft.com/en-gb/azure/databricks/structured-streaming/kafka#use-ssl-to-connect-azu...) recommends that certificates for authenticating with kafka be kept in cloud storage, and the example appears to hint that it should be possible to read directly from that location...but in practice, it appears that spark is unable to read from abfss paths directly.

Setting kafka.ssl.keystore.location and kafka.ssl.truststore.location to abfss paths, for me, results in:

kafkashaded.org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
...
Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore abfss://{container}@{account}.dfs.core.windows.net/client.keystore.p12 of type PKCS12
...
Caused by: java.nio.file.NoSuchFileException: abfss:/{container}@{account}.dfs.core.windows.net/client.keystore.p12

paths have been double-checked as correct, and the account has read permission granted to the external location.

Can we get confirmation that reading direct is not possible, and if the recommendation is to copy the file(s) to a local temp path first, and referencing those paths in the kafka.ssl.*.location config options? Or should it be possible to read directly from abfss paths?

0 Kudos