topic Spark readStream kafka.ssl.keystore.location abfss path in Data Engineering

Spark readStream kafka.ssl.keystore.location abfss path

mwoods — Mon, 18 Sep 2023 16:29:39 GMT

Similar to https://community.databricks.com/t5/data-engineering/kafka-unable-to-read-client-keystore-jks/td-p/23301 - the documentation (https://learn.microsoft.com/en-gb/azure/databricks/structured-streaming/kafka#use-ssl-to-connect-azure-databricks-to-kafka) recommends that certificates for authenticating with kafka be kept in cloud storage, and the example appears to hint that it should be possible to read directly from that location...but in practice, it appears that spark is unable to read from abfss paths directly.

Setting kafka.ssl.keystore.location and kafka.ssl.truststore.location to abfss paths, for me, results in:

kafkashaded.org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
...
Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore abfss://{container}@{account}.dfs.core.windows.net/client.keystore.p12 of type PKCS12
...
Caused by: java.nio.file.NoSuchFileException: abfss:/{container}@{account}.dfs.core.windows.net/client.keystore.p12

paths have been double-checked as correct, and the account has read permission granted to the external location.

Can we get confirmation that reading direct is not possible, and if the recommendation is to copy the file(s) to a local temp path first, and referencing those paths in the kafka.ssl.*.location config options? Or should it be possible to read directly from abfss paths?

Re: Spark readStream kafka.ssl.keystore.location abfss path

mwoods — Fri, 22 Sep 2023 14:16:59 GMT

Hi @Retired_mod - thanks for your response.

Not sure what's happened here...this is now working for me, so either the issue has been patched, or the issue was somehow related to my group management where the external location read permissions were mapped to a "Data Engineer" group that existed at the account level, but was not actually being properly mapped via databricks_mws_permission_assignment to the respective workspace (which I have now rectified).

...I think that's probably the case, though I'm not sure in that situation why I was able to copy the file successfully via dbutils.fs.cp as a workaround until now (as that seemed to imply that I did have access to read the file).

Re: Spark readStream kafka.ssl.keystore.location abfss path

mwoods — Fri, 22 Sep 2023 19:09:59 GMT

@Retired_mod- quick update - managed to find the cause. It's neither of the above, it's a bug in the DataBricks 14.0 runtime. I had switched back to the 13.3 LTS runtime, and that is what caused the error to disappear.

As soon as I try to read directly from abfss paths using a compute resource with the DBR 14.0 runtime, I get the error again, so it appears a bug has been introduced in that release.