Databricks

mwoods · 09-22-2023

We have a spark streaming job that consumes data from a Kafka topic and writes out to delta tables in Unity Catalog.Looking to refactor it to use Delta Live Tables, but it appears that it is not possible at present to have a DLT Pipeline that can acc...

mwoods · 09-20-2023

I have an external location setup "auth_kafka" which is mapped to an abfss url:abfss://{container}@{account}.dfs.core.windows.net/auth/kafkaand, critically, is marked as readonly.Using dbutils.fs I can successfully read the files (i.e. the ls and hea...

mwoods · 09-18-2023

Similar to https://community.databricks.com/t5/data-engineering/kafka-unable-to-read-client-keystore-jks/td-p/23301 - the documentation (https://learn.microsoft.com/en-gb/azure/databricks/structured-streaming/kafka#use-ssl-to-connect-azure-databricks...

mwoods · 09-22-2023

Ok, scrub that - the problem in my case was that I was using the 14.0 databricks runtime, which appears to have a bug relating to abfss paths here. Switching back to the 13.3 LTS release resolved it for me. So if you're in the same boat finding abfss...

mwoods · 09-22-2023

@Kaniz- quick update - managed to find the cause. It's neither of the above, it's a bug in the DataBricks 14.0 runtime. I had switched back to the 13.3 LTS runtime, and that is what caused the error to disappear.As soon as I try to read directly from...

mwoods · 09-22-2023

@Jayanth746- FYI, as of today, reading the keystore/truststore from abfss paths directly is now working for me, so may be worth a retry on your end.Not sure whether it was fixed on the DataBricks side, or if it was down to a change of setup on my sid...

mwoods · 09-22-2023

Hi @Kaniz - thanks for your response.Not sure what's happened here...this is now working for me, so either the issue has been patched, or the issue was somehow related to my group management where the external location read permissions were mapped to...

mwoods · 09-20-2023

@Kaniz Sorry...not sure how that's relevant? Was that posted to the wrong topic?This question is in regards to what appears to be a bug in dbutils.fs where the cp function appears to require write access to the data source (as opposed to just read ac...

Databricks

User Stats

User Activity

Delta Live Tables error with Kafka SSL

dbutils.fs.cp requires write permissions on the source

Spark readStream kafka.ssl.keystore.location abfss path

Re: Kafka unable to read client.keystore.jks.

Re: Spark readStream kafka.ssl.keystore.location abfss path

Re: Kafka unable to read client.keystore.jks.

Re: Spark readStream kafka.ssl.keystore.location abfss path

Re: dbutils.fs.cp requires write permissions on the source