topic How to use Azure Data lake as a storage location to store the Delta Live Tables? in Data Engineering

How to use Azure Data lake as a storage location to store the Delta Live Tables?

SM — Thu, 20 Jan 2022 14:32:50 GMT

I am trying write data into Azure Datalake. I am reading files from Azure Blob Storage however when I try to create the Delta Live Table to Azure Datalake I get error the following error

shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException: Invalid configuration value detected for fs.azure.account.key

I also tried to create the DLT in Azure Blob Storage, but it doesnt seem to recognize the container in the Azure Storage Account

I just specify the storage location via the DLT UI, not sure if there is any additional parameters to be configured to make it work.

Re: How to use Azure Data lake as a storage location to store the Delta Live Tables?

Anonymous — Thu, 20 Jan 2022 16:40:58 GMT

Hi there,

My name is Piper, and I'm a moderator for Databricks. Welcome to the community! Thank you for your question. Let's give your peers a chance to respond before we circle back to this.

Thank you for your patience!

Re: How to use Azure Data lake as a storage location to store the Delta Live Tables?

RThornton — Wed, 05 Oct 2022 20:29:54 GMT

@Kaniz Fatma I don't think you quite understand the question. I'm running into the same problem. When creating a Delta Live Table pipeline to write to Azure Data Lake Storage (abfss://etc...) as the Storage Location, the pipeline fails with the error @Shikha Mathew mentioned in the original post.

Typically, when you're creating a cluster to process a regular notebook, you have to configure the various OAuth configurations in the cluster's advanced settings. I've tried adding those configurations to the Delta Live Table pipeline, but I get the same error.

Also, we're not using the fs.azure.account.key settings since we're using OAuth authentication to ADLS.

Any suggestions?

Re: How to use Azure Data lake as a storage location to store the Delta Live Tables?

evogelpohl — Fri, 25 Nov 2022 01:45:31 GMT

Same issue using abfss//... path and using it for the dlt pipeline storage location. @Robert Thornton , i mounted the abfss path as '/mnt/data' using the same service principal and secret as I used when i configured the oauth. I then changed my dlt pipelines & storage location to use /mnt/data/..path.. and it worked.