How to use Azure Data lake as a storage location to store the Delta Live Tables?

[SM]

I am trying write data into Azure Datalake. I am reading files from Azure Blob Storage however when I try to create the Delta Live Table to Azure Datalake I get error the following error

shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException: Invalid configuration value detected for fs.azure.account.key

I also tried to create the DLT in Azure Blob Storage, but it doesnt seem to recognize the container in the Azure Storage Account

I just specify the storage location via the DLT UI, not sure if there is any additional parameters to be configured to make it work.

[Anonymous]

Hi there,

My name is Piper, and I'm a moderator for Databricks. Welcome to the community! Thank you for your question. Let's give your peers a chance to respond before we circle back to this.

Thank you for your patience!

[Kaniz]

Hi @Shikha Mathew​ , If your sink data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from Azure Databricks Delta table to sink.

Please go through this link for the step-by-step process.

[RThornton]

@Kaniz Fatma​ I don't think you quite understand the question. I'm running into the same problem. When creating a Delta Live Table pipeline to write to Azure Data Lake Storage (abfss://etc...) as the Storage Location, the pipeline fails with the error @Shikha Mathew​ mentioned in the original post.

Typically, when you're creating a cluster to process a regular notebook, you have to configure the various OAuth configurations in the cluster's advanced settings. I've tried adding those configurations to the Delta Live Table pipeline, but I get the same error.

Also, we're not using the fs.azure.account.key settings since we're using OAuth authentication to ADLS.

Any suggestions?

[evogelpohl]

Same issue using abfss//... path and using it for the dlt pipeline storage location. @Robert Thornton​ , i mounted the abfss path as '/mnt/data' using the same service principal and secret as I used when i configured the oauth. I then changed my dlt pipelines & storage location to use /mnt/data/..path.. and it worked.