DLT pipeline access external location with abfss protocol was failed

harvey-c — Mon, 18 Dec 2023 22:50:04 GMT

Dear Databricks Community Members:

The symptom:

The DLT pipeline was failed with the error message:

Failure to initialize configuration for storage account storageaccount.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.key Invalid configuration value detected for fs.azure.account.key

The context:

An external location was created and tested for the above storage location in a unity catalog. The workspace was enabled for the unity catalog.

The notebook to list or load a csv file in the above location is successful. I also noticed that the cluster to run the notebook must enable unity catalog.

DLT code is like this:

import dlt

@dlt.table(name="test_data")

def test_data():

path="abfss://container@storageaccount.dfs.core.windows.net/test.csv"

return (

spark.read.csv(path,header=True,inferSchema=True)

)

The above code can be verified in the notebook.

However, the above code was failed in DLT pipeline with key configuration issue.

Please advise if there is anything missing?

Thanks.

topic DLT pipeline access external location with abfss protocol was failed in Get Started Discussions

DLT pipeline access external location with abfss protocol was failed