cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT pipeline access external location with abfss protocol was failed

harvey-c
New Contributor III

Dear Databricks Community Members:

The symptom: 

The DLT pipeline was failed with the error message:  

Failure to initialize configuration for storage account storageaccount.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.key Invalid configuration value detected for fs.azure.account.key

The context:

An external location was created and tested for the above storage location in a unity catalog. The workspace was enabled for the unity catalog.

The notebook to list or load a csv file in the above location is successful. I also noticed that the cluster to run the notebook must enable unity catalog.

DLT code is like this:

import dlt
@dlt.table(name="test_data")
def test_data():
    path="abfss://container@storageaccount.dfs.core.windows.net/test.csv"
    return (
        spark.read.csv(path,header=True,inferSchema=True)
    )

The above code can be verified in the notebook.

However, the above code was failed in DLT pipeline with key configuration issue.

Please advise if there is anything missing? 

Thanks.

 

 

 

 

 

 

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!