topic Re: Connecting to Blob storage using abfss not working with serverless compute in Data Engineering

Connecting to Blob storage using abfss not working with serverless compute

Mathias — Fri, 04 Oct 2024 10:39:55 GMT

I tried to follow the instructions found here: Connect to Azure Data Lake Storage Gen2 and Blob Storage - Azure Databricks | Microsoft Learn

E.g. this code:

spark.conf.set(
"fs.azure.account.key.<storage-account>.dfs.core.windows.net",
dbutils.secrets.get(scope="<scope>", key="<storage-account-access-key>"))

Doesn't work and gives error:

Configuration fs.azure.account.key.mystorageaccount.dfs.core.windows.net is not available.

Is there any plans to support this with serverless compute?

Re: Connecting to Blob storage using abfss not working with serverless compute

szymon_dybczak — Fri, 04 Oct 2024 11:57:28 GMT

Hi @Mathias ,

This is legacy way of configuring storage? Is there any reason why you're not using Unity Catalog storage credentials?

Re: Connecting to Blob storage using abfss not working with serverless compute

Mathias — Fri, 04 Oct 2024 12:10:13 GMT

Can you point me to some documentation on how to do that?

Re: Connecting to Blob storage using abfss not working with serverless compute

szymon_dybczak — Fri, 04 Oct 2024 15:23:11 GMT

Hi @Mathias ,

Sure, here it is:

https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/