Databricks Community

DataEnginerrOO1 · ‎02-19-2025

I have an issue when trying to use the command display(dbutils.fs.ls("abfss://test@test.dfs.core.windows.net")). When I execute the command on my personal cluster, it works, and I can see the files. Before that, I set the following configurations:

spark.conf.set("fs.azure.account.auth.type.test.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.test.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.test.dfs.core.windows.net", client_id)
spark.conf.set("fs.azure.account.oauth2.client.secret.test.dfs.core.windows.net", client_secret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.test.dfs.core.windows.net", f"https://login.microsoftonline.com/{tenant_id}/oauth2/token")

However, when I perform the same action on a serverless environment, I get the following error:

Configuration fs.azure.account.auth.type.test.dfs.core.windows.net is not available. SQLSTATE: 42K0I

How can I access files stored in Data Lake with serverless?

Thank you.

Alberto_Umana · ‎02-19-2025

Hi @DataEnginerrOO1,

Can you try with this configurations set at notebook level? also ensure that variables values are correct:

service_credential = dbutils.secrets.get(scope="<secret-scope>",key="<service-credential-key>")

spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net", "<application-id>")

spark.conf.set("fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net", service_credential)

spark.conf.set("fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

https://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage

Set Spark Properties in SQL Warehouse Settings:

Instead of setting them directly in your notebook, configure these properties in the SQL warehouse settings if you are working in Databricks SQL, or adjust the serverless cluster configuration.

DataEnginerrOO1 · ‎02-19-2025

This settings are special for serverless? because as i said, the configurations above work with personal cluster.

Rjdudley · ‎02-20-2025

> This settings are special for serverless? because as i said, the configurations above work with personal cluster.

Yes, personal compute uses classic compute, which runs in your Azure subscription. Serverless runs in Databricks' cloud estate and needs special permissions, see https://docs.databricks.com/aws/en/security/network/#secure-network-connectivity

Rjdudley · ‎02-19-2025

Can your serverless compute access any storage in that storage account? Something else to check is if your NCC is configured correctly: Configure private connectivity from serverless compute - Azure Databricks | Microsoft Learn. However, if your serverless can access other storage in that account, the NCC is probably working.