topic Access for delta lake with serverless in Data Engineering

Access for delta lake with serverless

DataEnginerrOO1 — Wed, 19 Feb 2025 11:43:11 GMT

I have an issue when trying to use the command display(dbutils.fs.ls("abfss://test@test.dfs.core.windows.net")). When I execute the command on my personal cluster, it works, and I can see the files. Before that, I set the following configurations:

spark.conf.set("fs.azure.account.auth.type.test.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.test.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.test.dfs.core.windows.net", client_id)
spark.conf.set("fs.azure.account.oauth2.client.secret.test.dfs.core.windows.net", client_secret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.test.dfs.core.windows.net", f"https://login.microsoftonline.com/{tenant_id}/oauth2/token")

However, when I perform the same action on a serverless environment, I get the following error:

Configuration fs.azure.account.auth.type.test.dfs.core.windows.net is not available. SQLSTATE: 42K0I

How can I access files stored in Data Lake with serverless?

Thank you.

Re: Access for delta lake with serverless

Alberto_Umana — Wed, 19 Feb 2025 12:39:34 GMT

Hi @DataEnginerrOO1,

Can you try with this configurations set at notebook level? also ensure that variables values are correct:

service_credential = dbutils.secrets.get(scope="<secret-scope>",key="<service-credential-key>")

spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net", "<application-id>")

spark.conf.set("fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net", service_credential)

spark.conf.set("fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

https://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage

Set Spark Properties in SQL Warehouse Settings:

Instead of setting them directly in your notebook, configure these properties in the SQL warehouse settings if you are working in Databricks SQL, or adjust the serverless cluster configuration.

Re: Access for delta lake with serverless

Rjdudley — Wed, 19 Feb 2025 13:44:52 GMT

Can your serverless compute access any storage in that storage account? Something else to check is if your NCC is configured correctly: Configure private connectivity from serverless compute - Azure Databricks | Microsoft Learn. However, if your serverless can access other storage in that account, the NCC is probably working.

Re: Access for delta lake with serverless

DataEnginerrOO1 — Wed, 19 Feb 2025 16:35:27 GMT

Yes, i can access tables that stored in the catalog on the same data lake.

Re: Access for delta lake with serverless

DataEnginerrOO1 — Wed, 19 Feb 2025 16:37:49 GMT

This settings are special for serverless? because as i said, the configurations above work with personal cluster.

Re: Access for delta lake with serverless

Rjdudley — Thu, 20 Feb 2025 14:13:46 GMT

> This settings are special for serverless? because as i said, the configurations above work with personal cluster.

Yes, personal compute uses classic compute, which runs in your Azure subscription. Serverless runs in Databricks' cloud estate and needs special permissions, see https://docs.databricks.com/aws/en/security/network/#secure-network-connectivity