Prabakar
Databricks Employee
Databricks Employee

Please refer to the doc Connect to Azure Data Lake Storage Gen2 and Blob Storage - Azure Databricks | Microsoft Learn. This has the command that you can use to set the spark config from the notebook level.

service_credential = dbutils.secrets.get(scope="<secret-scope>",key="<service-credential-key>")

spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net", "<application-id>")
spark.conf.set("fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net", service_credential)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

Replace

  • <secret-scope> with the Databricks secret scope name.
  • <service-credential-key> with the name of the key containing the client's secret.
  • <storage-account> with the name of the Azure storage account.
  • <application-id> with the Application (client) ID for the Azure Active Directory application.
  • <directory-id> with the Directory (tenant) ID for the Azure Active Directory application.