Accessing TenantId via secret to connect to Azure Data Lake Storage Gen2 doesn't work

kp12 — Wed, 23 Aug 2023 10:41:50 GMT

Hello,

I'm following instructions in this article to connect to ADLS gen2 using Azure service principal. I can access service principal's app id and secret via Databricks key vault backed secret scope. However, this doesn't work for directory-id and I have to hard code it and then the connection to storage works.

If we were to add this config on the cluster, is there a way to not hard code directory-id?

Thanks,

Kalyani

Re: Accessing TenantId via secret to connect to Azure Data Lake Storage Gen2 doesn't work

kp12 — Wed, 23 Aug 2023 13:58:34 GMT

Hi @Retired_mod , Thanks for the prompt reply. As per the document, the syntax is the text highlighted in red below for accessing keys from secret scope in spark config. I used the same for app id too and that works. But I if use the same syntax for tenant id i.e., directory-id, it doesn't work. Hardcoding directory-id works.

I'm hardcoding storage account name too. Haven't tested accessing it via secret scope.

I'm adding this config on the cluster, so would dbutils.secrets.get work in spark config?

fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net {{secrets/<secret-scope>/<service-credential-key>}}
fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net https://login.microsoftonline.com/<directory-id>/oauth2/token

topic Accessing TenantId via secret to connect to Azure Data Lake Storage Gen2 doesn't work in Get Started Discussions

Accessing TenantId via secret to connect to Azure Data Lake Storage Gen2 doesn't work

Re: Accessing TenantId via secret to connect to Azure Data Lake Storage Gen2 doesn't work