Accessing TenantId via secret to connect to Azure Data Lake Storage Gen2 doesn't work

[kp12]

Hello,

I'm following instructions in this article to connect to ADLS gen2 using Azure service principal. I can access service principal's app id and secret via Databricks key vault backed secret scope. However, this doesn't work for directory-id and I have to hard code it and then the connection to storage works.

If we were to add this config on the cluster, is there a way to not hard code directory-id?

Thanks,

Kalyani

[kp12]

Hi @Retired_mod , Thanks for the prompt reply. As per the document, the syntax is the text highlighted in red below for accessing keys from secret scope in spark config. I used the same for app id too and that works. But I if use the same syntax for tenant id i.e., directory-id, it doesn't work. Hardcoding directory-id works.

I'm hardcoding storage account name too. Haven't tested accessing it via secret scope.

I'm adding this config on the cluster, so would dbutils.secrets.get work in spark config? 

fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net {{secrets/<secret-scope>/<service-credential-key>}}
fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net https://login.microsoftonline.com/<directory-id>/oauth2/token