Accessing TenantId via secret to connect to Azure Data Lake Storage Gen2 doesn't work

[kp12]

Hello,

I'm following instructions in this article to connect to ADLS gen2 using Azure service principal. I can access service principal's app id and secret via Databricks key vault backed secret scope. However, this doesn't work for directory-id and I have to hard code it and then the connection to storage works.

If we were to add this config on the cluster, is there a way to not hard code directory-id?

Thanks,

Kalyani

[Kaniz]

Hi @kp12, Yes, you can avoid hardcoding the directory id by storing it in Databricks secret scope, similar to storing the service principal's app id and secret. You can use the dbutils.secrets.get function to retrieve the directory id from the secret scope and use it in your Spark configuration. 

[kp12]

Hi @Kaniz , Thanks for the prompt reply. As per the document, the syntax is the text highlighted in red below for accessing keys from secret scope in spark config. I used the same for app id too and that works. But I if use the same syntax for tenant id i.e., directory-id, it doesn't work. Hardcoding directory-id works.

I'm hardcoding storage account name too. Haven't tested accessing it via secret scope.

I'm adding this config on the cluster, so would dbutils.secrets.get work in spark config? 

fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net {{secrets/<secret-scope>/<service-credential-key>}}
fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net https://login.microsoftonline.com/<directory-id>/oauth2/token

[Kaniz]

Hi @kp12, - You are trying to access a secret scope in your Spark configuration.

- You are facing issues when accessing the’ directory-id’ or’ tenant id’ using the same syntax as for’ app id’.
- The correct syntax to reference a secret in Spark configuration or environment variable is {{secrets/<scope-name>/<secret-name>}}.
- For example, if the secret scope is’ my_secret_scope’ and the private key is’ my_key’, you would use {{secrets/my_secret_scope/my_key}}.
- If you are trying to access the ’directory-id’ or ’tenant id’ as a secret, you should store it as a secret in a secret scope and then refer to it using the above syntax.
- dbutils.secrets.get It is a method to retrieve the value of a secret stored in a Databricks secret scope.
- It can be used in notebooks or jobs where you can access the dbutils object.
- Secrets in Spark configuration are usually referenced using the {{secrets/<scope-name>/<secret-name>}} syntax.
- Ensure you have the correct permissions to access the secret scope and key.
- Also, ensure that the directory id or tenant ID is correctly stored as a secret in the secret scope.

- Sources:
 - [Docs: secrets](https://docs.databricks.com/security/secrets/secrets.html)
 - [Docs: configure](https://docs.databricks.com/clusters/configure.html)