S60 Eliminate SPN secrets - Connect Azure Databric... - Databricks Community

Community Discussions

Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.

Hi Team,

In Azure Databricks, we currently use Service Principal when creating Mount Points to Azure storage ( ADLS Gen1, ADLS Gen 2 and Azure Blob Storage).

As part of S360 action to eliminate SPN secrets, we were asked to move to SPN+certificate / MSI/ User Assigned Managed Identity.

We tried connecting using a custom Active directory (AD) token rather than a client secret through databricks. PEM certificate was used to generate the custom AD token through Java code. We tried to pass the AD token via OAUTH2 method setting below spark configuration.

%scala

// Set up Spark configurations for ADLS Gen2 access with Azure AD token

spark.conf.set("fs.azure.account.auth.type.<Your storage Account>.dfs.core.windows.net", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type.<Your storage Account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id.<Your storage Account>.dfs.core.windows.net", "clientid")

spark.conf.set("fs.azure.account.oauth2.client.secret.<Your storage Account>.dfs.core.windows.net", "")

spark.conf.set("fs.azure.account.oauth2.client.endpoint.<Your storage Account>.dfs.core.windows.net",https://login.microsoftonline.com/<Tenanant-Id>/oauth2/v2.0/token)

spark.conf.set("fs.azure.account.oauth2.access.token.provider", access_token)

To link to an ADLS Gen 2 storage account and access files, we are encountering the following issue.

Please refer the attachment to attach error screenshot

If you come across similar issue would you be able to help?