cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

S60 Eliminate SPN secrets - Connect Azure Databricks to ADLS Gen2 , Gen1 via custom AD token

rameshkumar610
New Contributor
Hi Team,
In Azure Databricks, we currently use Service Principal when creating Mount Points to Azure storage ( ADLS Gen1, ADLS Gen 2 and Azure Blob Storage).

As part of S360 action to eliminate SPN secrets, we were asked to move to SPN+certificate / MSI/ User Assigned Managed Identity.

We tried connecting using a custom Active directory (AD) token rather than a client secret through databricks. PEM certificate was used to generate the custom AD token through Java code. We tried to pass the AD token via OAUTH2 method setting below spark configuration.

 

%scala

// Set up Spark configurations for ADLS Gen2 access with Azure AD token

spark.conf.set("fs.azure.account.auth.type.<Your storage Account>.dfs.core.windows.net", "OAuth")

spark.conf.set("fs.azure.account.oauth.provider.type.<Your storage Account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")

spark.conf.set("fs.azure.account.oauth2.client.id.<Your storage Account>.dfs.core.windows.net", "clientid")

spark.conf.set("fs.azure.account.oauth2.client.secret.<Your storage Account>.dfs.core.windows.net", "")

spark.conf.set("fs.azure.account.oauth2.client.endpoint.<Your storage Account>.dfs.core.windows.net",https://login.microsoftonline.com/<Tenanant-Id>/oauth2/v2.0/token)

spark.conf.set("fs.azure.account.oauth2.access.token.provider", access_token)

To link to an ADLS Gen 2 storage account and access files, we are encountering the following issue.

Please refer the attachment to attach error screenshot
 
If you come across similar issue would you be able to help?
5 REPLIES 5

iakshaykr
New Contributor

I don’t think SPN certificate is supported by spark configuration.

Slash
New Contributor II

This is an old way of mounting storage account. Is there any reason why you guys are not using unity catalog? Because if you have unity catalog assigned to your workspace, you can just create databricks access connector (databricks managed identity) and configure access to external location using this managed identity.

ramesitexp
Visitor

Hello, @Slash. We would want to authenticate using a Custom Token or a Keyvault Certificate rather than a secret value or other keys. If we use managed identity, we must set the complete resource level to restrict the access. We would like to authenticate using cluster or notebook level authentication...

Slash
New Contributor II

Unfortunately, the way you're currently trying to do this is not supported by databricks. The only valid authentication options are listed in below article: 

 

https://learn.microsoft.com/en-us/azure/databricks/connect/storage/azure-storage#--connect-to-azure-...

 

https://stackoverflow.com/questions/77437928/access-adls-gen2-using-pem-certificate-from-apache-spar...

iakshaykr
New Contributor

@ramesitexp Yes @Slash is correct for now only valid option is below : 

OAuth 2.0 with a Microsoft Entra ID service principal

Shared access signatures (SAS)

Account keys

 

For now we are using OAuth 2.0 with a Microsoft Entra ID service principal  with client secret not certificate . 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!