cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
cancel
Showing results for 
Search instead for 
Did you mean: 

Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

m997al
Contributor

Hi,

We are attempting to set up Databricks with Unity Catalog (metastore) using a service principal (as opposed to the managed identity).

Instructions we are using are here:  Create a Unity Catalog metastore - Azure Databricks | Microsoft Learn

The challenge is that when we attempt to create the metastore in the Databricks account console, there is a required entry of "Access Connector ID".  In a previous trial, we successfully configured a Databricks metastore using a Databricks Access Connector and a managed identity.

But we deleted that metastore, and we are trying to use the service principal setup instead (a requirement by IT).  It is unclear what the "Access Connector ID" field should be, or if we still need a Databricks Access Connector if we are using a service principal.

The steps in the instructions do not mention anything about an "Access Connector ID" for the creation of a metastore using a service principal, so we are confused as to how to proceed.

Has anyone run into this?  Thank you!

2 ACCEPTED SOLUTIONS

Accepted Solutions

nkvuong
New Contributor III
New Contributor III

The UI only supports configuring metastore with Managed Identity + Access Connector, to configure it with a service principal, you would need to do programmatic via the API - https://docs.databricks.com/api/azure/workspace/storagecredentials/create

View solution in original post

som_natarajan
New Contributor III
New Contributor III

We only support an API workflow for SP based UC set up. Please note that it will not work if your ADLS is behind a firewall (which is where MI is required)

View solution in original post

9 REPLIES 9

nkvuong
New Contributor III
New Contributor III

The UI only supports configuring metastore with Managed Identity + Access Connector, to configure it with a service principal, you would need to do programmatic via the API - https://docs.databricks.com/api/azure/workspace/storagecredentials/create

Hi - thanks!

som_natarajan
New Contributor III
New Contributor III

We only support an API workflow for SP based UC set up. Please note that it will not work if your ADLS is behind a firewall (which is where MI is required)

m997al
Contributor

Hi - I am a bit worried about this not working behind a firewall.  Our ADLS Gen2 will indeed have a private endpoint.

som_natarajan
New Contributor III
New Contributor III

Yes..hence the recommended approach to use MI instead of SPs..which is also why the UI only supports MI based pathway to setting up UC 

So there is no way, even with whitelisting, to get the service principal approach to work with a private ADLS Gen2 endpoint?

som_natarajan
New Contributor III
New Contributor III

No

karthik_p
Esteemed Contributor

@m997al For UC ADLS Gen 2 behind Firewall config is not needed and support wise limitations as far as i know, if you have security concerns you can Restrict ADLS Gen2 folders to be access by particular users/ groups , which we can do from ADLS Gen 2 config settings.

m997al
Contributor

Thanks to all for the suggestions.  Ultimately, we went with the Managed Identity configuration (after all that investigation).  Answers very much appreciated.  Thank you.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.