cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

m997al
Contributor III

Hi,

We are attempting to set up Databricks with Unity Catalog (metastore) using a service principal (as opposed to the managed identity).

Instructions we are using are here:  Create a Unity Catalog metastore - Azure Databricks | Microsoft Learn

The challenge is that when we attempt to create the metastore in the Databricks account console, there is a required entry of "Access Connector ID".  In a previous trial, we successfully configured a Databricks metastore using a Databricks Access Connector and a managed identity.

But we deleted that metastore, and we are trying to use the service principal setup instead (a requirement by IT).  It is unclear what the "Access Connector ID" field should be, or if we still need a Databricks Access Connector if we are using a service principal.

The steps in the instructions do not mention anything about an "Access Connector ID" for the creation of a metastore using a service principal, so we are confused as to how to proceed.

Has anyone run into this?  Thank you!

2 ACCEPTED SOLUTIONS

Accepted Solutions

nkvuong
New Contributor III

The UI only supports configuring metastore with Managed Identity + Access Connector, to configure it with a service principal, you would need to do programmatic via the API - https://docs.databricks.com/api/azure/workspace/storagecredentials/create

View solution in original post

som_natarajan
New Contributor III

We only support an API workflow for SP based UC set up. Please note that it will not work if your ADLS is behind a firewall (which is where MI is required)

View solution in original post

9 REPLIES 9

nkvuong
New Contributor III

The UI only supports configuring metastore with Managed Identity + Access Connector, to configure it with a service principal, you would need to do programmatic via the API - https://docs.databricks.com/api/azure/workspace/storagecredentials/create

m997al
Contributor III

Hi - thanks!

som_natarajan
New Contributor III

We only support an API workflow for SP based UC set up. Please note that it will not work if your ADLS is behind a firewall (which is where MI is required)

m997al
Contributor III

Hi - I am a bit worried about this not working behind a firewall.  Our ADLS Gen2 will indeed have a private endpoint.

som_natarajan
New Contributor III

Yes..hence the recommended approach to use MI instead of SPs..which is also why the UI only supports MI based pathway to setting up UC 

So there is no way, even with whitelisting, to get the service principal approach to work with a private ADLS Gen2 endpoint?

som_natarajan
New Contributor III

No

karthik_p
Esteemed Contributor

@m997al For UC ADLS Gen 2 behind Firewall config is not needed and support wise limitations as far as i know, if you have security concerns you can Restrict ADLS Gen2 folders to be access by particular users/ groups , which we can do from ADLS Gen 2 config settings.

m997al
Contributor III

Thanks to all for the suggestions.  Ultimately, we went with the Managed Identity configuration (after all that investigation).  Answers very much appreciated.  Thank you.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group