Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

[m997al]

Hi,

We are attempting to set up Databricks with Unity Catalog (metastore) using a service principal (as opposed to the managed identity).

Instructions we are using are here:  Create a Unity Catalog metastore - Azure Databricks | Microsoft Learn

The challenge is that when we attempt to create the metastore in the Databricks account console, there is a required entry of "Access Connector ID".  In a previous trial, we successfully configured a Databricks metastore using a Databricks Access Connector and a managed identity.

But we deleted that metastore, and we are trying to use the service principal setup instead (a requirement by IT).  It is unclear what the "Access Connector ID" field should be, or if we still need a Databricks Access Connector if we are using a service principal.

The steps in the instructions do not mention anything about an "Access Connector ID" for the creation of a metastore using a service principal, so we are confused as to how to proceed.

Has anyone run into this?  Thank you!

[nkvuong · Accepted solution]

The UI only supports configuring metastore with Managed Identity + Access Connector, to configure it with a service principal, you would need to do programmatic via the API - https://docs.databricks.com/api/azure/workspace/storagecredentials/create

[m997al]

Hi - thanks!

[som_natarajan · Accepted solution]

We only support an API workflow for SP based UC set up. Please note that it will not work if your ADLS is behind a firewall (which is where MI is required)

[m997al]

Hi - I am a bit worried about this not working behind a firewall.  Our ADLS Gen2 will indeed have a private endpoint.

[som_natarajan]

Yes..hence the recommended approach to use MI instead of SPs..which is also why the UI only supports MI based pathway to setting up UC 

[m997al]

So there is no way, even with whitelisting, to get the service principal approach to work with a private ADLS Gen2 endpoint?

[som_natarajan]

No

[karthik_p]

@m997al For UC ADLS Gen 2 behind Firewall config is not needed and support wise limitations as far as i know, if you have security concerns you can Restrict ADLS Gen2 folders to be access by particular users/ groups , which we can do from ADLS Gen 2 config settings.

[m997al]

Thanks to all for the suggestions.  Ultimately, we went with the Managed Identity configuration (after all that investigation).  Answers very much appreciated.  Thank you.