topic Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity) in Administration & Architecture

Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

m997al — Mon, 09 Oct 2023 20:10:43 GMT

Hi,

We are attempting to set up Databricks with Unity Catalog (metastore) using a service principal (as opposed to the managed identity).

Instructions we are using are here: Create a Unity Catalog metastore - Azure Databricks | Microsoft Learn

The challenge is that when we attempt to create the metastore in the Databricks account console, there is a required entry of "Access Connector ID". In a previous trial, we successfully configured a Databricks metastore using a Databricks Access Connector and a managed identity.

But we deleted that metastore, and we are trying to use the service principal setup instead (a requirement by IT). It is unclear what the "Access Connector ID" field should be, or if we still need a Databricks Access Connector if we are using a service principal.

The steps in the instructions do not mention anything about an "Access Connector ID" for the creation of a metastore using a service principal, so we are confused as to how to proceed.

Has anyone run into this? Thank you!

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

nkvuong — Tue, 10 Oct 2023 09:01:11 GMT

The UI only supports configuring metastore with Managed Identity + Access Connector, to configure it with a service principal, you would need to do programmatic via the API - https://docs.databricks.com/api/azure/workspace/storagecredentials/create

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

som_natarajan — Tue, 10 Oct 2023 09:19:45 GMT

We only support an API workflow for SP based UC set up. Please note that it will not work if your ADLS is behind a firewall (which is where MI is required)

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

m997al — Tue, 10 Oct 2023 14:47:51 GMT

Hi - thanks!

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

m997al — Tue, 10 Oct 2023 18:57:29 GMT

Hi - I am a bit worried about this not working behind a firewall. Our ADLS Gen2 will indeed have a private endpoint.

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

som_natarajan — Tue, 10 Oct 2023 19:04:53 GMT

Yes..hence the recommended approach to use MI instead of SPs..which is also why the UI only supports MI based pathway to setting up UC

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

m997al — Tue, 10 Oct 2023 19:59:42 GMT

So there is no way, even with whitelisting, to get the service principal approach to work with a private ADLS Gen2 endpoint?

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

som_natarajan — Tue, 10 Oct 2023 20:27:02 GMT

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

karthik_p — Tue, 17 Oct 2023 15:20:26 GMT

@m997al For UC ADLS Gen 2 behind Firewall config is not needed and support wise limitations as far as i know, if you have security concerns you can Restrict ADLS Gen2 folders to be access by particular users/ groups , which we can do from ADLS Gen 2 config settings.

Re: Setting up Databricks with Unity Catalog using a service principal (instead of managed identity)

m997al — Tue, 17 Oct 2023 15:28:11 GMT

Thanks to all for the suggestions. Ultimately, we went with the Managed Identity configuration (after all that investigation). Answers very much appreciated. Thank you.