โ10-09-2023 01:10 PM
Hi,
We are attempting to set up Databricks with Unity Catalog (metastore) using a service principal (as opposed to the managed identity).
Instructions we are using are here: Create a Unity Catalog metastore - Azure Databricks | Microsoft Learn
The challenge is that when we attempt to create the metastore in the Databricks account console, there is a required entry of "Access Connector ID". In a previous trial, we successfully configured a Databricks metastore using a Databricks Access Connector and a managed identity.
But we deleted that metastore, and we are trying to use the service principal setup instead (a requirement by IT). It is unclear what the "Access Connector ID" field should be, or if we still need a Databricks Access Connector if we are using a service principal.
The steps in the instructions do not mention anything about an "Access Connector ID" for the creation of a metastore using a service principal, so we are confused as to how to proceed.
Has anyone run into this? Thank you!
โ10-10-2023 02:01 AM
The UI only supports configuring metastore with Managed Identity + Access Connector, to configure it with a service principal, you would need to do programmatic via the API - https://docs.databricks.com/api/azure/workspace/storagecredentials/create
โ10-10-2023 02:19 AM
We only support an API workflow for SP based UC set up. Please note that it will not work if your ADLS is behind a firewall (which is where MI is required)
โ10-10-2023 02:01 AM
The UI only supports configuring metastore with Managed Identity + Access Connector, to configure it with a service principal, you would need to do programmatic via the API - https://docs.databricks.com/api/azure/workspace/storagecredentials/create
โ10-10-2023 07:47 AM
Hi - thanks!
โ10-10-2023 02:19 AM
We only support an API workflow for SP based UC set up. Please note that it will not work if your ADLS is behind a firewall (which is where MI is required)
โ10-10-2023 11:57 AM
Hi - I am a bit worried about this not working behind a firewall. Our ADLS Gen2 will indeed have a private endpoint.
โ10-10-2023 12:04 PM
Yes..hence the recommended approach to use MI instead of SPs..which is also why the UI only supports MI based pathway to setting up UC
โ10-10-2023 12:59 PM
So there is no way, even with whitelisting, to get the service principal approach to work with a private ADLS Gen2 endpoint?
โ10-10-2023 01:27 PM
No
โ10-17-2023 08:20 AM
@m997al For UC ADLS Gen 2 behind Firewall config is not needed and support wise limitations as far as i know, if you have security concerns you can Restrict ADLS Gen2 folders to be access by particular users/ groups , which we can do from ADLS Gen 2 config settings.
โ10-17-2023 08:28 AM
Thanks to all for the suggestions. Ultimately, we went with the Managed Identity configuration (after all that investigation). Answers very much appreciated. Thank you.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group