https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-1-azure/ba-p/117220 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <H2><SPAN>Introduction</SPAN></H2> <P><SPAN>Unity Catalog (UC) is the foundation for all governance and management of data objects in Databricks Data Intelligence Platform. Since its launch several years ago Unity Catalog has become the best way to experience Azure Databricks.</SPAN></P> <P><SPAN>For most of the time Databricks has existed the primary method of managing data objects like tables is the built-in Hive Metastore (HMS) attached to each Azure Databricks Workspace and by default UC was not enabled on new workspaces. This introduces complexity when onboarding new workspaces and requires retrospective configuration in the Databricks Account console. In order to ease the path to getting started with Unity Catalog Azure Databricks now allows the </SPAN><A href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-metastore#auto-assign" target="_blank" rel="noopener"><SPAN>automatic assignment</SPAN></A><SPAN> of a workspace catalog for each new workspace at provisioning time.</SPAN></P> <P><SPAN>This article will present an overview of exactly what happens when a workspace catalog is automatically provisioned for a new Azure Databricks workspace. There is an assumption the reader has an understanding of what <A href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/" target="_blank" rel="noopener">Unity Catalog is</A> and how <A href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/get-started" target="_blank" rel="noopener">to set it up on Azure</A>, along with how its <A href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/" target="_blank" rel="noopener">securables (data objects) and permissions</A> are managed.</SPAN></P> <P><SPAN><A href="https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-2-aws/ba-p/117237" target="_self">Part two</A> of this article&nbsp;will show the differences between the Azure implementation of UC automatic assignment and the AWS implementation.</SPAN></P> <H2><SPAN>What is a Workspace Catalog?</SPAN></H2> <P><SPAN>When a new Azure Databricks Workspace is automatically enabled for Unity Catalog an initial catalog is created called a workspace catalog. This catalog allows workspace users to easily get started using UC by granting some initial permissions to both the Workspace Administrators and Workspace Users.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-securables-azure.png" style="width: 803px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16456iF8C6F91A43DB33A6/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-securables-azure.png" alt="uc-by-default-securables-azure.png" /></span></P> <P><SPAN>The workspace catalog has the following properties:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The name of the workspace catalog will match the workspace name</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Will be owned by a system owned group called </SPAN><FONT face="courier new,courier"><SPAN>_workspace_admins_${workspace_name}_${workspace_id}</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Will have its storage root located in the managed Azure storage account for the workspace (sometimes called the DBFS root) on a dedicated container called </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog-storage</SPAN></FONT><SPAN> and a folder named after the </SPAN><FONT face="courier new,courier"><SPAN>workspace_id</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>A system owned group called </SPAN><FONT face="courier new,courier"><SPAN>_workspace_users_${workspace_name}_${workspace_id}</SPAN></FONT><SPAN> which has </SPAN><FONT face="courier new,courier"><SPAN>USE_CATALOG</SPAN></FONT><SPAN> rights on the workspace catalog. This user also has enough rights to create objects in the default schema of the workspace catalog.</SPAN></LI> </UL> <P><SPAN>The workspace catalog is made up of three </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges" target="_blank" rel="noopener"><SPAN>Unity Catalog securables</SPAN></A><SPAN>:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Credential</STRONG><SPAN>: this adds the </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog-access-connector</SPAN></FONT><SPAN> in the managed resource group to Unity Catalog. The name of this credential is the workspace name</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>External location</STRONG><SPAN>: this adds the </SPAN><SPAN>unity-catalog-storage</SPAN><SPAN> container on the managed storage account as a valid path in Unity Catalog. The name of this external location is also the Workspace name. The path is </SPAN><FONT face="courier new,courier"><SPAN>abfss://unity-catalog-storage@${managed_storage_account}.dfs.core.windows.net/${workspace_id}</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Catalog</STRONG><SPAN>: this is the Workspace catalog that has a storage root pointing to the </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog-storage</SPAN></FONT><SPAN> container on the external location. The name of this external location is also the workspace name</SPAN></LI> </UL> <P><SPAN>All three of these UC securables are </SPAN><A href="https://learn.microsoft.com/en-us/azure/databricks/catalogs/binding" target="_blank" rel="noopener"><SPAN>bound</SPAN></A><SPAN> to the workspace and not by default available to any other workspace sharing the metastore.</SPAN></P> <H2><SPAN>How Automatic Workspace Assignment Works</SPAN></H2> <P><SPAN>All new Azure Databricks Accounts created after 9th November 2023 are enabled for <A href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-metastore#auto-assign" target="_blank" rel="noopener">Automatic Workspace Assignment</A>, which means that when a new workspace is created it will have a workspace catalog provisioned for it. The process of enabling older Databricks Accounts is ongoing at the time of writing. Organisations who have not had Automatic Workspace Assignment enabled on their Account can request to opt-in by contacting their Databricks Account team.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-flow-azure.png" style="width: 881px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16349iAFFCF8BE047793A9/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-flow-azure.png" alt="uc-by-default-flow-azure.png" /></span></SPAN></P> <P><SPAN>When creating a workspace in a region where Automatic Workspace Assignment is enabled on the Account but there is no metastore a metastore will be created for you. The properties of this metastore are:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The metastore will be called </SPAN><FONT face="courier new,courier"><SPAN>metastore_azure_${cloud_region}</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The metastore will have no metastore owner (it will show </SPAN><FONT face="courier new,courier"><SPAN>System user</SPAN></FONT><SPAN>)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The metastore will be created without a storage root location</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Delta sharing will be disabled</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Automatic Workspace Assignment will be enabled</SPAN></LI> </UL> <P><SPAN>If required a <A href="https://learn.microsoft.com/en-gb/azure/databricks/data-governance/unity-catalog/manage-privileges/admin-privileges#assign-a-metastore-admin" target="_blank" rel="noopener">Metastore Owner</A> can be allocated by an Account Administrator.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-azure-no-metastore.png" style="width: 765px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16350i23ED1041C48EA90A/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-azure-no-metastore.png" alt="uc-by-default-azure-no-metastore.png" /></span></SPAN></P> <P><SPAN>In order to automatically enable all new workspaces in a region for Unity Catalog on an existing metastore in that region the checkbox in </SPAN><FONT face="courier new,courier"><SPAN>Workspace assignment</SPAN></FONT><SPAN> under the metastore settings in the Catalog section of the Account Console has to be checked.</SPAN></P> <P><SPAN>When a metastore is assigned to a workspace a default catalog name is set for all users of that workspace. If the workspace is created via the UI and automatically enabled for UC then the default catalog will be the workspace catalog. If the workspace is created via an API (including using Terraform or an SDK) the </SPAN><A href="https://learn.microsoft.com/en-us/azure/databricks/catalogs/default#the-default-catalog-configuration-when-unity-catalog-is-enabled" target="_blank" rel="noopener"><SPAN>default catalog</SPAN></A><SPAN> will be the </SPAN><FONT face="courier new,courier"><SPAN>hive_metastore</SPAN></FONT><SPAN>.</SPAN></P> <H2><SPAN>Azure infrastructure deployed during automatic enablement</SPAN></H2> <P><SPAN>There are several items in Azure that need to have been created in order to allow objects to be physically stored in the Workspace managed storage account:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>A storage container in the managed storage account called </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog-storage</SPAN></FONT><SPAN>&nbsp;</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>An Access Connector (Managed Identity) with the </SPAN><FONT face="courier new,courier"><SPAN>Storage Blob Data Contributor</SPAN></FONT><SPAN> role assignment on the managed storage account</SPAN></LI> </UL> <P><SPAN>In the Azure Portal you can go to the managed resource group attached to the Azure Databricks Workspace and see the Access Connector called </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog-access-connector</SPAN></FONT><SPAN>.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-azure-infra.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16351i93B32C1DFD25996A/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-azure-infra.png" alt="uc-by-default-azure-infra.png" /></span></SPAN></P> <H2><SPAN>System-owned groups and permissions</SPAN></H2> <P><SPAN>There are system owned groups that are provisioned with the workspace in order to be granted enough permissions to ensure the workspace catalog and other securable objects created with the workspace can be managed. These groups do not appear in most surfaces in the Workspace UI, Account Console or APIs and can not be used to grant Unity Catalog privileges to other securables. The membership of these groups is kept in sync with all the users who have been pushed to the workspace as either the <FONT face="courier new,courier">ADMIN</FONT> or <FONT face="courier new,courier">USER</FONT> role <A href="https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/best-practices#assign-groups" target="_blank" rel="noopener">using Identity Federation</A>.</SPAN></P> <P>&nbsp;</P> <TABLE border="1" width="100%"> <TBODY> <TR> <TD width="33.333333333333336%"><STRONG>Group</STRONG></TD> <TD width="33.333333333333336%"><STRONG>Name</STRONG></TD> <TD width="33.333333333333336%"><STRONG>Unity Catalog Grants</STRONG></TD> </TR> <TR> <TD width="33.333333333333336%"> <P><SPAN>Workspace Admin</SPAN></P> </TD> <TD width="33.333333333333336%"> <P><FONT face="courier new,courier"><SPAN>_workspace_admins_${workspace_name}_${workspace_id}</SPAN></FONT></P> </TD> <TD width="33.333333333333336%"><FONT face="courier new,courier"><SPAN>OWNER</SPAN></FONT><SPAN> on credential, external location and workspace catalog in addition to the metastore level rights listed in the next section</SPAN></TD> </TR> <TR> <TD width="33.333333333333336%"> <P><SPAN>Workspace Users</SPAN></P> </TD> <TD width="33.333333333333336%"> <P><FONT face="courier new,courier"><SPAN>_workspace_users_${workspace_name}_${workspace_id}</SPAN></FONT></P> </TD> <TD width="33.333333333333336%"><SPAN>Usage (</SPAN><FONT face="courier new,courier"><SPAN>USE_CATALOG</SPAN></FONT><SPAN>) rights on workspace catalog and&nbsp; usage rights on </SPAN><FONT face="courier new,courier"><SPAN>default</SPAN></FONT><SPAN> schema (see below)</SPAN></TD> </TR> </TBODY> </TABLE> <P><SPAN>The following shows the grants on the </SPAN><FONT face="courier new,courier"><SPAN>default</SPAN></FONT><SPAN> schema.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-users-default-schema.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16352i9E536D5385B4F617/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-users-default-schema.png" alt="uc-by-default-users-default-schema.png" /></span></SPAN></P> <H2><SPAN>Metastore-level grants for Auto-Enabled Workspace Administrators</SPAN></H2> <P><SPAN>In order to create all these Unity Catalog securables the Workspace Admins system owned group needs some <A href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-privileges/privileges" target="_blank" rel="noopener">grants</A> on the Unity Catalog Metastore (the screenshot below also shows this workspace was provisioned via Terraform so it has <FONT face="courier new,courier">hive_metastore</FONT> as a default catalog)</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-azure-metastore-grants.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16353i3E089113435147CD/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-azure-metastore-grants.png" alt="uc-by-default-azure-metastore-grants.png" /></span></SPAN></P> <P><SPAN>These grants do not include ownership of the metastore, meaning the workspace admin can not delete metastore level UC securables that were created or owned by other identities, including the workspace catalog and securables on other workspaces created with UC by default.</SPAN></P> <P><SPAN>These grants also allow the Workspace Administrators to create other catalogs and related underlying securables like credentials and external locations. By default any securable created will be owned by the individual identity that created that securable and ownership allows transfer of ownership to a group.</SPAN></P> <H2><SPAN>Best practices for using the workspace catalog</SPAN></H2> <P><SPAN>While the provisioning of a workspace catalog greatly simplifies the initial setup of Unity Catalog for new workspaces it does also tie the catalog directly to the lifecycle of the workspace: if the workspace is decommissioned the managed storage account that contains any unity catalog objects in the workspace catalog will also be lost.</SPAN></P> <P><SPAN>The recommendation is to adhere to </SPAN><A href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices" target="_blank" rel="noopener"><SPAN>existing best practices</SPAN></A><SPAN> for creating catalogs, aligning them with SDLC (Software Development Lifecycle), business units, and/or projects. This allows more flexibility to segregate storage away from the workspace and to bind these catalogs to multiple workspaces where required. It also means that the addition or removal of a workspace does not impact the lifecycle of any data stored in Unity Catalog.</SPAN></P> <P><SPAN>The metastore permissions granted to the system owned Workspace Admins group give enough permissions to create the required securables (credentials, external locations, catalogs etc) to achieve the required catalog design for your organisation.</SPAN></P> <H2><SPAN>Conclusion</SPAN></H2> <P><SPAN>To recap, when Azure Databricks workspaces are auto-enabled for Unity Catalog, a default workspace catalog is created along with necessary cloud resources and permissions – all without manual effort. This makes it much easier for the users of new workspaces to start using Unity Catalog immediately, however it is still important to follow best practices in catalog design.</SPAN></P> <P><SPAN>Unity Catalog will continue to be the foundation that the Databricks Data Intelligence Platform is built on. The ability to automatically enable Unity Catalog for all new workspaces greatly reduces the friction to start getting all the benefits of the platform.</SPAN></P> <P><SPAN>For details on what happens when using UC automatic assignment on AWS please see&nbsp;<A href="https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-2-aws/ba-p/117237" target="_self">part two</A> of this article.</SPAN></P> Tue, 06 May 2025 11:07:22 GMT stevejohansen 2025-05-06T11:07:22Z https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-1-azure/ba-p/117220 <P><SPAN>Unity Catalog (UC) is the foundation for all governance and management of data objects in Databricks Data Intelligence Platform. Since its launch several years ago Unity Catalog has become the best way to experience Azure Databricks. This article will present an overview of exactly what happens when a workspace catalog is automatically provisioned for a new Azure Databricks workspace. </SPAN></P> Tue, 06 May 2025 11:07:22 GMT https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-1-azure/ba-p/117220 stevejohansen 2025-05-06T11:07:22Z