https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-2-aws/ba-p/117237 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <H2><SPAN>Introduction</SPAN></H2> <P><SPAN>In <A href="https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-1-azure/ba-p/117220" target="_self">part one</A>&nbsp;of this article we introduced the ability to automatically enable Unity Catalog (UC) on new Azure Databricks workspaces and details on exactly what happens during this process. This article will take a deep dive into the same Unity Catalog </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-metastore#enable-a-metastore-to-be-automatically-assigned-to-new-workspaces" target="_blank" rel="noopener"><SPAN>Workspace Automatic Enablement</SPAN></A><SPAN> process when using Databricks workspaces on AWS and highlight the differences.</SPAN></P> <P><SPAN>Much like the previous there is an assumption the reader has an understanding of what </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog" target="_blank" rel="noopener"><SPAN>Unity Catalog is</SPAN></A><SPAN> and how </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started" target="_blank" rel="noopener"><SPAN>to set it up on AWS</SPAN></A><SPAN>, along with how its </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/" target="_blank" rel="noopener"><SPAN>securables (data objects) and permissions</SPAN></A><SPAN> are managed.</SPAN></P> <H2><SPAN>What is a Workspace Catalog?</SPAN></H2> <P data-unlink="true"><SPAN>When a workspace is created with Unity Catalog automatically enabled a workspace catalog is created assigned to that workspace. If there is no existing Unity Catalog metastore in the cloud region that the workspace is being created in then a metastore will also be created with the defaults described below. To the Workspace Administrator and Workspace User groups this should look almost identical to how it is done on Azure, with the underlying cloud infrastructure details mostly abstracted away. </SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-aws-overview.png" style="width: 800px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16407i126F28AC3047A20C/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-aws-overview.png" alt="uc-by-default-aws-overview.png" /></span></P> <P><SPAN>The workspace catalog on AWS has the following properties:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The name of the workspace catalog will match the workspace name</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Will be owned by a system owned group called </SPAN><FONT face="courier new,courier"><SPAN>_workspace_admins_${workspace_name}_${workspace_id}</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Will have its storage root located in the S3 workspace root bucket in a dedicated folder called </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>A system owned group called </SPAN><FONT face="courier new,courier"><SPAN>_workspace_users_${workspace_name}_${workspace_id}</SPAN></FONT><SPAN> which has </SPAN><FONT face="courier new,courier"><SPAN>USE_CATALOG</SPAN></FONT><SPAN> rights on the workspace catalog. This user also has enough rights to create objects in the default schema of the workspace catalog.</SPAN></LI> </UL> <P><SPAN>The workspace catalog is made up of three </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges" target="_blank" rel="noopener"><SPAN>Unity Catalog securables</SPAN></A><SPAN>:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Credential</STRONG><SPAN>: the biggest difference in AWS is there needs to be an extra IAM role provision for the UC storage credential that will be used for the workspace catalog (details below). The name of the UC credential will match the workspace name.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>External location</STRONG><SPAN>: this adds the </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog</SPAN></FONT><SPAN>&nbsp;folder in the S3 workspace root bucket as a valid path in Unity Catalog. The name of this external location is also the Workspace name. The path is </SPAN><FONT face="courier new,courier"><SPAN>s3://${workspace_root}/unity-catalog/${workspace_id}</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Catalog</STRONG><SPAN>: this is the Workspace catalog that has a storage root pointing to the </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog</SPAN></FONT><SPAN> folder on the external location. The name of this external location is also the workspace name</SPAN></LI> </UL> <P><SPAN>All three of these UC securables are </SPAN><A href="https://docs.databricks.com/aws/en/catalogs/binding" target="_blank" rel="noopener"><SPAN>bound</SPAN></A><SPAN> to the workspace and not by default available to any other workspace sharing the metastore.</SPAN></P> <H2><SPAN>How Automatic Workspace Assignment Works</SPAN></H2> <P><SPAN>In order to automatically enable a workspace for Unity Catalog there are several requirements that must be met. Automatic enablement of Unity Catalog for AWS requires two main prerequisites:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>If a metastore exists it must be enabled for automatic assignment</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>A UC storage credential IAM role allowing access to the S3 workspace root bucket must be created</SPAN></LI> </UL> <P><SPAN>The first requirement is identical enabling automatic assignment on Azure Databricks. As shown in part one of this article only Databricks Accounts created after 9 November 2023 are automatically set up for </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started#enablement" target="_blank" rel="noopener"><SPAN>Automatic Workspace Assignment</SPAN></A><SPAN>.&nbsp;</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-flow-aws.png" style="width: 825px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16408i1969309A215C722D/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-flow-aws.png" alt="uc-by-default-flow-aws.png" /></span></SPAN></P> <P><SPAN>When creating a workspace in a region where Automatic Workspace Assignment is enabled on the Account but there is no metastore then a metastore will be created for you in the same way as it in Azure. The properties of this metastore are:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>The metastore will be called </SPAN><FONT face="courier new,courier"><SPAN>metastore_aws_${cloud_region}</SPAN></FONT></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The metastore will have no metastore owner (it will show </SPAN><FONT face="courier new,courier"><SPAN>System user</SPAN></FONT><SPAN>)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>The metastore will be created without a storage root location</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Delta sharing will be disabled</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Automatic Workspace Assignment will be enabled</SPAN></LI> </UL> <P><SPAN>If required a <A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/admin-privileges#assign-metastore-admin" target="_blank" rel="noopener">Metastore Owner</A> can be allocated by an Account Administrator.</SPAN></P> <P><SPAN>In order to automatically enable all new workspaces in a region for Unity Catalog on an existing metastore in that region the checkbox in <FONT face="courier new,courier">Workspace assignment</FONT> under the metastore settings in the Catalog section of the Account Console has to be checked.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-aws-metastore-tick.png" style="width: 613px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16412i27504B007FDA0296/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-aws-metastore-tick.png" alt="uc-by-default-aws-metastore-tick.png" /></span></SPAN></P> <P><SPAN>The second prerequisite is specific to Databricks on AWS. The </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started#enablement" target="_blank" rel="noopener"><SPAN>documentation</SPAN></A><SPAN> states:</SPAN></P> <P class="lia-indent-padding-left-30px"><I><SPAN>Your workspace gets the workspace catalog only if the workspace creator provided an appropriate IAM role and storage location during workspace provisioning.</SPAN></I></P> <P data-unlink="true"><SPAN>We will go into detail on how this IAM role works in the section on </SPAN><SPAN>AWS Infrastructure</SPAN>&nbsp;<SPAN> below.</SPAN></P> <P><SPAN>Like Azure, when a metastore is assigned to a workspace a default catalog name is set for all users of that workspace. If the workspace is created via the UI and automatically enabled for UC then the default catalog will be the workspace catalog. If the workspace is created via an API (currently supported via the <A href="https://docs.databricks.com/aws/en/admin/workspace/quick-start" target="_blank" rel="noopener">Quickstart CloudFormation template</A>) the default catalog will be the <FONT face="courier new,courier">hive_metastore</FONT>.</SPAN></P> <H2><SPAN>AWS infrastructure deployed during automatic enablement</SPAN></H2> <P><SPAN>When provisioning a Databricks Workspace on AWS there have always been the following required items of supporting AWS infrastructure:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>S3 workspace root bucket</STRONG><SPAN>: This is an S3 bucket used for workspace storage, including DBFS and the workspace filesystem.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>IAM cross account role</STRONG><SPAN>: This allows Databricks to launch EC2 instances for cluster nodes in the customer compute plane AWS account. The S3 workspace root bucket includes a bucket policy to allow this cross account access.</SPAN></LI> </UL> <P><SPAN>When setting up a workspace for automatic UC enablement we also need to provide another IAM role to be used as the UC storage credential for the workspace catalog storage. This role allows UC to access the </SPAN><FONT face="courier new,courier"><SPAN>unity-catalog</SPAN></FONT><SPAN> folder in the S3 workspace root and to put the workspace catalog storage root in that folder.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-aws-infrapng.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16413iC34B978385954F19/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-aws-infrapng.png" alt="uc-by-default-aws-infrapng.png" /></span></SPAN></P> <P><SPAN>This storage credential IAM role is a standard <A href="https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials" target="_blank" rel="noopener">Unity Catalog self-assuming IAM role</A>, as outlined in the documentation for creating storage credentials in UC. The attached IAM policy on that role grants access to the unity-catalog folder on the UC workspace root bucket. The trust role with the self-assuming policy setup is the same as any other UC storage credential IAM role. This <A href="https://medium.com/databricks-platform-sme/a-storage-configuration-mystery-f3b47acd8815" target="_blank" rel="noopener">Medium post covers</A> the details.</SPAN></P> <LI-CODE lang="javascript">{ "Version": "2012-10-17", "Id": "databricks-uc-dbfs-bucket-access", "Statement": [ { "Action": [ "s3:DeleteObject", "s3:GetBucketLocation", "s3:GetObject", "s3:ListBucket", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::anzps-uc-by-default", "arn:aws:s3:::anzps-uc-by-default/unity-catalog/*" ], "Effect": "Allow" }, { "Action": [ "sts:AssumeRole" ], "Effect": "Allow", "Resource": [ "arn:aws:iam::332745928618:role/anzps-uc-by-default-workspace-cred" ] } ] }</LI-CODE> <P><SPAN>When provisioning the workspace there are now steps in the S3 workspace storage provisioning to provide your Unity Catalog UC storage credential IAM role ARN. You can also see the generated bucket policy explicitly denies the workspace cross account role used to access the workspace root bucket for use in compute and DBFS denies access to the <FONT face="courier new,courier">unity-catalog</FONT> folder, which ensures all access to that folder must be via Unity Catalog.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-aws-workspace-provisioning.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16414i991D4FDBE9B150C3/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-aws-workspace-provisioning.png" alt="uc-by-default-aws-workspace-provisioning.png" /></span></SPAN></P> <H2><SPAN>System-owned groups and permissions</SPAN></H2> <P><SPAN>The system owned groups that are provisioned with the workspace work the same way they do in Azure, and just like in Azure these groups do not appear in most surfaces in the Workspace UI, Account Console or APIs and can not be used to grant Unity Catalog privileges to other securables. The membership of these groups is kept in sync with all the users who have been pushed to the workspace as either the <FONT face="courier new,courier">ADMIN</FONT> or <FONT face="courier new,courier">USER</FONT> role <A href="https://docs.databricks.com/aws/en/admin/users-groups/best-practices#assign-groups-workspace-permissions" target="_blank" rel="noopener">using Identity Federation</A>. These groups have enough permissions for the Workspace Administrators to manage the workspace catalog and for Workspace Users to start using UC in the workspace catalog <FONT face="courier new,courier">default</FONT> schema.</SPAN></P> <P>&nbsp;</P> <TABLE border="1" width="100%"> <TBODY> <TR> <TD width="33.333333333333336%"><STRONG>Group</STRONG></TD> <TD width="33.333333333333336%"><STRONG>Name</STRONG></TD> <TD width="33.333333333333336%"><STRONG>Unity Catalog Grants</STRONG></TD> </TR> <TR> <TD width="33.333333333333336%"> <P><SPAN>Workspace Admin</SPAN></P> </TD> <TD width="33.333333333333336%"> <P><FONT face="courier new,courier"><SPAN>_workspace_admins_${workspace_name}_${workspace_id}</SPAN></FONT></P> </TD> <TD width="33.333333333333336%"> <P><FONT face="courier new,courier"><SPAN>OWNER</SPAN></FONT><SPAN> on credential, external location and workspace catalog in addition to the metastore level rights listed in the next section</SPAN></P> </TD> </TR> <TR> <TD width="33.333333333333336%"> <P><SPAN>Workspace Users</SPAN></P> </TD> <TD width="33.333333333333336%"> <P><FONT face="courier new,courier"><SPAN>_workspace_users_${workspace_name}_${workspace_id}</SPAN></FONT></P> </TD> <TD width="33.333333333333336%"> <P><SPAN>Usage (</SPAN><FONT face="courier new,courier"><SPAN>USE_CATALOG</SPAN></FONT><SPAN>) rights on workspace catalog and&nbsp; usage rights on </SPAN><FONT face="courier new,courier"><SPAN>default</SPAN></FONT><SPAN> schema (see below)</SPAN></P> </TD> </TR> </TBODY> </TABLE> <P><SPAN>The following shows the grants on the </SPAN><FONT face="courier new,courier"><SPAN>default</SPAN></FONT><SPAN> schema for the Workspace Users.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-aws-user-grants-default-schema.png" style="width: 702px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16454i9B5CC75648B34B4D/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-aws-user-grants-default-schema.png" alt="uc-by-default-aws-user-grants-default-schema.png" /></span></SPAN></P> <H2><SPAN>Metastore-level grants for Auto-Enabled Workspace Administrators</SPAN></H2> <P><SPAN>The implementation of Unity Catalog Metastore <A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/privileges" target="_blank" rel="noopener">grants</A> to allow the Workspace Admins group to create other top level objects in Unity Catalog is the same as was shown for the Azure deployment. The screenshot also shows this workspace was deployed using the Account Console meaning the default catalog is the workspace catalog.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="uc-by-default-aws-metastore-admin-grants.png" style="width: 982px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/16455iA8303A30B886EB5B/image-size/large?v=v2&amp;px=999" role="button" title="uc-by-default-aws-metastore-admin-grants.png" alt="uc-by-default-aws-metastore-admin-grants.png" /></span></SPAN></P> <P><SPAN>These grants do not include ownership of the metastore, meaning the workspace admin can not delete metastore level UC securables that were created or owned by other identities, including the workspace catalog and securables on other workspaces created with UC by default.</SPAN></P> <P><SPAN>These grants also allow the Workspace Administrators to create other catalogs and related underlying securables like credentials and external locations. By default any securable created will be owned by the individual identity that created that securable and ownership allows transfer of ownership to a group.</SPAN></P> <H2><SPAN>Best practices for using the workspace catalog</SPAN></H2> <P><SPAN>The guidance of using the workspace catalog follows the same given for Azure: the workspace catalog is great for initial enablement but it does also tie your data to the lifecycle of the workspace. The recommendation remains to adhere to </SPAN><A href="https://docs.databricks.com/aws/en/data-governance/unity-catalog/best-practices" target="_blank" rel="noopener"><SPAN>existing best practices</SPAN></A><SPAN> for creating catalogs, aligning them with SDLC (Software Development Lifecycle), business units, and/or projects. This allows more flexibility to segregate storage away from the workspace and to bind these catalogs to multiple workspaces where required. It also means that the addition or removal of a workspace does not impact the lifecycle of any data stored in Unity Catalog.</SPAN></P> <P><SPAN>The metastore permissions granted to the system owned Workspace Admins group give enough permissions to create the required securables (credentials, external locations, catalogs etc) to achieve the required catalog design for your organisation.</SPAN></P> <H2><SPAN>Conclusion</SPAN></H2> <P><SPAN>The experience of using Unity Catalog with an automatically provisioned workspace catalog on AWS should not look materially different to the Workspace Admins and Users than it would for an Azure Databricks workspace. The underlying cloud infrastructure is mostly abstracted away but at times it can be useful to understand these differences, especially for those planning the initial deployment of a workspace on either cloud. The end result in both is a way to start using Unity Catalog from the first time the workspace is deployed, allowing the workspace users to get all the benefits of the Databricks Data Intelligence Platform.</SPAN></P> <P><SPAN>For further information and best practices on how to get the most out of UC, please follow the <A href="https://medium.com/databricks-unity-catalog-sme" target="_blank" rel="noopener">Databricks Unity Catalog SME page on Medium</A>. </SPAN></P> Tue, 06 May 2025 11:07:46 GMT stevejohansen 2025-05-06T11:07:46Z https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-2-aws/ba-p/117237 <P><SPAN>In part one we discussed how Unity Catalog (UC) is the foundation for all governance and management of data objects in Databricks Data Intelligence Platform. Since its launch several years ago Unity Catalog has become the best way to experience Databricks. This article covers deployments of Databricks on AWS and will present an overview of exactly what happens when a workspace catalog is automatically provisioned for a new Databricks workspace on AWS and the key differences with Azure Databricks.</SPAN></P> Tue, 06 May 2025 11:07:46 GMT https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-2-aws/ba-p/117237 stevejohansen 2025-05-06T11:07:46Z