https://community.databricks.com/t5/technical-blog/you-should-know-these-4-things-about-setting-up-unity-catalog/ba-p/76407 <P><SPAN>Over the past year we’ve seen huge adoption of Unity Catalog, and we've noticed certain considerations that are key to all successful deployments.&nbsp;</SPAN><SPAN>This blog focuses on how to set up a key artifact in Unity - the catalog - and the things you should be thinking about when creating them!</SPAN></P> <H2><SPAN>What is a catalog?</SPAN></H2> <P><SPAN>Catalogs are the first layer in the data hierarchy, used to organize your data assets.&nbsp; A catalog contains schemas (databases), and a schema contains tables, views, volumes, models, and functions.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2024-07-01 at 11.57.44 AM.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9122iCB1792C9AC3176F4/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2024-07-01 at 11.57.44 AM.png" alt="Screenshot 2024-07-01 at 11.57.44 AM.png" /></span><BR /></SPAN><FONT size="2"><I><SPAN>You find catalogs in the Catalog Explorer UI!</SPAN></I></FONT></P> <P><FONT size="1 2 3 4 5 6 7"><I><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2024-07-01 at 11.59.06 AM.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9123iCE8BAFF6E4DB9C05/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2024-07-01 at 11.59.06 AM.png" alt="Screenshot 2024-07-01 at 11.59.06 AM.png" /></span></SPAN></I></FONT></P> <P><FONT size="2"><I><SPAN>Logically, catalogs contain schemas, which in turn contain your data assets such as tables and models.</SPAN></I></FONT></P> <P><SPAN>Catalogs should map to high level data domains in your organization and also be used to separate data that is used in development, test, and production environments.&nbsp;</SPAN></P> <P><SPAN>Take for example a company that sells solar powered products - catalogs could be used to separate telemetry data from each of the company’s products and be specific to dev/test/prod, e.</SPAN><SPAN>g. solar_lights_prod, solar_lights_dev, solar_panels_prod, etc.</SPAN></P> <P><SPAN>Now that we know what a catalog is, here are 4 things you should consider when setting them up!</SPAN></P> <H2><SPAN>4 things to consider when creating a catalog</SPAN></H2> <P><SPAN>You can follow along with the below steps directly within the ‘Create Catalog’ UI, found within the Catalog Explorer.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2024-07-01 at 11.50.48 AM.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9117iC7ABDCB85344EABB/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2024-07-01 at 11.50.48 AM.png" alt="Screenshot 2024-07-01 at 11.50.48 AM.png" /></span></P> <H3><SPAN>Picking the right catalog type</SPAN></H3> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2024-07-01 at 11.55.19 AM.png" style="width: 471px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9121iB2D89547A70062C8/image-dimensions/471x399?v=v2" width="471" height="399" role="button" title="Screenshot 2024-07-01 at 11.55.19 AM.png" alt="Screenshot 2024-07-01 at 11.55.19 AM.png" /></span></P> <P><SPAN>The first consideration when creating a catalog is catalog ‘type’. There are 3 possible types of catalogs that can be created in UC: Standard, Foreign, and Shared.</SPAN></P> <UL> <LI><STRONG>Standard</STRONG> <STRONG>catalogs</STRONG><SPAN> are your ‘bread and butter’ catalog - used to hold tables, views, models and other data and AI assets. When creating a standard catalog, you must specify a storage location for the catalog. A storage location specifies a location in cloud object storage (S3/ADLS/GCS) for storing data for managed tables that you add to the catalog.</SPAN></LI> </UL> <P><SPAN>You can now specify “Default storage” (currently in preview) as the managed storage location for the Standard catalog. Using default storage, you get a new catalog instantaneously with out-of-the-box security, unified data access, and cost observability without the overhead of managing cloud infrastructure. Click </SPAN><A href="https://forms.gle/j144aG72uRGFvR4y9" target="_blank" rel="noopener"><SPAN>here</SPAN></A><SPAN> to sign up for the private preview.</SPAN></P> <UL> <LI><STRONG style="font-family: inherit;">Foreign</STRONG> <STRONG style="font-family: inherit;">catalogs</STRONG><SPAN> mirror a database in an external data system such as Postgres, Redshift, Snowflake, etc. They enable users to perform read-only queries on that data system right from Databricks workspace! You can apply all of Unity’s fine grained security on top of this data, as well as track lineage of how this data is used.</SPAN></LI> <LI><STRONG>Shared</STRONG> <STRONG>catalogs</STRONG><SPAN> allow you to access data shared with you by other organizations. This includes any external data shared with your organization through the Delta Sharing protocol, including </SPAN><A href="https://docs.databricks.com/en/marketplace/index.html" target="_blank" rel="noopener"><SPAN>Marketplace</SPAN></A><SPAN>. You can also use shared catalogs to access data between regions in your own account.</SPAN></LI> </UL> <P><SPAN>Take the example once again of the company that sells solar-powered equipment:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Admins could set up standard catalogs to hold telemetry data ingested into Databricks from sensors attached to their products.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Foreign catalogs could be configured to make sales data of these products that live in an external system such as Postgres available to users in Databricks.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Shared catalogs could be leveraged to make data from the Databricks Marketplace related to general weather measurements provided by meteorological agencies or weather services available to users.</SPAN></LI> </UL> <P><SPAN>All these datasets are now made accessible to users in one place and can simply be joined and queried through Unity Catalog! Furthermore, admins can use a single governance model to control who has access to what data.</SPAN></P> <H3><SPAN>Configuring workspace-catalog bindings</SPAN></H3> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2024-07-01 at 11.52.19 AM.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9120i3A77B93F60ADB23A/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2024-07-01 at 11.52.19 AM.png" alt="Screenshot 2024-07-01 at 11.52.19 AM.png" /></span></P> <P><SPAN>The second consideration has to do with the Workspaces that you want the catalog to be accessed from.&nbsp;</SPAN></P> <P><SPAN>As a refresher, a Workspace is the environment users enter to run their workloads. It is common to have a Workspace for development and production use cases, as well as separating per business unit.&nbsp;</SPAN></P> <P><SPAN>Going back to the example of the company that sells solar-powered products, say you configured different catalogs to hold telemetry data from various products and divided it into data that should be used in development, testing, and production Workspaces. To ensure that production data held in the </SPAN><I><SPAN>solar_panels_prod</SPAN></I><SPAN> catalog can only ever be accessed from production Workspaces, data owners can use a feature called </SPAN><A href="https://docs.databricks.com/en/data-governance/unity-catalog/create-catalogs.html#workspace-catalog-binding-example" target="_blank" rel="noopener"><SPAN>Workspace-catalog binding</SPAN></A><SPAN>. This feature allows catalog owners to specify an allow-list of Workspaces from which their catalog can be accessed.&nbsp;</SPAN></P> <P><SPAN>Note that data access is still governed by user level permissions. If a user is given access to a table </SPAN><I><SPAN>solar_lights_prod.weather.actuals_daily</SPAN></I><SPAN>, then catalog bindings simply dictates which Workspaces the user can exercise that data access from.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="MurtN_19-1719858970187.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9105iD4E5CB9549BFBE9D/image-size/large?v=v2&amp;px=999" role="button" title="MurtN_19-1719858970187.png" alt="MurtN_19-1719858970187.png" /></span></P> <P><FONT size="2"><I><SPAN>Workspace-catalog bindings allow you to specify an allow-list of Workspaces that are trusted to access a catalog.</SPAN></I></FONT></P> <H3><SPAN>Permissioning the catalog</SPAN></H3> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2024-07-01 at 11.52.01 AM.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9119i22EDD35BDA5DE237/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2024-07-01 at 11.52.01 AM.png" alt="Screenshot 2024-07-01 at 11.52.01 AM.png" /></span></P> <P><SPAN>The third consideration has to do with the users, groups, and service principals that have permissions to operate on the catalog.</SPAN></P> <P><SPAN>As an admin, you will want to determine:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><STRONG>Who can create assets? </STRONG><SPAN>Usually includes data engineers and service principals who are running ETL pipelines to bring in and curate data in Databricks. Also you should consider a ‘scratch’ space for other users to write intermediate tables and results.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Who can read data?</STRONG><SPAN> Data scientists and analysts who are querying this data for various insights and development purposes.</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><STRONG>Who can view metadata?</STRONG><SPAN> General users who can search for/discover data to know it exists, and view lineage.</SPAN></LI> </UL> <P><SPAN>We strongly encourage using groups instead of users for assigning access to data secured in catalogs. This helps simplify access provisioning as every individual user doesn’t need to be granted access and can simply be added to the established group that has privileges on a data product.</SPAN></P> <P><SPAN>Groups should always be </SPAN><A href="https://docs.databricks.com/en/administration-guide/users-groups/scim/index.html#sync-users-and-groups-from-your-identity-provider" target="_blank" rel="noopener"><SPAN>synchronized with the system</SPAN></A><SPAN> they are managed in via the SCIM API (standard REST APIs for users/group synchronization). This can be accomplished using an Identity Provider (IdP), such as OKTA or Azure Active Directory.</SPAN></P> <P><SPAN>Let’s once again look at the example of the company that sells solar-powered products. A catalog has been configured called </SPAN><I><SPAN>solar_lights_prod</SPAN></I><SPAN> to hold all the telemetry data collected from this particular line of products. An admin starts out by creating 3 schemas in this catalog called bronze, silver, and gold based on the lakehouse </SPAN><A href="https://www.databricks.com/glossary/medallion-architecture" target="_blank" rel="noopener"><SPAN>medallion architecture</SPAN></A><SPAN>. Additionally, a “sandbox schema” is created to house ad-hoc asset creation. Groups can be leveraged here to configure who can do what. See the example below of reader/writer groups being used to provision access:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="MurtN_21-1719858970187.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9107iA3994DB5C5504E2D/image-size/large?v=v2&amp;px=999" role="button" title="MurtN_21-1719858970187.png" alt="MurtN_21-1719858970187.png" /></span></P> <P><FONT size="2"><I><SPAN>Permissioning setup example using groups and medallion architecture in Unity Catalog</SPAN></I></FONT></P> <P><SPAN>We encourage as a best practice for admins to grant the </SPAN><A href="https://docs.databricks.com/en/release-notes/product/2024/march.html#delegate-the-ability-to-view-an-objects-metadata-in-unity-catalog-public-preview" target="_blank" rel="noopener"><SPAN>BROWSE metadata privilege</SPAN></A><SPAN> to all users! Granting BROWSE on a catalog allows users to see metadata for its child objects in catalog explorer, </SPAN><I><SPAN>without</SPAN></I><SPAN> being able to read (SELECT) the actual data. Additionally, users can see the data lineage for these objects.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="MurtN_22-1719858970189.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9108i3FDD08336D29E3F2/image-size/large?v=v2&amp;px=999" role="button" title="MurtN_22-1719858970189.png" alt="MurtN_22-1719858970189.png" /></span></P> <P><FONT size="2"><I><SPAN>BROWSE allows users to see metadata for objects in a catalog encouraging data discovery</SPAN></I></FONT></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="MurtN_23-1719858970188.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9111iF1AFFEC3DCF4F254/image-size/large?v=v2&amp;px=999" role="button" title="MurtN_23-1719858970188.png" alt="MurtN_23-1719858970188.png" /></span></P> <P><FONT size="2"><I><SPAN>BROWSE makes data lineage accessible so users can better understand data flows</SPAN></I></FONT></P> <H3><SPAN>Enriching the catalog’s metadata</SPAN></H3> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Screenshot 2024-07-01 at 11.51.33 AM.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9118i64E7F65807187433/image-size/large?v=v2&amp;px=999" role="button" title="Screenshot 2024-07-01 at 11.51.33 AM.png" alt="Screenshot 2024-07-01 at 11.51.33 AM.png" /></span></P> <P><SPAN>The fourth and final consideration has to do with adding metadata for the catalog. Metadata includes tags and comments. Adding metadata is crucial to organize datasets in Unity and make it easier for workspace users to find the right data they are looking for. Furthermore, metadata helps provide in-product AI assistants greater context for more informed answers.</SPAN></P> <P><STRONG>Comments</STRONG><SPAN> support basic markdown syntax and can be used to add descriptions and links to enable easier data documentation and discovery for users.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="MurtN_25-1719858970184.png" style="width: 545px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9109i8A7EE1E774143CD2/image-dimensions/545x444?v=v2" width="545" height="444" role="button" title="MurtN_25-1719858970184.png" alt="MurtN_25-1719858970184.png" /></span></P> <P><FONT size="2"><I><SPAN>Markdown comments can be leveraged to give users info about data stewards for questions about datasets and links for how to gain access.</SPAN></I></FONT></P> <P><STRONG>Tags</STRONG><SPAN> are attributes containing keys and optional values that you can apply to securable objects in Unity Catalog. Tagging is useful for organizing and categorizing different securable objects within a metastore. Using tags also simplifies search and discovery of your data assets.</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="MurtN_26-1719858970185.png" style="width: 562px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9112i2811CA424C330A2B/image-dimensions/562x249?v=v2" width="562" height="249" role="button" title="MurtN_26-1719858970185.png" alt="MurtN_26-1719858970185.png" /></span></P> <P><FONT size="2"><I><SPAN>Add tags to classify data and indicate things such as data source and sensitivity level.</SPAN></I></FONT></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="MurtN_27-1719858970189.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9113i0593CF183069F487/image-size/large?v=v2&amp;px=999" role="button" title="MurtN_27-1719858970189.png" alt="MurtN_27-1719858970189.png" /></span></P> <P><FONT size="2"><I><SPAN>Users can search by tags to quickly find relevant datasets.</SPAN></I></FONT></P> <H2><SPAN>Get started today!</SPAN></H2> <P><SPAN>In this blog, we’ve discussed the 4 main considerations when setting up a catalog object and the best practices that go along with it. Try it out yourself by </SPAN><A href="https://docs.databricks.com/en/data-governance/unity-catalog/create-catalogs.html" target="_blank" rel="noopener"><SPAN>creating a catalog</SPAN></A><SPAN> today!</SPAN></P> Wed, 24 Jul 2024 18:35:10 GMT MurtN 2024-07-24T18:35:10Z https://community.databricks.com/t5/technical-blog/you-should-know-these-4-things-about-setting-up-unity-catalog/ba-p/76407 <P>This blog focuses on how to set up a key artifact in Unity - the catalog - and the things you should be thinking about when creating them!</P> Wed, 24 Jul 2024 18:35:10 GMT https://community.databricks.com/t5/technical-blog/you-should-know-these-4-things-about-setting-up-unity-catalog/ba-p/76407 MurtN 2024-07-24T18:35:10Z