topic Unity Catalog - Metastore is it at account or workspace in Data Governance

Unity Catalog - Metastore is it at account or workspace

SanjayJanardhan — Sun, 04 Dec 2022 04:32:20 GMT

I was trying to understand Unity Catalog, if metastore is at the account level or the each workspace in the account will have there own metastore?

My understanding is the metastore cant be shared between two workspace.

Re: Unity Catalog - Metastore is it at account or workspace

labtech — Sun, 04 Dec 2022 05:11:12 GMT

yes you're right, metastore can be shared between workspace and it is an isolated for example the first time when we create a workspace with Aws (because i familiar with it) an default hive metastore will be created. And later then we can create a new metastore and can connect that metastore to exist workspace so now we have two metastore for using purpose.

Re: Unity Catalog - Metastore is it at account or workspace

Hubert-Dudek — Sun, 04 Dec 2022 11:55:40 GMT

Yes, as @Gam Nguyen said. Additionally, the Unity catalog is regional so you can have 1 per region.

Re: Unity Catalog - Metastore is it at account or workspace

bergmaal — Thu, 15 Dec 2022 21:13:00 GMT

The whole point of Unity Catalog is that you can share it between workspaces. It sits at the account level, and you assign workspaces to it from the Account Console (or script assignment through Terraform or similar technologies).

The beauty of Unity Catalog is that you can share the same data between workspaces hassle free, and you can secure the objects (catalogs, schemas, tables, etc) which means you will have 1 place to do governance, instead of having to replicate it for each individual workspace (like you'd have to do if using the classic hive_metastore). This means that if you write data to a UC table from workspace1, it will immediatly be visible in workspace2, and vice versa.

Unity Catalog metastore also uses a data lake for default storage, which you create and control in you cloud provider. This means you have way more flexibility in setting up security on the storage, which you don't have if you are using hive. Hive uses dbfs, which is not secure by any means, and should not be used for storing sensitive data.

Re: Unity Catalog - Metastore is it at account or workspace

Aviral-Bhardwaj — Fri, 16 Dec 2022 15:52:05 GMT

@Sanjay Janardhan , no i have done this, you can share it in multiple workspaces