cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Clarification on Unity Catalog Metastore - Metadata and storage

APJESK
New Contributor III


Where does the Unity Catalog metastore- metadata actually reside?

Is it stored and managed in the Databricks account (control plane)?

Or does it get stored in the customer-managed S3 bucket when we create a bucket for Unity Catalog metastore?

I want to clarify whether the S3 bucket we configure is only for managed tables/volumes, while the UC metadata itself stays with Databricks.

2 REPLIES 2

Brahmareddy
Esteemed Contributor

Hi @APJESK,

How are you doing today? as per my understanding, the Unity Catalog metadata itself (like table definitions, permissions, lineage, etc.) is always stored and managed in the Databricks control plane, not in your S3 bucket. The S3 bucket you configure when creating a UC metastore is only used to hold the actual data files for managed tables and volumes. So think of it this way: the metadata lives with Databricks, while the data lives in your bucket. This separation makes sure Databricks can handle governance and consistency, while you keep ownership of the storage.

Please let me know for any queries.

Regards,

Brahma

nayan_wylde
Honored Contributor III

1. Unity Catalog metadata (schemas, tables, permissions, lineage, etc.) is stored and managed in the Databricks control plane, not in your S3 bucket. This metadata resides in a Databricks-managed database and is not in your customer-managed storage.


2. The S3 bucket you configure when creating a Unity Catalog metastore is only for storing data files of managed tables and volumes (and optionally for external locations). It does not hold the metadata itself.

3. If you donโ€™t specify a storage location, Databricks will use its own managed storage account for managed tables.

So, in short:

  • Metadata โ†’ Databricks control plane
  • Data files (managed tables/volumes) โ†’ Your configured S3 bucket (or Databricks default if none provided)

Sharing some previous links.

https://community.databricks.com/t5/data-engineering/unity-catalog-metastore-details/td-p/66232

https://community.databricks.com/t5/data-governance/where-is-the-default-unity-catalog-metastore/td-...