cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Managed Table

Mani2105
New Contributor

Hi Experts,

I have a workspace created and associated a metastore with it, the metastore points to a storage location USDATA and then I create two catalogs in the workspace and one is using default meta store as the external storage location and other catalog is sales and i created a external sales storage location and provided security access. 

Mani2105_0-1730143852722.png

Abive screen shot  says storage for managed tables on external locations, so now my question is if i create a table in the sales catalog without  specifiying any external location, will the tables created be managed and will go to the Sales storage account and in the event of deleting the table ,will it delete the files associated as welland auto optimize? What will my metastore USDATA  have information about the sales catalog , if the sales catalog had a seperate storage location , its the metadata about the catalog goes to metastore?

1 REPLY 1

agallardrivilla
New Contributor II

Hi @Mani2105,

if i create a table in the sales catalog without  specifiying any external location, will the tables created be managed and will go to the Sales storage account

๐Ÿ‘‰ Yes, if you create a table in the sales catalog without specifying any external location, this table will be automatically managed and the data will be stored in the default storage location configured for the sales catalog.

and in the event of deleting the table ,will it delete the files associated as welland auto optimize?

๐Ÿ‘‰ Yes, in the case of managed tables within Unity Catalog in Databricks, deleting the table will also delete the associated files stored in the catalogโ€™s storage location

  • Delta Lakeโ€™s auto-optimization features (like autoCompact and optimizeWrite) apply to managed tables. If youโ€™ve enabled these settings, they will continuously optimize the storage layout, such as compacting small files or applying Z-ordering, to improve query performance and storage efficiency.

  • You can enable auto-optimization for the entire workspace, catalog, or individual tables using configuration settings:

 

# Enabling auto-optimization
spark.conf.set("spark.databricks.delta.autoCompact.enabled", "true")
spark.conf.set("spark.databricks.delta.optimizeWrite.enabled", "true")

 

What will my metastore USDATA have information about the sales catalog , if the sales catalog had a seperate storage location , its the metadata about the catalog goes to metastore?

๐Ÿ‘‰ The USDATA metastore stores metadata for all catalogs in your workspace, including the Sales catalog. Although Sales has a separate storage location, only metadata about Sales (such as its tables, schemas, and storage path) is stored in the USDATA metastoreโ€”the actual data files reside in the storage location designated for Sales.

The metastore (USDATA) holds metadata about all catalogs, schemas, and tables within your Databricks workspace, including:

  • Information about each catalog (e.g., Sales).
  • Schemas (databases) within each catalog.
  • Tables and views, including details such as columns, data types, and table properties.
  • Access control configurations, permissions, and security settings.

Let me know if youโ€™d like more details on any part of this process!

Regards!

 

Alfonso Gallardo
-------------------
๏”ง I love working with tools like Databricks, Python, Azure, Microsoft Fabric, Azure Data Factory, and other Microsoft solutions, focusing on developing scalable and efficient solutions with Apache Spark

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group