cancel
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forĀ 
Search instead forĀ 
Did you mean:Ā 

Managed Tables on Azure databricks

YOUKE
Visitor

Hi everyone,

I was trying to understand: when a managed table is created, Databricks stores the metadata in the Hive metastore and the data in the cloud storage managed by it, which in the case of Azure Databricks will be an Azure Storage Account. But i donā€™t see this storage account from the Azure portal?

The second question is: besides the fact that if the table is managed, and if the table or the Azure Databricks resource is deleted, both the data and the metadata are lost, does the same apply to clusters? So, if I create a managed table TEST with cluster A and i delete the cluster A, and then recreate cluster B, do I still see the table TEST or is it lost in this case as well?

1 ACCEPTED SOLUTION

Accepted Solutions

MuthuLakshmi
Databricks Employee
Databricks Employee

@YOUKE 
Are you trying to check the storage location of the schema?
Managed tables in Hive metastore are always created in the storage location of the containing schema. 
https://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore

Regarding your other question. The tables are not cluster specific. If you have created a managed table from cluster A it will be accessible from cluster B as well if you have enough privilege to access the table. 

 

 

View solution in original post

3 REPLIES 3

MuthuLakshmi
Databricks Employee
Databricks Employee

@YOUKE 
Are you trying to check the storage location of the schema?
Managed tables in Hive metastore are always created in the storage location of the containing schema. 
https://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore

Regarding your other question. The tables are not cluster specific. If you have created a managed table from cluster A it will be accessible from cluster B as well if you have enough privilege to access the table. 

 

 

YOUKE
Visitor

Yes, exactly. Iā€™m trying to understand where they are stored, but it seems to me that they are present in the Hive Warehouse, which should be the storage managed by Azure Databricks using Apache Hive, it's correct?

Hi @YOUKE ,

As @MuthuLakshmi  pointed out, managed tables in Hive metastore are always created in the storage location of the containing schema. So, if you donā€™t provide a location when creating schema in hive_metastore, the default location user/hive/warehouse/schema-name is used. This location is on the DBFS root, which is not recommended for storing any production data.

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonā€™t want to miss the chance to attend and share knowledge.

If there isnā€™t a group near you, start one and help create a community that brings people together.

Request a New Group