topic Managed Tables on Azure databricks in Data Engineering

Managed Tables on Azure databricks

YOUKE — Fri, 29 Nov 2024 11:34:52 GMT

Hi everyone,

I was trying to understand: when a managed table is created, Databricks stores the metadata in the Hive metastore and the data in the cloud storage managed by it, which in the case of Azure Databricks will be an Azure Storage Account. But i don’t see this storage account from the Azure portal?

The second question is: besides the fact that if the table is managed, and if the table or the Azure Databricks resource is deleted, both the data and the metadata are lost, does the same apply to clusters? So, if I create a managed table TEST with cluster A and i delete the cluster A, and then recreate cluster B, do I still see the table TEST or is it lost in this case as well?

Re: Managed Tables on Azure databricks

MuthuLakshmi — Fri, 29 Nov 2024 11:49:46 GMT

@YOUKE
Are you trying to check the storage location of the schema?
Managed tables in Hive metastore are always created in the storage location of the containing schema.
https://learn.microsoft.com/en-us/azure/databricks/database-objects/hive-metastore

Regarding your other question. The tables are not cluster specific. If you have created a managed table from cluster A it will be accessible from cluster B as well if you have enough privilege to access the table.

Re: Managed Tables on Azure databricks

YOUKE — Fri, 29 Nov 2024 12:15:20 GMT

Yes, exactly. I’m trying to understand where they are stored, but it seems to me that they are present in the Hive Warehouse, which should be the storage managed by Azure Databricks using Apache Hive, it's correct?

Re: Managed Tables on Azure databricks

szymon_dybczak — Fri, 29 Nov 2024 12:22:47 GMT

Hi @YOUKE ,

As @MuthuLakshmi pointed out, managed tables in Hive metastore are always created in the storage location of the containing schema. So, if you don’t provide a location when creating schema in hive_metastore, the default location user/hive/warehouse/schema-name is used. This location is on the DBFS root, which is not recommended for storing any production data.

Re: Managed Tables on Azure databricks

BraydenJordan — Fri, 06 Dec 2024 05:39:31 GMT

Thank you so much for the solution.