cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

Changing Root Metastore Location

yawningbrain
New Contributor II

We currently have a default root metastore location linked to an ADLS container.
At the same time we are working on a ADLS migration project.
All of the tables and files (everything we use for day to day business related) have moved to the new ADLS so there are no dependencies on the legacy ADLS - except for the system and __databricks_internal catalogs.

How can we safely change the root metastore location without breaking our environment - ideally with minimal to 0 impact on our pipelines and system metadata.

1 REPLY 1

ivanvyd
New Contributor II

@yawningbrain this is a great topic. The Azure Databricks documentation⁠ supports removing the existing metastore root and adding a new path. You cannot edit the existing path in place. This does not migrate existing data.

When you remove the root, catalogs without their own storage location inherit the old path as their catalog-level storage. That preserves access, but means replacing the metastore root does not necessarily remove dependencies on the legacy account.

The two catalogs you mentioned need separate consideration:

  • system: Operational system-table data is stored in a Databricks-hosted storage account, according to the system tables reference⁠. A legacy path displayed in catalog properties would not, by itself, establish that those tables are physically stored there.
  • __databricks_internal: Databricks identifies this as an internal catalog used for pipeline materializations and advises against accessing its tables directly. And I never tried to. For that dependency, I would get Databricks Support to confirm the migration or cleanup procedure before changing internal paths or retiring the storage account.

That said, keep the old account and its access permissions intact until those dependencies are resolved - the documented root-removal behavior is not a guarantee of zero impact when the underlying storage is deleted.

Are you seeing actual objects stored in the legacy account, or only the old path in these catalogs' properties?