Hi there,

I'm currently migrating Databricks (metastores, workspaces, etc.) from Azure to AWS using the Databricks migration tool. During the migration process and after digging in the code, I've come to the conclusion that the tool only supports a migration of the the built-in Hive metastore but not custom metastores in Unity Catalog.

Q1: Is this correct or have I overseen something?

If yes, I thought about extending the code, i.e. adapting the export / import functionality for the Hive metastore in the HiveClient to support other custom metastores from Unity Catalog as well. Specifically, this would mean adaptations in the MetastoreExportTask and MetastoreTableACLExportTask.

Q2: Is there anything else to consider?

I've also realised, that the migration tool is only exporting / importing database and table definitions, but not the data itself. This is also stated on the migration tool's documentation page.

"Note on DBFS Data Migration:

DBFS is a protected object storage location on AWS and Azure. Please contact your Databricks support team for information about migrating DBFS resources."

Q3: What is the preferred way to migrate the data in the DBFS from Azure to AWS? Is it possible to just move all files / folders under the old DBFS root to the new DBFS root?

Thanks a lot in advance! 🙂