Move metastore to another azure subscription

[alesventus]

Hi, 

We need to migrate our metastore with Unity Catalog to a new Azure subscription while remaining in the same Azure region. Currently, we have two workspaces attached to a single Unity Catalog. I’m looking for the best approach to move the metastore from one Azure subscription to another.

One idea is to detach both Databricks workspaces and physically move the ADLS—which contains our data and Unity Catalog—to the new subscription. After the move, we would delete the old metastore and reconnect the workspaces to the new metastore. Will this approach work? Has anyone had experience with migrating a Databricks metastore?

Additionally, where is the information about catalogs, schemas, and table names stored within the metastore? I have only found a folder named "unitycatalogs" containing generic table names.

[Nivethan_Venkat]

Hi @alesventus,

There are few points to be considered before migrating from One metastore to another. We need to see how the catalogs, schemas and tables are created as of now.

If you have created everything has managed like managed catalog, schema and table everything will remain in your unitycatalogs folder that you have highlighted. You can recreate the metastore with the same ADLS container after you have migrated.

If the schemas and tables are created as external. The path(s) will be pointed one-to-one in your ADLS location, so you can recreate the schema with the same locations pointed earlier.

But there are few points to be considered when it comes to managed tables, if you are deleting a metastore:

Attaching the link for your reference: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-metastore

Please let us know for more clarifications needed.

Best Regards,
Nivethan V

[alesventus]

Hi Nivethan,

Thanks for your thoughts. 

So, if all my tables are managed, then all I have to do is drop existing metastore, move the storage account to another subscription and recreate it based on moved data?

All objects like catalogs, schemas and tables should be automatically recreated when I connect to new metastore?

Thanks.

[SaugatMukherjee]

Hi,

I have done this in the past successfully. You do not have to delete a metastore if you are moving anywhere within the same tenant.

Metastore is essentially 2 things

1. Storage account , whose identifier is of the format abfss://container@datalakename.dfs.core.windows.net. If you see it only has the datalekename. When you move to a new subscription, the datalakename does not change. At least in azure, providing you are just moving it to a new subscription. Since the metastore only depends on the datalake name, you can move it anywhere provided the name remains the same.

2. Databricks access connector- this we found can’t be moved to another location. So we created a new one. You should ideally create a separate access connector for the metastore, separate from ones accessing other locations/resources.

what we did:

1. Moved the storage account associated with the metastore to the new location using azure portal’s move option . You can also use cli. 

2. created a new access connector in the same subscription as the metatsore as we wanted that to be separate.

3. Assign the new access connector “storage blob data contributor role” to the metastore storage account.

4. provided you are a metastore admin, login to any workspace associated with the metastore, go to the catalog menu, select settings->external locations. Click on the external location, corresponding to the metastore. Yes, workspace attached to a metastore has the details under external locations. Click edit and update it with the new access connector that you created in step 2. 

Steps 4 is required, as from Account console, after a metatstore is created, you can’t edit it from the ui. But you can from catalog settings on a connected workspace. You can also do the same from rest api/cli.

We did it smoothly without any glitch.