I have two Azure subscriptions: one for Prod and another for Non-Prod. During the initial setup of the Non-Production Databricks Workspace, I configured the metastore storage account in the Non-Prod subscription. However, I now want to move this metastore storage account from the Non-Prod subscription to the Prod subscription. Azure provides an option to move storage accounts between subscriptions, but I'm concerned about whether this will break existing workflows, catalogs, and other components in both the Prod and Non-Prod Databricks workspaces
Assuming the metastore is the same for your DEV and PROD environments and what you want is just to use the same storage account + container to place managed tables, volumes, etc. in theory you just need to copy all content from your source storage account / container to the shared target storage account / container, taking into account that DEV and PROD will share "ADLS Gen 2 path".
If I tried to do that in my scenario below, I would have to copy all "tables" and "volumes" from source to target shared path (in green). Not sure if this works as I haven't tested but that would be the solution in your case.
On the other hand, as you can see only metadata about catalogs is stored in there, no info about workflows, jobs, etc.
As a suggestion, instead of using same default ADLS Gen 2, I would go with two different "external locations", one for DEV and the other one for PROD, in order to place catalogs in different containers associated to each "external location". You can use same storage account for both or even separated storage accounts for better performance and isolation.
I hope this helps.
Announcements
