Greetings @yxnz1e, I did some digging and here is what I found.
Thanks for coming back with your test results, and thanks to ThomazNeto for the groundwork. His read of the docs holds up on both points, so I'll build on it rather than repeat it.
- Empty catalogs taking longer
Slower doesn't mean stuck. The replication point is aggregate: it only advances once every in-scope asset has been copied together, and during INITIAL_REPLICATION the replication_lag_ms column stays NULL until each asset has replicated at least once. The errors column is the real indicator of a blocking problem. A NULL lag with no errors is normal early on. A lag that keeps climbing, or a group that sits in INITIAL_REPLICATION well past your other tests, is the thing to raise with your account team.
Two things can also skew a small test. Creating a group provisions a connection and a foreign catalog in each metastore and runs a first full cycle before the group goes ACTIVE, and that fixed overhead is the same whether the catalog holds zero rows or a few thousand. And system.replication.states can run up to three hours behind, so if you're timing from the system table, that delay alone can swamp the difference between two small catalogs. For the live view, use the failover group's detail page under Resilience in the account console. For the latest status event, this is the docs' query (the name is fully qualified and starts with accounts/..., which is why Thomaz used LIKE๐
SELECT event_time, replication_state, replication_lag_ms, errors
FROM system.replication.states
WHERE failover_group_name = :group_name
ORDER BY event_time DESC
LIMIT 1;
For future tests, put a schema and a small table in the catalog so there's something concrete to verify in the secondary. The docs recommend a separate read-only monitor workspace in that region, since replicated catalogs are read-only and you can't run compute on the secondary workspace.
- The deleted-metastore error
Thomaz's diagnosis matches the API. The failover group object only takes workspace IDs; the metastore IDs the system table shows under managed_assets come from those workspaces, and nothing in the docs says they refresh when a workspace's metastore assignment changes. So the group kept a reference to a metastore that no longer existed, and teardown had nothing to clean up on that side. Deleting the workspace detached it, emptied the group, and let the delete succeed, which fits that picture. Treat it as an observed outcome in a sandbox, though, not a documented cascade. Workspace deletion is irreversible and the docs don't promise it removes the group. In a real environment the path is still the one Thomaz gave: account team or Support, with the group name, state, old and new metastore IDs, workspace IDs, and the full error. I'd report it even though you're unblocked, since a stale metastore reference is something they'd want to know about. (The account-level Disaster Recovery API also has Get and Update on the failover group, and Update takes the workspace set. I haven't tested whether a PATCH can detach a workspace, so ask before relying on it.)
Cleanup now that the group is gone:
- Check the secondary metastore in Catalog Explorer for leftovers: replica catalogs plus the DR-created connection and foreign catalog (their comment says Databricks DR manages them). The secondary can't hold catalogs sharing names with ones you'll replicate, and leftover securables are a listed cause of
DR_INVALID_CONFIGURATION.OVERLAPPING_EXTERNAL_LOCATIONS, so they'll trip your next test. Never remove DR-created objects while a group exists, and confirm with your account team before removing orphans.
- Turn off Mission Critical on the surviving secondary workspace (Add-ons tab) or you'll keep paying that rate.
- Going forward, treat a workspace's metastore assignment as frozen while it's in a failover group. Documented order: delete the group, turn off Mission Critical, then make metastore changes.
Thanks for closing the loop. The thread now has a documented workaround for the next person who hits this.
References:
Regards, Louis.
