Metastore - One per Account/Region Limitation

AdamMcGuinness
New Contributor III

Looking at Databricks’ suggested use of catalogs. My instincts are now leading me to the conclusion having separate metastore for each SDLC environment (dev, test, prod) is preferable. I think if this pattern were followed, this means due to current constraints, a separate account for each environment is required as we would not want to be in different regions for the same account. This approach yields the full benefits of a three-level namespace as you are not giving up the top level to an environment as per this "best practice"
https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices#--or...

My rationale:

  • By dedicating a catalog to an environment, you do not get the full benefit of the three-level namespace e.g. for a source dataset:

    Catalog - bronze_systemA (One catalog dedicated to a source in the environment’s metastore)
              Schema – raw
                        Schema child objects
              Schema – historised (optional if you need to collect time series data from source)
                        Schema child objects
              Schema – curated (optional curation of source data without aggregating to other sources as you would in silver.)
                        Schema child objects

    Is better than:

    Catalog - bronze_all_systems_dev (one catalog dedicated to all sources by environment in same metastore)
               Schema - systemA_raw
                        Schema child objects
              Schema - systemA_historised
                        Schema child objects
              Schema - systemA_curated
                        Schema child objects
              Many more schemas

                          “
                          “

                          “

  • On platform deployments from lower to higher environments would not have to manage the change in catalog name where an object is referenced e.g. A view’s SQL definition:
    …..
    FROM bronze_systemA.raw.table_abc

    Is better than:
    …..
    FROM bronze_all_systems_dev.systemA_raw.table_abc

    When deploying to higher environments “_dev” needs to change.

    I anticipate this may also apply to other objects such as:
       Workflows
       DLT
       Jobs
       Maybe more ...

  • An external connection in an external tool will only have change connection string for the higher environment and not catalog name.

  • Binding of catalogs to workspaces provides a clean method to manage data access compared to cherry picking schemas into ACLs and associating with authorised users.

Interested if I have missed something and other points of view.

Thanks