Catalog name's in DTAP scenario

ThomasVanBilsen — Thu, 03 Aug 2023 08:15:36 GMT

Hi everyone,

I'm currently in the process of migrating to Unity Catalog. I have several Azure Databricks Workspaces, one for each phase of the development phase (development, test, acceptance, and production).

In accordance with the best practices (https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices#--organize-your-data) I've created a separate catalog for each workspace/environment.

This works great but it poses a challenge to my SQL notebooks that reference this catalog. Since my catalog name on development is different from test, acceptance, and production and I'm using CI/CD to deploy my notebooks I need a way to dynamically change the catalog that is being referenced in my SQL notebooks.

So far I've come up with the following two options:

1. Create a widget in my SQL notebooks that accept a catalog name and use a python cell to change to the right catalog by using spark.sql("use catalog xxxx")

2. Set the default catalog for each workspace to the corresponding catalog by using the Databricks rest API (Update an assignment | Metastores API | REST API reference | Azure Databricks).

Although these are both valid options in my opinion I'm wondering if I'm overlooking something or if there might be a better way to approach this. Any suggestions that you might have are very welcome.

Thanks in advance!

Re: Catalog name's in DTAP scenario

-werners- — Thu, 03 Aug 2023 11:31:33 GMT

you could also store the environment name in a config file f.e. in the databricks filestore.

These config files can also be managed by ci/cd.
tbh my preferred way of working lately.

topic Re: Catalog name's in DTAP scenario in Data Engineering

Catalog name's in DTAP scenario

Re: Catalog name's in DTAP scenario