Hi everyone,
I'm currently in the process of migrating to Unity Catalog. I have several Azure Databricks Workspaces, one for each phase of the development phase (development, test, acceptance, and production).
In accordance with the best practices (https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/best-practices#--or...) I've created a separate catalog for each workspace/environment.
This works great but it poses a challenge to my SQL notebooks that reference this catalog. Since my catalog name on development is different from test, acceptance, and production and I'm using CI/CD to deploy my notebooks I need a way to dynamically change the catalog that is being referenced in my SQL notebooks.
So far I've come up with the following two options:
1. Create a widget in my SQL notebooks that accept a catalog name and use a python cell to change to the right catalog by using spark.sql("use catalog xxxx")
2. Set the default catalog for each workspace to the corresponding catalog by using the Databricks rest API (Update an assignment | Metastores API | REST API reference | Azure Databricks).
Although these are both valid options in my opinion I'm wondering if I'm overlooking something or if there might be a better way to approach this. Any suggestions that you might have are very welcome.
Thanks in advance!
