topic Re: Provisioning Unity Metastore for each environment using Terraform in Data Governance

Provisioning Unity Metastore for each environment using Terraform

mathiaskvist — Thu, 10 Aug 2023 08:56:55 GMT

Unity Metastore can currently only be provisioned once for each Azure region. I have three environments in Azure each consisting of a separate Subscription and Resource Group (RG). Each RG contains a Databricks Workspace and a Storage Account holding data. This setup is provisioned using Terraform and each environment has its' own state file. Thus I would like to provision a Unity Metastore for each environment, which is not possible, since there can only be one metastore per region.

This means that only one of my environments can provision the Unity Metastore which will lead to my Terraform code being a huge mess as it should be environment aware. E.g. when running Terraform for the dev environment it would create the metastore and when running for production it would import the metastore as a Terraform data source (or vice versa).

What do others do to handle this issue?

Re: Provisioning Unity Metastore for each environment using Terraform

Corbin — Thu, 10 Aug 2023 16:13:55 GMT

You're not wrong. How you've described it is currently the only way to do it with TF. This should be resolved once the account-level API for UC is incorporated into TF (see PR here). Then you'd have 4 statefiles, 1 for account-level tf state and 3 for dev/test/prod. Until that day, you're correct that you need to create in one and reference in the others. Quite clunky and definitely looking forward to account-level UC API integrated into TF.

Re: Provisioning Unity Metastore for each environment using Terraform

Corbin — Fri, 25 Aug 2023 13:55:46 GMT

Ask and ye shall receive: https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.24.0

They've implemented account-level UC API in terraform now!