topic Deploying Metastore with Terraform in Data Engineering

Deploying Metastore with Terraform

JCooke — Mon, 30 Jun 2025 11:49:37 GMT

my goal is to be able to enable unity catalog on a clean Azure deployment of databricks with absolutely no history of databricks.

I know I need to create a metastore for the Azure Region. And to do this I know I need Account Admin from the accounts page of databricks.

So if I deploy a new Azure Databricks workspace from terraform, how am I able to gain account admin for an account automatically?

It seems you need account admin to give account admin, I know manually I would get an azure admin to log into the accounts page and assign a new account admin, but how can I do that via terraform? Or does there always have to be this manual step?

If there does have to be a manual step - is it possible to do this prior to the creation of the workspace? e.g. setup before any pipeline would execute running the terraform commands?

how have other people done this?

Re: Deploying Metastore with Terraform

szymon_dybczak — Mon, 30 Jun 2025 13:17:44 GMT

Hi @JCooke ,

The first assignment of the Databricks Account Admin role is a bit of a special case. There is always a manual step required to assign the first Account Admin in a new Databricks account on Azure. This step cannot be fully automated via Terraform (or any other API) for security reasons (because it requires Microsoft Entra ID Global Administrator roles (which as you can guess is really high privilege ).

But after that first step you can create an Entra ID group and assing to that group required permission (for example ability to create metastore etc.)
Then you devops identity could be added to that group and you'll have ability to create metastore, workspaces in a fully automated way using terraform (or apis, scripts etc.)

Take a look at article written by my colleague. He managed to automate full process in terraform:

Terraforming Databricks #1: Unity Catalog Metastore – Seequality

Re: Deploying Metastore with Terraform

JCooke — Mon, 30 Jun 2025 13:17:41 GMT

@szymon_dybczak

Hi,

Yes, I read this article earlier - I thought this was the case, that an account admin needs to log in and crate a service principal that has account admin or such. I was just looking to get this confirmed.

So the flow would be;

1. Someone registers their with Databricks.com
They manually log in and create an account admin / service principal with account admin
Terraform can use that - do its stuff

forgive my ignorance (im not an azure admin or bill payer) - can you register with accounts.databricks.com prior to having workspaces created? e.g. could I do this as a pre-requisite for any pipelines to deploy infrastructure

Re: Deploying Metastore with Terraform

szymon_dybczak — Mon, 30 Jun 2025 13:31:20 GMT

Hi @JCooke ,

Yes, just go to: accounts.azuredatabricks.net and log in with account that has global administrator role privilege. Azure Databricks automatically creates an account admin role for you