topic Re: Problem with Metastore in Administration & Architecture

Problem with Metastore

jzu — Fri, 03 Oct 2025 15:02:34 GMT

Hello community.

We are facing an issue when deploying and configuring metastore using terraform.

We are using Azure Devops pipeline for deployment. The identity running the pipeline is a managed identity and it's set as account admin in Account portal.

When we run the pipeline first time everything is created successfully. However after some time, without any change to the code, we run the very same pipeline with following error: Error: cannot read metastore: User not authorized.

If I remove the metastore and create it again with the very same pipeline, it's successfully created. Even immediate pipeline runs are successful, after some time however we again get the error above. Is there any additional configuration required except what we have?

provider "databricks" {

alias = "accounts"

host = "https://accounts.azuredatabricks.net"

account_id = local.account_id

}

resource "databricks_metastore" "metastore" {

name = var.metastore_name

region = var.location

owner = var.metastore_owner

provider = databricks.accounts

}

Any help is highly appreciated.

Thank you.

Jozef

Re: Problem with Metastore

Louis_Frolio — Fri, 03 Oct 2025 16:17:02 GMT

Greetings @jzu , I did some digging around with internal docs and references and put together some helpful tips and things to consider.

This is a common authorization issue related to permission propagation delays and ownership configuration when managing Databricks Unity Catalog metastores with Terraform using service principals or managed identities.

Root Cause

The "User not authorized" error occurs after the initial successful deployment because of how Databricks handles metastore ownership and permission caching. When the metastore is created, the `owner` parameter transfers ownership away from the creating identity (the managed identity) to the specified owner. Once this happens, the managed identity running the pipeline no longer has implicit permissions to read or manage the metastore, causing subsequent Terraform runs to fail when trying to read the metastore state.

Additionally, metastore admin assignment changes can take up to 30 seconds to propagate across the account, and may take even longer to take effect in workspaces due to caching protocols.

Solutions

Make the Managed Identity the Metastore Owner

Instead of setting a different owner, keep the managed identity as the owner of the metastore. Modify the resource configuration:

```hcl+
resource "databricks_metastore" "metastore" {
name = var.metastore_name
region = var.location
owner = azuread_service_principal.pipeline_identity.application_id
provider = databricks.accounts
}
```

This ensures the managed identity retains ongoing permissions to manage the metastore.

Assign Metastore Admin Role

If the managed identity needs to use a different owner, explicitly grant the managed identity the Metastore Admin role. This can be done through the Azure Databricks account console or via Terraform after the metastore is created:

```hcl
resource "databricks_metastore" "metastore" {
name = var.metastore_name
region = var.location
owner = var.metastore_owner
provider = databricks.accounts
}

resource "databricks_grant" "metastore_admin" {
metastore = databricks_metastore.metastore.id
principal = azuread_service_principal.pipeline_identity.application_id
privileges = ["CREATE_CATALOG", "CREATE_STORAGE_CREDENTIAL", "CREATE_EXTERNAL_LOCATION"]
provider = databricks.accounts
}
```

Account admins who create metastores become the initial metastore admin automatically, but when ownership is transferred, explicit permissions are needed.

Use a Dedicated Group for Metastore Administration

Databricks recommends using an Azure AD group as the metastore owner/admin rather than individual identities. Add the managed identity to this group:

1. Create an Azure AD group for Databricks administrators
2. Add the managed identity to this group
3. Set the group as the metastore owner

This approach provides better permission management and reduces propagation issues.

Account-Level vs Workspace-Level Provider

While using the account-level provider (`databricks.accounts`) is correct for metastore creation, be aware that some operations may work better with workspace-level providers. However, for metastore creation specifically, the account provider is the appropriate choice.

Verify Account Admin Status

Ensure the managed identity is properly configured as an Account Admin in the Databricks account console. Being an account admin alone may not be sufficient if ownership is transferred without explicit ongoing permissions.

Hope this helps, Louis.

Re: Problem with Metastore

jzu — Tue, 04 Nov 2025 13:12:38 GMT

I would wonder who set this to Solved. It is not solved at all. When I try the approach suggested I get:

Error: cannot create grant: invalid Databricks Workspace configuration

which is really confusing because the suggested solution is using account provider config.

Could you elaborate what might be the problem?

Thank you

Re: Problem with Metastore

Advika — Tue, 04 Nov 2025 13:58:38 GMT

Hello @jzu, I’ve unmarked the reply as the accepted solution for now since the issue persists.
Tagging @Louis_Frolio for further insights on the invalid Databricks Workspace configuration error.

Re: Problem with Metastore

Louis_Frolio — Tue, 04 Nov 2025 16:26:34 GMT

@jzu , is this a new error or is it the same as before. I need more details please. Louis.

Re: Problem with Metastore

jzu — Tue, 11 Nov 2025 10:59:13 GMT

Hello Louis,

I can give you a little background. We are using azure pipelines to deploy 4 databricks environments using terraform. We using service connection with underlying user assigned managed identities with workload federation in ado. We had several support calls with Databricks, unfortunatelly no technician we worked with was able to configure terraform deployment correctly with uami that would include creation of metastore and all workspaces and could run twice. Eventually we ended up with using local databricks spn with local password. We are quite upset we could not configure it passwordless, but we cannot invest more time into this.