topic Re: Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region in Data Engineering

Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region

george_ognyanov — Thu, 09 Nov 2023 09:54:44 GMT

I am trying to create a metastore via the Terraform Azure databricks_metastore resource but I keep getting the error:

This is the exact code I am using to create the resource:

I have tried using both my Databricks account and a service principal application id, which are both in fact Account Admins, for the owner argument in the code block above to no avail.

Needless to say, there is no metastore in the region I am trying to create it in and if I use the GUI with my account I am able to successfully create and assign the metastore to the workspace.

Re: Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region

george_ognyanov — Thu, 09 Nov 2023 10:49:21 GMT

Hi @Retired_mod the image above is the entire error output I get when I execute terraform apply.

Error: cannot create metastore: Failed to check metastore quota limit for region westeurope
│
│ with databricks_metastore.this,
│ on main.tf line 150, in resource "databricks_metastore" "this":
│ 150: resource "databricks_metastore" "this" {

Re: Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region

george_ognyanov — Thu, 09 Nov 2023 11:15:19 GMT

Hi @Retired_mod as far as I understand one region can have one metastore. I am able to create a metastore in the same region if I log into the Databricks GUI and do it there.

Alternatively, if I already have a metastore created and try to execute the above terraform code I get the error that only one metastore is acceptable - meaning terraform can see the quotas and recognize that there already is a metastore. Unfortunately, when I delete the metastore from the GUI and try to create it from terraform I am back to square one with the error above.

Re: Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region

PetrSindelar100 — Wed, 29 Nov 2023 09:58:41 GMT

Hi, @george_ognyanov were you able to resolve this issue? I am facing exactly the same problem.

Re: Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region

george_ognyanov — Wed, 29 Nov 2023 10:08:47 GMT

Hi @PetrSindelar100 I am almost 90% sure the issue is using a free azure & Databricks accounts. I am working with my company on testing it out once the premium accounts are provisioned and will get back with the result here.

I have asked the same question on StackOverflow, sending the link to the post there so you can see the discussion:

https://stackoverflow.com/questions/77446106/terraform-azure-databricks-unity-catalogue-failed-to-check-metastore-quota-lim

Re: Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region

PetrSindelar100 — Wed, 29 Nov 2023 10:48:27 GMT

Thank you for your quick answer. I just figured out my mistake. I noticed that my problem was, that the service principal that I used for running terraform didnt have sufficient privileges to create a metastore. Once I have added the account admin rule to him in the Databricks Account, it started working. I hope this helps to you too.