Terraform Azure Databricks Unity Catalogue - Failed to check metastore quota limit for region

[george_ognyanov]

I am trying to create a metastore via the Terraform Azure databricks_metastore resource but I keep getting the error:

 This is the exact code I am using to create the resource:

I have tried using both my Databricks account and a service principal application id, which are both in fact Account Admins, for the owner argument in the code block above to no avail.

Needless to say, there is no metastore in the region I am trying to create it in and if I use the GUI with my account I am able to successfully create and assign the metastore to the workspace.

[Kaniz]

Hi @george_ognyanov, Could you please paste the error stack here?

[george_ognyanov]

Hi @Kaniz the image above is the entire error output I get when I execute terraform apply. 

Error: cannot create metastore: Failed to check metastore quota limit for region westeurope
│
│ with databricks_metastore.this,
│ on main.tf line 150, in resource "databricks_metastore" "this":
│ 150: resource "databricks_metastore" "this" {

[Kaniz]

Hi @george_ognyanov , Make sure that your account in the West Europe region has sufficient quota limits for creating a metastore. Sometimes, hitting resource limits can cause this kind of error.

[george_ognyanov]

Hi @Kaniz as far as I understand one region can have one metastore. I am able to create a metastore in the same region if I log into the Databricks GUI and do it there.

Alternatively, if I already have a metastore created and try to execute the above terraform code I get the error that only one metastore is acceptable - meaning terraform can see the quotas and recognize that there already is a metastore. Unfortunately, when I delete the metastore from the GUI and try to create it from terraform I am back to square one with the error above. 

[PetrSindelar100]

Hi, @george_ognyanov  were you able to resolve this issue? I am facing exactly the same problem.

[george_ognyanov]

Hi @PetrSindelar100 I am almost 90% sure the issue is using a free azure & Databricks accounts. I am working with my company on testing it out once the premium accounts are provisioned and will get back with the result here. 

I have asked the same question on StackOverflow, sending the link to the post there so you can see the discussion: 

https://stackoverflow.com/questions/77446106/terraform-azure-databricks-unity-catalogue-failed-to-ch...

[PetrSindelar100 · Accepted solution]

Thank you for your quick answer. I just figured out my mistake. I noticed that my problem was, that the service principal that I used for running terraform didnt have sufficient privileges to create a metastore. Once I have added the account admin rule to him in the Databricks Account, it started working. I hope this helps to you too.