Metastore creation - Azure Databricks - Internal Server Error

[YakupOeztuerk]

Recently it has not been possible to create a metastore. The error "Internal Server Error" appears. Despite authorization of the Managed Identities as Storage Blob Data Contributor. Apparently the error has started showing up on a few more recently, so I'm creating a new ticket.

[m997al]

Hi @Retired_mod ... we are having the same problem.  In your instructions, you say to give the Databricks access connector the "Storage Blob Data Contributor" AND the "Storage Blob Data Owner" roles on the Storage Account.  Definitely did not see the "Storage Blob Data Owner" role specified in the instructions.  Will try and see if that was the issue.

[m997al]

Hi @Retired_mod... no luck.  Still the same error of "Access Validation Failed" with a sub-error of "Internal server error" when we try to create the unity catalog metastore from the Databricks account console.  To the best we can tell, everything else works.  Databricks is currently working for us, this is just a protoype unity catalog that we are setting up to create some new workspaces for.  But no luck on the "create metastore" step.

[arpit]

We have identified this issue and are reviewing it

[youssefmrini]

Based on the error message you provided, it seems like there might be an issue with the Databricks service itself. Since this is an internal server error, it’s likely something that only the Databricks support team can resolve.

I recommend contacting the Databricks support team directly. You can submit a ticket through the Databricks support portal or email them at support@databricks.com. Be sure to include as much detail as possible about the issue, including any relevant error messages or logs.

In the meantime, you can try re-creating the metastore at a later time to see if the issue has been resolved.

[arpit · Accepted solution]

We confirm that this was some regression on Databricks and we have rolled out the fix for it. You can try to test again.

[m997al]

Hi @arpit - that is great news.  

As an aside... in the instructions, of course a Global AAD Admin needs to be the first to the Databricks account console and create the metastore...but... does it matter if it is someone else who sets up the Azure Storage account for the metastore (even with all the right permissions and ownership), or does the Azure Storage account creation have to also be done by the Global Admin?

I'm guessing the answer is "no", that someone with enough permissions over the storage account and Databricks access connector can do all that... but we weren't quite positive everything that is being tried by Databricks at the step of "Create metastore" (in case it is something beyond creating data in the storage container)...

Thanks!

[m997al]

Just following up - confirmed that the fix from Databricks solved our problem.  Also, confirmed that the Global Admin does NOT have to be the person to create the Azure Storage Account for the metastore, or the Databricks Access Connector.  The Global Admin does need to be the first to create the metastore within the Databricks account console, but then can assign Databricks account console access to others.

So it all worked.  Thank you!

[comcrazy]

Hi all, I got a similar error as described in this discussion. Internal server error (500) on https://accounts.azuredatabricks.net/api/2.0/accounts/4bfcxxxx-bfea-483f-b59b-c2b2e2xxxx4/metastores...

with following response

{
"error_code": "INTERNAL_ERROR",
"message": "",
"details": [
{
"@type": "type.googleapis.com/google.rpc.RequestInfo",
"request_id": "cc16c5cc-13dc-49a0-8473-8685bfa73a34",
"serving_data": ""
}
]
}

Can advise how to solve this? 

Thank you.