topic Re: Azure Databricks with metastore, cannot create managed table in Administration & Architecture

Azure Databricks with metastore, cannot create managed table

m997al — Fri, 20 Oct 2023 12:08:04 GMT

We have set up Azure Databricks with Unity Catalog (metastore) in an ADLS Gen2 storage account.

Used Managed Identity (Databricks Access Connector) for connection from workspace(s) to ADLS Gen2
ADLS Gen2 storage account has Storage Blob Data Owner and Storage Blob Data Contributor at the storage account level granted to the Databricks Access Connector
ADLS Gen2 storage account set to have a private endpoint.
Everything (workspaces, ADLS Gen2, etc) is in the same Azure region

Now in the Databricks Workspace that has been assigned to this metastore, we have the following background:

Can create managed catalogs
Can create schemas in catalogs
Can create volumes in catalogs
Can upload files to volume in catalog and verify on ADLS Gen2 the files are stored there
Have all permissions set (perhaps over-set) on my personal access to the catalogs and metastore to allow connection (i.e., workspace all privileges, metastore all privileges, catalog all privileges)
I am a Databricks account admin and the metastore admin

With all this, I cannot create tables in any catalog.

I get an error:

So as we have enabled a private endpoint on the ADLS Gen2 storage account (metastore), one clear place to look is that. But somehow I can add files to volumes there despite the private endpoint on the metastore.

So it makes me think it is something to do with the Databricks cluster I am using when I run the sql commands from a notebook. I have tried with both a single-user and shared access mode for the cluster, but same result.

Does this background and problem seem familiar to anyone else? Thanks!

Re: Azure Databricks with metastore, cannot create managed table

karthik_p — Fri, 20 Oct 2023 17:58:33 GMT

@m997al If i am not wrong ADLS Gen 2 Private endpoint config is not needed, if you want to have more security group/user level security can be applied on ADLS Gen2 folders. Data Governance will be taken care by UC. can you please revisit your design and test without private endpoint on ADLS Gen2

Re: Azure Databricks with metastore, cannot create managed table

m997al — Fri, 20 Oct 2023 18:06:57 GMT

We have found that without a private endpoint on the ADLS Gen2, unity catalog (for managed tables anyway) works just fine. I was able to create managed tables.

We are focusing now on this: Create an Azure Databricks workspace in your own Virtual Network quickstart | Microsoft Learn

The Databricks workspace was created with no settings for networking other than public. I believe the fix isn't that hard, based on the document linked above.

Re: Azure Databricks with metastore, cannot create managed table

Kris2 — Tue, 23 Jan 2024 14:41:14 GMT

@m997al Were you able to resolve this issue? I have same issue as you described. I am able to upload into Volumes but not able to create managed tables in ADLS metastore storage account..