Azure Databricks with metastore, cannot create managed table

[m997al]

We have set up Azure Databricks with Unity Catalog (metastore) in an ADLS Gen2 storage account.

• Used Managed Identity (Databricks Access Connector) for connection from workspace(s) to ADLS Gen2
• ADLS Gen2 storage account has Storage Blob Data Owner and Storage Blob Data Contributor at the storage account level granted to the Databricks Access Connector
• ADLS Gen2 storage account set to have a private endpoint.
• Everything (workspaces, ADLS Gen2, etc) is in the same Azure region

Now in the Databricks Workspace that has been assigned to this metastore, we have the following background:

• Can create managed catalogs
• Can create schemas in catalogs
• Can create volumes in catalogs
• Can upload files to volume in catalog and verify on ADLS Gen2 the files are stored there
• Have all permissions set (perhaps over-set) on my personal access to the catalogs and metastore to allow connection (i.e., workspace all privileges, metastore all privileges, catalog all privileges)
• I am a Databricks account admin and the metastore admin

With all this, I cannot create tables in any catalog.

I get an error:

So as we have enabled a private endpoint on the ADLS Gen2 storage account (metastore), one clear place to look is that.  But somehow I can add files to volumes there despite the private endpoint on the metastore.

So it makes me think it is something to do with the Databricks cluster I am using when I run the sql commands from a notebook.  I have tried with both a single-user and shared access mode for the cluster, but same result.

Does this background and problem seem familiar to anyone else?  Thanks!

[Kaniz]

Hi @m997al, It seems there is a problem with the connection to your Unity Metastore. Please verify that the Private Endpoint is set up correctly. Check if your Private Endpoint is associated with the correct virtual network and subnet and if the necessary DNS settings have been configured. Also, verify if you have enabled Private Link for the Blob service in your Storage account. Also, the error message you received suggests that the problem might be related to the role assignments of your Databricks service principal.

[karthik_p]

@m997al If i am not wrong ADLS Gen 2 Private endpoint config is not needed, if you want to have more security group/user level security can be applied on ADLS Gen2 folders. Data Governance will be taken care by UC. can you please revisit your design and test without private endpoint on ADLS Gen2

[m997al]

We have found that without a private endpoint on the ADLS Gen2, unity catalog (for managed tables anyway) works just fine.  I was able to create managed tables.

We are focusing now on this:  Create an Azure Databricks workspace in your own Virtual Network quickstart | Microsoft Learn

The Databricks workspace was created with no settings for networking other than public.  I believe the fix isn't that hard, based on the document linked above.

[Kris2]

@m997al Were you able to resolve this issue? I have same issue as you described. I am able to upload into Volumes but not able to create managed tables in ADLS metastore storage account..