cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Azure Databricks with metastore, cannot create managed table

m997al
Contributor III

We have set up Azure Databricks with Unity Catalog (metastore) in an ADLS Gen2 storage account.

  • Used Managed Identity (Databricks Access Connector) for connection from workspace(s) to ADLS Gen2
  • ADLS Gen2 storage account has Storage Blob Data Owner and Storage Blob Data Contributor at the storage account level granted to the Databricks Access Connector
  • ADLS Gen2 storage account set to have a private endpoint.
  • Everything (workspaces, ADLS Gen2, etc) is in the same Azure region

Now in the Databricks Workspace that has been assigned to this metastore, we have the following background:

  • Can create managed catalogs
  • Can create schemas in catalogs
  • Can create volumes in catalogs
  • Can upload files to volume in catalog and verify on ADLS Gen2 the files are stored there
  • Have all permissions set (perhaps over-set) on my personal access to the catalogs and metastore to allow connection (i.e., workspace all privileges, metastore all privileges, catalog all privileges)
  • I am a Databricks account admin and the metastore admin

With all this, I cannot create tables in any catalog.

I get an error:

m997al_0-1697803440437.png

So as we have enabled a private endpoint on the ADLS Gen2 storage account (metastore), one clear place to look is that.  But somehow I can add files to volumes there despite the private endpoint on the metastore.

So it makes me think it is something to do with the Databricks cluster I am using when I run the sql commands from a notebook.  I have tried with both a single-user and shared access mode for the cluster, but same result.

Does this background and problem seem familiar to anyone else?  Thanks!

 

4 REPLIES 4

Kaniz_Fatma
Community Manager
Community Manager

Hi @m997al, It seems there is a problem with the connection to your Unity Metastore. Please verify that the Private Endpoint is set up correctly. Check if your Private Endpoint is associated with the correct virtual network and subnet and if the necessary DNS settings have been configured. Also, verify if you have enabled Private Link for the Blob service in your Storage account. Also, the error message you received suggests that the problem might be related to the role assignments of your Databricks service principal.

karthik_p
Esteemed Contributor

@m997al If i am not wrong ADLS Gen 2 Private endpoint config is not needed, if you want to have more security group/user level security can be applied on ADLS Gen2 folders. Data Governance will be taken care by UC. can you please revisit your design and test without private endpoint on ADLS Gen2

m997al
Contributor III

We have found that without a private endpoint on the ADLS Gen2, unity catalog (for managed tables anyway) works just fine.  I was able to create managed tables.

We are focusing now on this:  Create an Azure Databricks workspace in your own Virtual Network quickstart | Microsoft Learn

The Databricks workspace was created with no settings for networking other than public.  I believe the fix isn't that hard, based on the document linked above.

Kris2
New Contributor II

@m997al Were you able to resolve this issue? I have same issue as you described. I am able to upload into Volumes but not able to create managed tables in ADLS metastore storage account..

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group