Azure Databricks with metastore, cannot create managed table

m997al
Contributor III

‎10-20-2023 05:08 AM

We have set up Azure Databricks with Unity Catalog (metastore) in an ADLS Gen2 storage account.

  • Used Managed Identity (Databricks Access Connector) for connection from workspace(s) to ADLS Gen2
  • ADLS Gen2 storage account has Storage Blob Data Owner and Storage Blob Data Contributor at the storage account level granted to the Databricks Access Connector
  • ADLS Gen2 storage account set to have a private endpoint.
  • Everything (workspaces, ADLS Gen2, etc) is in the same Azure region

Now in the Databricks Workspace that has been assigned to this metastore, we have the following background:

  • Can create managed catalogs
  • Can create schemas in catalogs
  • Can create volumes in catalogs
  • Can upload files to volume in catalog and verify on ADLS Gen2 the files are stored there
  • Have all permissions set (perhaps over-set) on my personal access to the catalogs and metastore to allow connection (i.e., workspace all privileges, metastore all privileges, catalog all privileges)
  • I am a Databricks account admin and the metastore admin

With all this, I cannot create tables in any catalog.

I get an error:

m997al_0-1697803440437.png

So as we have enabled a private endpoint on the ADLS Gen2 storage account (metastore), one clear place to look is that.  But somehow I can add files to volumes there despite the private endpoint on the metastore.

So it makes me think it is something to do with the Databricks cluster I am using when I run the sql commands from a notebook.  I have tried with both a single-user and shared access mode for the cluster, but same result.

Does this background and problem seem familiar to anyone else?  Thanks!

 

0 Kudos

karthik_p
Databricks Partner

‎10-20-2023 10:58 AM

@m997al If i am not wrong ADLS Gen 2 Private endpoint config is not needed, if you want to have more security group/user level security can be applied on ADLS Gen2 folders. Data Governance will be taken care by UC. can you please revisit your design and test without private endpoint on ADLS Gen2

karthik.p
0 Kudos

m997al
Contributor III

‎10-20-2023 11:06 AM

We have found that without a private endpoint on the ADLS Gen2, unity catalog (for managed tables anyway) works just fine.  I was able to create managed tables.

We are focusing now on this:  Create an Azure Databricks workspace in your own Virtual Network quickstart | Microsoft Learn

The Databricks workspace was created with no settings for networking other than public.  I believe the fix isn't that hard, based on the document linked above.

0 Kudos

Kris2
Databricks Partner

‎01-23-2024 06:41 AM

@m997al Were you able to resolve this issue? I have same issue as you described. I am able to upload into Volumes but not able to create managed tables in ADLS metastore storage account..

0 Kudos