Setting up Unity Catalog in Azure

[leelee3000]

Trying to create a metastore that will be connected to an external storage (ADLS) but we don't have the option to create a new metastore in 'Catalog' tab in the UI. Based on some research, we see that we'll have to go into "Manage Account" and then create a metastore from there. However, once we click "Manage Account" from the drop down menu under our username on the top right, we are given the choice of selecting the workspace. When we pick the correct workspace, we are directed back to the home page of the workspace instead of the account page. We are both owners of this workspace so I'm not sure what other permissions we need to grant ourselves to solve this. The workspace is also a premium workspace.

[Kaniz_Fatma · Accepted solution]

Hi @leelee3000, Creating a metastore in Unity Catalog and linking it to workspaces involves a few steps.

Let’s break it down:

Create a Storage Location for Metastore-Level Storage:

• In your Azure account, set up a storage location specifically for managing tables and volumes at the metastore level.

Create an Azure Managed Identity:

• Next, create an Azure-managed identity that will have access to the storage location you just set up.

Create the Metastore:

• In Azure Databricks, create the metastore and attach it to the storage location.
• Assign workspaces to this metastore.

Here are the detailed steps:

Step 1: Create a Storage Container for Metastore-Level Managed Storage:

• This container will hold metadata about securable objects (such as tables, volumes, external locations, and shares) and the permissions governing access to them.
• Remember that each metastore exposes a three-level namespace (catalog.schema.table) for organizing data.

Step 2: Create an Azure Managed Identity:

• This identity will grant access to the storage location.
• Ensure that you meet the following requirements:
  • You must be an Azure Databricks account admin.
  • The first Azure Databricks account admin must be a Microsoft Enterprise ID (formerly Azure Active Directory Global Administrator) when they first log in to the Azure Databricks account console.
  • Upon first login, that user becomes an Azure Databricks account admin and no longer needs the Microsoft Enterprise ID Global Administrator role to access the Azure Databricks account.
  • The first account admin can assign users in the Microsoft Enterprise ID tenant as additional account admins (who can themselves assign more account admins). Additional account admins do not require specific roles in Microsoft Enterprise ID.
  • The workspaces you attach to the metastore must be on the Azure Databricks Premium plan.

Step 3: Create the Metastore and Attach Workspaces:

• In Azure Databricks, create the metastore, attach the storage location, and assign workspaces to the metastore.

Remember that if your workspace was automatically enabled for Unity Catalog when it was created, a metastore is automatically provisioned for your workspace. You only need to perform these tasks if you were prompted to do so during workspace creation.

Feel free to follow these steps, and you’ll have your metastore set up for seamless data management! 🌟

[leelee3000]

@Kaniz_Fatma Thank you so much for the quick response, you rock!

[SSundaram]

Looks like you do not have account admin access.  

[bsadler]

I have been wrestling with this question for days now.  I seem to be the only one with this question so I am sure I am doing something wrong.  I am trying to create a UC metastore but there is not an option in "Catalog" to create a metastore.  This seems to be leelee3000's quesiton.  I have read the Databricks documentation thoroughly and it reads a lot like kaniz_fatma's response.  This response is almost exactly what the documentation says.  In step it says, " In Azure Databricks, create the metastore, attach the storage location, and assign workspaces to the metastore."   The question is "how do I create a metastore when it is not available in the catalog" and the answer is "In Azure Databricks, create the metastore."  At this point, I would pay quite a bit to figure out how to enable my workspace with the UC, be able to create a compute that is UC enabled (I have done this Spark Init in the advanced section but the blue "Unity Catalog" icon does not appear in the Summary), and then finally create a UC Catalog so that I can create a volume to that I can implement and deploy a RAG model.   Again, very likely my user error, can someone help?