Problems with Unity Catalog

[LukaszJ]

Hello.

I am using Azure Databricks since a few months and I want to add a Unity Catalog.

Unfortunately, I have some problems:

1. I am not able to create a Unity Catalog Cluster with settings:

- Policy: Unrestricted

- Multi Node with Single user access mode

- Runtime version is 11.3 LTS with Spark 3.3.0 and Scala 2.12

- Worker type is Standard_F4s (1-2 workers)

- spark.databricks.dataLineage.enabled true

- spark.databricks.delta.preview.enabled true

Do you know what did I do wrong?

2. I cannot create a new metastore (probably because I do not have special cluster).

I read the instruction and I do not have "Create Metastore" button (I am using Data Science & Engineering mode)

 

Could you help me with my problems?

Best regards,

Łukasz

[Pat]

Hi @Łukasz Jaremek​ ,

not sure where to start, the Unity Catalog cluster is the last thing you need to do.

To add/enable Unity Catalog in your workspace you need to create metastore.

Metastore is account level Object I would say. There are some requirements to create that metastore, i.e. storage and policies in azure. Then you need to create the metastore in the Databricks account and attach that metastore(storage adbfs://).

After you have metastore, you attach that metastore to the workspace. It makes your workspace Unity Catalog enabled - make sure do it in dev workspace, as once attached workspace to UC, stays UC attached.

Then the next steps are creating cluster (so far we didn't have to use the cluster, everything was azure and databricks api).

This explains well how to do it: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/get-started

--

2. I cannot create a new metastore (probably because I do not have special cluster).

I read the instruction and I do not have "Create Metastore" button (I am using Data Science & Engineering mode)

You shoud create metastore in the Databricks account (it's called E2 I believe), not in the workspace.

[LukaszJ]

Hello @Pat Sienkiewicz​ , thanks you for your response!

Okay, I have a new Azure Storage Account, and what next?

I have mounted it on the databricks with dbutils.mount, but what means "attach"?

I do not have button "Create Metastore" in Data tab.

Best regards,

Łukasz

[Pat]

Hi @Łukasz Jaremek​ ,

you don't have to mount this storage account.

When you create new storage account (to use as metastore root account) you need to attach/set it as metastore storage on the account level:

go to Manage Account:

 

then switch to metastore tab,

on the right side click create metastore (aka unity catalog)

 

here you need to specify which storage account should be used:

in the next tab it will ask you to attach workspace to unity catalog.

thanks,.

Pat.

[LukaszJ]

Hmm it is interesting.

I have a different app design than you.

I am a databricks admin and I have premium plan.

In my case it looks like this:

Do you know why?

[Pat]

I see you are on workspace level now, do you have possibility in top right corner to switch to the account level ? You need admin account permission:

source; https://www.databricks.com/blog/2022/08/26/databricks-workspace-administration-best-practices-for-ac...

[LukaszJ]

I have a Admin on Databricks.

At Azure level I have Contributor role. There are one "better" (Owner) but it is only for assign roles in Azure RBAC.

I have created a new Azure Databricks (again with premium plan and admin account) and still I have simple "User Settings" interface.

> "do you have possibility in top right corner to switch to the account level ?" - Do you mean change my databricks user for admin or something else?

[Pat]

I mean that you need to click on „manage account”, as I have showed before. If you don’t have account level permission then you won’t see this option I think.

 

you are currently in wrong „place”. To create unity catalog / metastore you need to sign i to databricks account, not the workspace.

thanks,

Pat.

[Pat]

@Łukasz Jaremek​ 

more details ;

https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/

[LukaszJ]

Okay I do not know what is wrong with my Databricks.

I clicked on my email, then "Manage Account",then I selected my Databricks instantion (the same as the photo from your link) and after that I am in the same Databricks workspace. There are no any difference 😞

User settings and Data page are the same.

Maybe there are some extra steps like creating or enabling something?

Based on the documentation, I have Azure Contributor, a premium plan and an admin account.

[LukaszJ]

Could someone from the administration please look at my problem?

I don't know who to tag so I apologize for the inaccurate tag.

@Kaniz Fatma​ 

[shudong]

I also cannot find the "Create Metastore" button from my console in Azure Databricks, although I am the one create my databricks stack... Could anyone help on this?

[Addi1]

I'm facing the same issues as other customers listed above. I do not have the "Create Metastore" button available.