topic Re: Problems with Unity Catalog in Data Governance

Problems with Unity Catalog

LukaszJ — Tue, 15 Nov 2022 12:21:09 GMT

Hello.

I am using Azure Databricks since a few months and I want to add a Unity Catalog.

Unfortunately, I have some problems:

1. I am not able to create a Unity Catalog Cluster with settings:

- Policy: Unrestricted

- Multi Node with Single user access mode

- Runtime version is 11.3 LTS with Spark 3.3.0 and Scala 2.12

- Worker type is Standard_F4s (1-2 workers)

- spark.databricks.dataLineage.enabled true

- spark.databricks.delta.preview.enabled true

Do you know what did I do wrong?

2. I cannot create a new metastore (probably because I do not have special cluster).

I read the instruction and I do not have "Create Metastore" button (I am using Data Science & Engineering mode)

Could you help me with my problems?

Best regards,

Łukasz

Re: Problems with Unity Catalog

Pat — Tue, 15 Nov 2022 15:04:56 GMT

Hi @Łukasz Jaremek ,

not sure where to start, the Unity Catalog cluster is the last thing you need to do.

To add/enable Unity Catalog in your workspace you need to create metastore.

Metastore is account level Object I would say. There are some requirements to create that metastore, i.e. storage and policies in azure. Then you need to create the metastore in the Databricks account and attach that metastore(storage adbfs://).

After you have metastore, you attach that metastore to the workspace. It makes your workspace Unity Catalog enabled - make sure do it in dev workspace, as once attached workspace to UC, stays UC attached.

Then the next steps are creating cluster (so far we didn't have to use the cluster, everything was azure and databricks api).

This explains well how to do it: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/get-started

2. I cannot create a new metastore (probably because I do not have special cluster).

I read the instruction and I do not have "Create Metastore" button (I am using Data Science & Engineering mode)

You shoud create metastore in the Databricks account (it's called E2 I believe), not in the workspace.

Re: Problems with Unity Catalog

LukaszJ — Thu, 17 Nov 2022 08:57:58 GMT

Hello @Pat Sienkiewicz , thanks you for your response!

Okay, I have a new Azure Storage Account, and what next?

I have mounted it on the databricks with dbutils.mount, but what means "attach"?

I do not have button "Create Metastore" in Data tab.

Best regards,

Łukasz

Re: Problems with Unity Catalog

Pat — Thu, 17 Nov 2022 09:15:27 GMT

Hi @Łukasz Jaremek ,

you don't have to mount this storage account.

When you create new storage account (to use as metastore root account) you need to attach/set it as metastore storage on the account level:

go to Manage Account:

then switch to metastore tab,

on the right side click create metastore (aka unity catalog)

here you need to specify which storage account should be used:

in the next tab it will ask you to attach workspace to unity catalog.

thanks,.

Pat.

Re: Problems with Unity Catalog

LukaszJ — Thu, 17 Nov 2022 09:51:34 GMT

Hmm it is interesting.

I have a different app design than you.

I am a databricks admin and I have premium plan.

In my case it looks like this:

Do you know why?

Re: Problems with Unity Catalog

Pat — Thu, 17 Nov 2022 09:55:56 GMT

I see you are on workspace level now, do you have possibility in top right corner to switch to the account level ? You need admin account permission:

source; https://www.databricks.com/blog/2022/08/26/databricks-workspace-administration-best-practices-for-account-workspace-and-metastore-admins.html

Re: Problems with Unity Catalog

LukaszJ — Thu, 17 Nov 2022 19:39:43 GMT

I have a Admin on Databricks.

At Azure level I have Contributor role. There are one "better" (Owner) but it is only for assign roles in Azure RBAC.

I have created a new Azure Databricks (again with premium plan and admin account) and still I have simple "User Settings" interface.

> "do you have possibility in top right corner to switch to the account level ?" - Do you mean change my databricks user for admin or something else?

Re: Problems with Unity Catalog

Pat — Thu, 17 Nov 2022 19:44:47 GMT

I mean that you need to click on „manage account”, as I have showed before. If you don’t have account level permission then you won’t see this option I think.

you are currently in wrong „place”. To create unity catalog / metastore you need to sign i to databricks account, not the workspace.

thanks,

Pat.

Re: Problems with Unity Catalog

Pat — Thu, 17 Nov 2022 20:10:18 GMT

@Łukasz Jaremek

more details ;

https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/

Re: Problems with Unity Catalog

LukaszJ — Thu, 17 Nov 2022 23:02:06 GMT

Okay I do not know what is wrong with my Databricks.

I clicked on my email, then "Manage Account",then I selected my Databricks instantion (the same as the photo from your link) and after that I am in the same Databricks workspace. There are no any difference 😞

User settings and Data page are the same.

Maybe there are some extra steps like creating or enabling something?

Based on the documentation, I have Azure Contributor, a premium plan and an admin account.

Re: Problems with Unity Catalog

LukaszJ — Mon, 21 Nov 2022 07:51:07 GMT

Could someone from the administration please look at my problem?

I don't know who to tag so I apologize for the inaccurate tag.

@Kaniz Fatma

Re: Problems with Unity Catalog

shudong — Mon, 21 Nov 2022 10:27:02 GMT

I also cannot find the "Create Metastore" button from my console in Azure Databricks, although I am the one create my databricks stack... Could anyone help on this?

Re: Problems with Unity Catalog

Addi1 — Sat, 26 Nov 2022 17:46:12 GMT

I'm facing the same issues as other customers listed above. I do not have the "Create Metastore" button available.