topic Re: Default catalog created wrong on my workspace in Data Engineering

Default catalog created wrong on my workspace

amarnadh-gadde — Thu, 30 Jan 2025 18:08:58 GMT

We have provisioned a new databricks account and workspace on premium plan. When built out workspace using terraform, we expected to see a default catalog matching workspace name as per this documentation. However I dont see it. All I see are the 3 catalogs below and the default catalog for the workspace is pointed to the legacy hive_metastore. Can you help how to fix this.

https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/unity-catalog-default

Re: Default catalog created wrong on my workspace

NandiniN — Fri, 31 Jan 2025 16:11:39 GMT

I am also able to see just these 3, I think it is meant to be "My organization"

Re: Default catalog created wrong on my workspace

saurabh18cs — Fri, 31 Jan 2025 16:30:03 GMT

The legacy Hive metastore is maintained for backward compatibility. When you start using Unity Catalog compatible compute i.e. when the workspace has an assigned Unity Catalog metastore, and the cluster is in access mode 'Shared' or 'Single User', or in SQL warehouses then you to reference a fully qualified 3 level name. Catalog.Schema.table.

Run the following SQL queries to check if Unity Catalog is enabled and to list the catalogs and schemas.

-- List catalogs

SHOW CATALOGS;

-- List schemas in a specific catalog

SHOW SCHEMAS IN <catalog_name>;

Re: Default catalog created wrong on my workspace

amarnadh-gadde — Fri, 31 Jan 2025 18:47:23 GMT

@saurabh18cs Ours is a new account and as I represented in the screenshots above, for new accounts databricks documents suggests we will have a new catalog created matching workspace name. However that never happened in our account and the default catalog for our workspace points to the hive_metastore catalog. I do not want to use hive_metastore catalog. I am working towards creating my own catalog for the workspace and attach it to the UN=nity metastore, but I expected a new catalog be already there as per their documentation.

Re: Default catalog created wrong on my workspace

amarnadh-gadde — Fri, 31 Jan 2025 18:58:06 GMT

@NandiniN My organization is a grouping not a catalog. When I run SHOW CATALOGS I get below, which tells me it could be probably "system", but that is not setup as default right now. The terraform documentation I link I provided above says we should have a catalog created by default matching workspace name.

Re: Default catalog created wrong on my workspace

saurabh18cs — Mon, 03 Feb 2025 10:32:45 GMT

@amarnadh-gadde let me explain you again to make it more clear.

1) My orgaisation will act as your unity catalog catalog.

2) Ignore what is showing default there and ignore hive_metastore

3) start using unity catalog via uc enabled compute and 3 level namespace (catalog.schema.table)

Just try using it , I believe everything is correct at your side.

Re: Default catalog created wrong on my workspace

loic — Wed, 11 Jun 2025 09:17:28 GMT

Hello,

Meanwhile I try to get help on Databricks default catalog behavior, I found this topic.
If I can give my advice here, one reason I see for the behavior of @amarnadh-gadde is that you deployed your new workspace in a region where there is already a metastore.
In this case, the behavior that has been referenced previously:
"A workspace catalog ... is named after your workspace"
Won't be effective.
Indeed, if you are well in the case where the metastore already exist in the region, you will have to change a setting at metastore level in order to enable the automatic assignment of the unity catalog. See:
https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/manage-metastore#auto-assign

At least, on my side, I am in this case. Each time we create a new workspace, there is no catalog with the name of the workspace that is created.

See this anwser for a nice graph explaining what I said previously:
https://community.databricks.com/t5/technical-blog/unity-catalog-with-automatic-enablement-part-1-azure/ba-p/117220#toc-hId-807739415

Regards,
Loïc