topic Re: Databricks SQL: catalog of each query in Data Engineering

Databricks SQL: catalog of each query

noimeta — Mon, 06 Feb 2023 10:03:42 GMT

Currently, we are migrating from hive metastore to UC.

We have several dashboards and a huge number of queries whose catalogs have been set to hive_metastore and using <db>.<table> access pattern.

I'm just wondering if there's a way to switch catalogs of all queries in one go? Switching catalog of each query one by one needs tons of work. Also, it would be best if we can switch back and forth, so that we can make sure that everything looks correct.

Re: Databricks SQL: catalog of each query

AdrianLobacz — Mon, 06 Feb 2023 14:08:20 GMT

Hi, check this link, its documentation about UC and hive metastore

https://docs.databricks.com/data-governance/unity-catalog/hive-metastore.html#language-sql

Re: Databricks SQL: catalog of each query

noimeta — Mon, 06 Feb 2023 15:39:20 GMT

Thank you. I did check the link.

I know that we can explicitly set the catalog name in a query. Anyway, my point is I don't want to manually edit each query one by one.

Re: Databricks SQL: catalog of each query

youssefmrini — Thu, 23 Feb 2023 09:47:57 GMT

There is a configuration that you can use to define the default catalog and you can have as many sql endpoints as catalogs. It will makes your life much easier.

Re: Databricks SQL: catalog of each query

noimeta — Wed, 01 Mar 2023 07:33:22 GMT

That sounds very promising.

Could you please pin point how to do that? Is there any document that I can follow?

Re: Databricks SQL: catalog of each query

NateAnth — Wed, 01 Mar 2023 13:41:40 GMT

Please see the link here.

This would be the setting to set on non-SQL Warehouse Databricks Clusters

spark.databricks.sql.initial.catalog.name

To change for the all of the SQL Warehouses in a workspace you would use:

databricks unity-catalog metastores assign --workspace-id 1234567890123456 \
                                       --metastore-id 12a345b6-7890-1cd2-3456-e789f0a12b34 \
                                       --default-catalog-name my_catalog

Re: Databricks SQL: catalog of each query

noimeta — Thu, 02 Mar 2023 09:35:59 GMT

Thank you so much. The unity-catalog CLI is working for me

Re: Databricks SQL: catalog of each query

h_h_ak — Thu, 31 Oct 2024 12:52:13 GMT

May you can also have a look here, if you need hot fix 🙂

https://github.com/databrickslabs/ucx