topic Re: Create a catalog with external location from different Metastore in Data Governance

Create a catalog with external location from different Metastore

erigaud — Mon, 22 Apr 2024 14:51:45 GMT

Hello community !

Here's my problem : I had a workspace with Unity Catalog enabled, and a catalog stored on an external location (Azure storage account). All my schemas and tables were storage on this storage.

However we recently deleted the metastore, and created a new one.

I am wondering if it is possible to re-create a catalog by specifying the same external location, to access all my schemas and tables again ?

I tried doing so but I no longer see my schemas and tables on Databricks, so I am wondering if now that the metastore changed, I will not be able to see my tables ?

Thank you for the help !

Re: Create a catalog with external location from different Metastore

ScottSmithDB — Tue, 23 Apr 2024 05:33:05 GMT

If both the Catalog and the Schema and tables are stored externally, and you have deleted the original catalog, you should be able to recreate the schema in a new catalog by pointing it to the previous data files and running the SYNC command. Here are the steps .
1. Define an External Location in the new catalog that points to the same location where your previous data files are stored. Sounds like you did this.
2. Use the SYNC command to create a new Unity Catalog table based on your legacy table location. This will only create metadata and not change the underlying data. The syntax is as follows:

SYNC hive_metastore.schema.table uc_catalog.schema.table This command will create a new table in your new catalog that points to the same data files as your old table. It sounds like you may not have done this.

Note that the SYNC command assumes that the data files are still in the same location and have not been moved or deleted. If the data files have been moved, you will need to adjust the External Location accordingly.

Hope this is helpful.

Re: Create a catalog with external location from different Metastore

erigaud — Thu, 25 Apr 2024 09:51:18 GMT

I have an issue when trying to create a new catalog based on the previous catalog location :

Any idea ? Thanks Scott !

Re: Create a catalog with external location from different Metastore

dkushari — Sat, 11 May 2024 19:42:55 GMT

Hi,

Looks like you are creating the catalog at the container level. Are there any other UC objects created pointing to that? Is the external location created at the container level? Can you please have a look at this and see if there is overlapping? Thanks!