topic Re: unity catalog external migration and downtime in Data Governance

unity catalog external migration and downtime

karthik_p — Thu, 15 Dec 2022 17:19:55 GMT

HI Team,

once after implementing unity catalog and start migrating external tables from legacy hive to unity catalog, i am seeing in articles that we need to make changes to our workloads to be in sync with 3 level name space

eg: if i have 50 notebooks which uses legacy hive, usually there is no 3-level name space and we can run our notebooks based on standard spark queries. but after migrating to unity catalog if i need to run those notebooks and i am not using default hive_metastore config in spark level, do i need to manually change to use catalog and followed by schema.tablename in spark/python codes in notebooks to work without any issues

Note: i have seen article where we can use both legacy and unity catalog until we feel confident on functionalities are in tack with our requirement and drop legacy tables. if we drop legacy, do we need to change all notebooks to use catalog and 3 level namespaces

do we have backup mechanism in case if we want to revert back to legacy. is that INFORMATION_SCHEMA backup or any other mechanisam

Re: unity catalog external migration and downtime

Aviral-Bhardwaj — Fri, 16 Dec 2022 15:53:15 GMT

you can check with the databricks team on this send email to help@databricks.com they will help you

Re: unity catalog external migration and downtime

karthik_p — Mon, 19 Dec 2022 21:31:57 GMT

@Aviral Bhardwaj this will be normal priority and it takes lot of time to get response.

team, we are looking for downtime possibilities and how to handle or change workflows or notebooks to use unity catalog 3 level namespace. do we have any utility to directly run, and it takes care to modify globally all over notebooks

eg: if i have 15 notebooks and i have multiple databases, do i need to change manually in notebooks to use unity catalog and databases in that with 3 level namespace and also if i have few jobs which uses external mount points, do i need to specify or modify to use unity catalog or once after migrating external tables to UC does it automatically takes care @pat any suggestions please

Re: unity catalog external migration and downtime

Aviral-Bhardwaj — Tue, 20 Dec 2022 01:44:23 GMT

yes right maybe @Kaniz Fatma can help you here