topic Re: Move a delta table from a non UC metastore to a UC metastore preserving history in Data Engineering

Move a delta table from a non UC metastore to a UC metastore preserving history

Jotav93 — Fri, 12 Apr 2024 08:02:17 GMT

Hi,

I am using Azure databricks and we recently enabled UC in our workspace. We have some tables in our non UC metastore that we want to move to a UC enabled metastore. Is there any way we can move these tables without loosing the delta table history? (i.e. sql command "describe history old_mestastore.db_name.table_name" should give exactly the same results than "describe history new_metstore.db_name.table_name" where old metastore is a non UC metastore while new_metastore is a UC enabled metastore).

Any info would be appreciated.

Thanks

Joaquim

Re: Move a delta table from a non UC metastore to a UC metastore preserving history

ThomazRossito — Sun, 14 Apr 2024 12:28:25 GMT

Hello,
It is possible to have the expected result with dbutils.fs.cp("Origin location", "Destination location", True) and then create the table with the LOCATION of the Destination location

Hope this helps

Re: Move a delta table from a non UC metastore to a UC metastore preserving history

Jotav93 — Wed, 17 Apr 2024 07:24:02 GMT

Hi Thomasz,

thanks for your answer. It partially works as it preserves the versions but it doesn't preserve version timestamps (I didn't specify that explicitly but I need to preserve version timestamps).

- Version before the copy

- Copy operation

Table creation and version after the copy

Am I doing anything wrong? Would you suggest either move or .zip and unzip to preserve file modified times? Should I create the table first and then copy the data inside? Or what would you suggest?

Thanks

Joaquim