cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Move a delta table from a non UC metastore to a UC metastore preserving history

Jotav93
New Contributor II

Hi,

 

I am using Azure databricks and we recently enabled UC in our workspace. We have some tables in our non UC metastore that we want to move to a UC enabled metastore. Is there any way we can move these tables without loosing the delta table history? (i.e. sql command "describe history old_mestastore.db_name.table_name" should give exactly the same results than "describe history new_metstore.db_name.table_name" where old metastore is a non UC metastore while new_metastore is a UC enabled metastore).

 

Any info would be appreciated.

 

Thanks

Joaquim

2 REPLIES 2

ThomazRossito
Contributor

Hello,
It is possible to have the expected result with dbutils.fs.cp("Origin location", "Destination location", True) and then create the table with the LOCATION of the Destination location

Hope this helps

Att.
Thomaz Antonio Rossito Neto
Master Data Specialist - Data Architect | Data Engineer @ CI&T

Hi Thomasz,

thanks for your answer. It partially works as it preserves the versions but it doesn't preserve version timestamps (I didn't specify that explicitly but I need to preserve version timestamps).

- Version before the copy

Jotav93_0-1713338360117.png

- Copy operation

Jotav93_1-1713338390241.png

Table creation and version after the copy

Jotav93_3-1713338535304.png

Am I doing anything wrong? Would you suggest either move or .zip and unzip to preserve file modified times? Should I create the table first and then copy the data inside? Or what would you suggest?

Thanks

 

Joaquim

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group