Azure Databricks, migrating delta table data with CDF on.

grazie
Contributor

We are on Azure Databricks over ADLS Gen2 and have a set of tables and workflows that process data from and between those tables, using change data feeds. (We are not yet using Unity Catalog, and also not Hive metastore, just accessing delta tables from storage). 

Now we are required by IT to move data to a different Azure storage account, and I am faced with a conundrum. Delta documentation states that it is important to maintain timestamps if you want to preserve CDF. https://docs.delta.io/latest/delta-faq.html#can-i-copy-my-delta-lake-table-to-another-location

On the other hand, Azure support rep states that the only way they can think of is to manually use Storage Explorer something like this:


On Storage Explorer

Export the table as CSV, with option to export the timestamp

Import the CSV into the the new Storage Account changing the name of the Timestamp to something else, like Timestamp2.

Which will add a new field in the new table and also consumes more space because of it.

I need to try that recipe to see if it actually makes sense, but in reality we have many tables and such a manual process would be inefficient and error-prone. 

Maybe someone here knows a better way?