topic Re: Copy CDF enabled delta table from one location to another by retaining history in Data Engineering

Copy CDF enabled delta table from one location to another by retaining history

RKNutalapati — Tue, 15 Feb 2022 08:35:06 GMT

I am currently doing some use case testing. I have to CLONE delta table with CDF enabled to a different S3 bucket. Deep clone doesn't meet the requirement. So I tried to copy the files using dbutils.fs.cp, it is copying all the versions but the timestamp is getting changed.

Is there any work around to retain the timestamp as well while copying / migrating delta table.

Appreciate your valuable suggestions

Re: Copy CDF enabled delta table from one location to another by retaining history

RKNutalapati — Wed, 23 Feb 2022 10:20:41 GMT

Hi @Kaniz Fatma , Thanks for your response, I am not using Azure. I am working on DataBricks AWS.

Re: Copy CDF enabled delta table from one location to another by retaining history

jose_gonzalez — Mon, 11 Apr 2022 18:59:50 GMT

Hi @Rama Krishna N ,

Just a friendly follow-up. Do you still need help? If you do, could you provide more details on your issue?

What type of cloning did you do? Shadow or deep clone?

A cloned table has an independent history from its source table. Time travel queries on a cloned table will not work with the same inputs as they work on its source table. You can find more information here https://docs.databricks.com/delta/delta-utility.html#clone-delta-table

Re: Copy CDF enabled delta table from one location to another by retaining history

RKNutalapati — Tue, 12 Apr 2022 03:44:18 GMT

Hi @Jose Gonzalez : Thanks for the follow-up. I am using Deep clone and was expecting the utility maintains the same historical timestamps. But that is not happening. Is there any good reason for independent history?

Re: Copy CDF enabled delta table from one location to another by retaining history

jose_gonzalez — Mon, 25 Apr 2022 19:16:17 GMT

@Rama Krishna N , According to the docs "A cloned table has an independent history from its source table.." are you trying to do use time travel? maybe the clone uses cases that we have in the docs might help to explain better https://docs.databricks.com/delta/delta-utility.html#clone-use-cases