ā02-15-2022 12:35 AM
I am currently doing some use case testing. I have to CLONE delta table with CDF enabled to a different S3 bucket. Deep clone doesn't meet the requirement. So I tried to copy the files using dbutils.fs.cp, it is copying all the versions but the timestamp is getting changed.
Is there any work around to retain the timestamp as well while copying / migrating delta table.
Appreciate your valuable suggestions
ā04-25-2022 12:16 PM
@Rama Krishna Nā , According to the docs "A cloned table has an independent history from its source table.." are you trying to do use time travel? maybe the clone uses cases that we have in the docs might help to explain better https://docs.databricks.com/delta/delta-utility.html#clone-use-cases
ā02-15-2022 01:46 AM
Hi @Rama Krishna Nā ! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
ā02-22-2022 01:55 PM
Hi @Rama Krishna Nā ,
Because you're on Azure, you can use Azure Data Factory's Data Copy Tool as it's described in the documentation - delta tables are just files in the container, and this tool can copy data, and potentially it would be cheaper than using Databricks cluster to do the copying.
ā02-23-2022 02:20 AM
Hi @Kaniz Fatmaā , Thanks for your response, I am not using Azure. I am working on DataBricks AWS.
ā02-23-2022 03:00 AM
Thank you for your response @Rama Krishna Nā . Got your point better now.
ā04-11-2022 11:59 AM
Hi @Rama Krishna Nā ,
Just a friendly follow-up. Do you still need help? If you do, could you provide more details on your issue?
What type of cloning did you do? Shadow or deep clone?
A cloned table has an independent history from its source table. Time travel queries on a cloned table will not work with the same inputs as they work on its source table. You can find more information here https://docs.databricks.com/delta/delta-utility.html#clone-delta-table
ā04-11-2022 08:44 PM
Hi @Jose Gonzalezā : Thanks for the follow-up. I am using Deep clone and was expecting the utility maintains the same historical timestamps. But that is not happening. Is there any good reason for independent history?
ā04-25-2022 12:16 PM
@Rama Krishna Nā , According to the docs "A cloned table has an independent history from its source table.." are you trying to do use time travel? maybe the clone uses cases that we have in the docs might help to explain better https://docs.databricks.com/delta/delta-utility.html#clone-use-cases
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.