Delta Table Log History not Updating

Dave_Nithio — Thu, 19 Sep 2024 15:36:14 GMT

I am running into an issue related to my Delta Log and an old version. I currently have default delta settings for

delta.checkpointInterval (10 commits as this table was created prior to DBR 11.1), delta.deletedFileRetentionDuration (7 days), and delta.logRetentionDuration (30 days). I have an older table (created 12/8/2022) with the following delta history:

I ran a `REPLACE TABLE AS SELECT` function to update my partitioning strategy which removed all logs prior to version 10 where my previous checkpoint was. This created a checkpoint at Version 16 and left my delta history as:

I then attempted to restore to Version 16 to see if I could still time travel to that point. It was successful and deleted all logs prior to the new checkpoint at Version 16:

Again this was as expected. The issue is, I then ran REPLACE TABLE AS SELECT again assuming that I would then delete any logs prior to my new checkpoint, meaning version 16 would no longer exist in the logs as it is well beyond my log retention. This was not the case though. I attempted to run multiple commits against the table which resulted in a few more checkpoints, but version 16 has not been removed from my delta history.

I am wondering if is it related to deletedFileRetentionDuration delta setting. Maybe version 16 is referenced in newer files (due to RESTORE) so it will continue to maintain that version in the delta log? If so, should I expect it to be deleted 7 days after today?

Re: Delta Table Log History not Updating

jennie258fitz — Fri, 20 Sep 2024 05:47:15 GMT

@Dave_Nithio wrote:
I am running into an issue related to my Delta Log and an old version. I currently have default delta settings for
delta.checkpointInterval (10 commits as this table was created prior to DBR 11.1), delta.deletedFileRetentionDuration (7 days), and delta.logRetentionDuration (30 days). I have an older table (created 12/8/2022) with the following delta history:

I ran a `REPLACE TABLE AS SELECT` function to update my partitioning strategy which removed all logs prior to version 10 where my previous checkpoint was. This created a checkpoint at Version 16 and left my delta history as:

I then attempted to restore to Version 16 to see if I could still time travel to that point. It was successful and deleted all logs prior to the new checkpoint at Version 16:

Again this was as expected. The issue is, I then ran REPLACE TABLE AS SELECT again assuming that I would then delete any logs prior to my new checkpoint, meaning version 16 would no longer exist in the logs as it is well beyond my log retention. This was not the case though. I attempted to run multiple commits against the table which resulted in a few more checkpoints, but version 16 has not been removed from my delta history.
I am wondering if is it related to deletedFileRetentionDuration delta setting. Maybe version 16 is referenced in newer files (due to RESTORE) so it will continue to maintain that version in the delta log? If so, should I expect it to be deleted 7 days after today? AarpMembership

Looking for the same issue. Bumped into your thread. Thanks for creating it. Looking forward for solution.

topic Delta Table Log History not Updating in Data Engineering

Delta Table Log History not Updating

Re: Delta Table Log History not Updating