cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Delta Table Log History not Updating

Dave_Nithio
Contributor

I am running into an issue related to my Delta Log and an old version. I currently have default delta settings for 

delta.checkpointInterval (10 commits as this table was created prior to DBR 11.1), delta.deletedFileRetentionDuration (7 days), and delta.logRetentionDuration (30 days). I have an older table (created 12/8/2022) with the following delta history:
 Dave_Nithio_4-1726759906146.png
I ran a `REPLACE TABLE AS SELECT` function to update my partitioning strategy which removed all logs prior to version 10 where my previous checkpoint was. This created a checkpoint at Version 16 and left my delta history as:
 Dave_Nithio_2-1726759822867.png
I then attempted to restore to Version 16 to see if I could still time travel to that point. It was successful and deleted all logs prior to the new checkpoint at Version 16:
Dave_Nithio_1-1726759722776.png 
Again this was as expected. The issue is, I then ran REPLACE TABLE AS SELECT again assuming that I would then delete any logs prior to my new checkpoint, meaning version 16 would no longer exist in the logs as it is well beyond my log retention. This was not the case though. I attempted to run multiple commits against the table which resulted in a few more checkpoints, but version 16 has not been removed from my delta history.
Dave_Nithio_5-1726760080078.pngDave_Nithio_6-1726760100546.png
I am wondering if is it related to deletedFileRetentionDuration delta setting. Maybe version 16 is referenced in newer files (due to RESTORE) so it will continue to maintain that version in the delta log? If so, should I expect it to be deleted 7 days after today?
1 REPLY 1

jennie258fitz
New Contributor II

@Dave_Nithio wrote:

I am running into an issue related to my Delta Log and an old version. I currently have default delta settings for 

delta.checkpointInterval (10 commits as this table was created prior to DBR 11.1), delta.deletedFileRetentionDuration (7 days), and delta.logRetentionDuration (30 days). I have an older table (created 12/8/2022) with the following delta history:
 Dave_Nithio_4-1726759906146.png
I ran a `REPLACE TABLE AS SELECT` function to update my partitioning strategy which removed all logs prior to version 10 where my previous checkpoint was. This created a checkpoint at Version 16 and left my delta history as:
 Dave_Nithio_2-1726759822867.png
I then attempted to restore to Version 16 to see if I could still time travel to that point. It was successful and deleted all logs prior to the new checkpoint at Version 16:
Dave_Nithio_1-1726759722776.png 
Again this was as expected. The issue is, I then ran REPLACE TABLE AS SELECT again assuming that I would then delete any logs prior to my new checkpoint, meaning version 16 would no longer exist in the logs as it is well beyond my log retention. This was not the case though. I attempted to run multiple commits against the table which resulted in a few more checkpoints, but version 16 has not been removed from my delta history.
Dave_Nithio_5-1726760080078.pngDave_Nithio_6-1726760100546.png
I am wondering if is it related to deletedFileRetentionDuration delta setting. Maybe version 16 is referenced in newer files (due to RESTORE) so it will continue to maintain that version in the delta log? If so, should I expect it to be deleted 7 days after today? AarpMembership

Looking for the same issue. Bumped into your thread. Thanks for creating it. Looking forward for solution.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group