cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to prevent Delta Lake checkpoints to be removed in Databricks?

User16826994223
Honored Contributor III

I am seeing with new commits the old checkpoints are getting removed and i can time travel only last 10 versions , Is there any way I can prevent it so that delat checkpoints are not removed

I'm using Azure Databricks 7.3 LTS ML.

1 ACCEPTED SOLUTION

Accepted Solutions

User16826994223
Honored Contributor III

If you want to keep your checkpoints X days, you can set delta.checkpointRetentionDuration to X days this way:

spark.sql(f"""
        ALTER TABLE delta.`path`
            SET TBLPROPERTIES (
                delta.checkpointRetentionDuration = 'X days'
            )
        """
)

View solution in original post

1 REPLY 1

User16826994223
Honored Contributor III

If you want to keep your checkpoints X days, you can set delta.checkpointRetentionDuration to X days this way:

spark.sql(f"""
        ALTER TABLE delta.`path`
            SET TBLPROPERTIES (
                delta.checkpointRetentionDuration = 'X days'
            )
        """
)

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.