How to prevent Delta Lake checkpoints to be removed in Databricks?

User16826994223
Databricks Employee
Databricks Employee

‎06-25-2021 09:53 AM

I am seeing with new commits the old checkpoints are getting removed and i can time travel only last 10 versions , Is there any way I can prevent it so that delat checkpoints are not removed

I'm using Azure Databricks 7.3 LTS ML.

Labels:
0 Kudos

User16826994223
Databricks Employee
Databricks Employee

‎06-25-2021 09:54 AM

If you want to keep your checkpoints X days, you can set delta.checkpointRetentionDuration to X days this way:

spark.sql(f"""
        ALTER TABLE delta.`path`
            SET TBLPROPERTIES (
                delta.checkpointRetentionDuration = 'X days'
            )
        """
)

View solution in original post

0 Kudos