topic Re: How to prevent Delta Lake checkpoints to be removed in Databricks? in Data Engineering

How to prevent Delta Lake checkpoints to be removed in Databricks?

User16826994223 — Fri, 25 Jun 2021 16:53:40 GMT

I am seeing with new commits the old checkpoints are getting removed and i can time travel only last 10 versions , Is there any way I can prevent it so that delat checkpoints are not removed

I'm using Azure Databricks 7.3 LTS ML.

Re: How to prevent Delta Lake checkpoints to be removed in Databricks?

User16826994223 — Fri, 25 Jun 2021 16:54:57 GMT

If you want to keep your checkpoints X days, you can set delta.checkpointRetentionDuration to X days this way:

spark.sql(f"""
        ALTER TABLE delta.`path`
            SET TBLPROPERTIES (
                delta.checkpointRetentionDuration = 'X days'
            )
        """
)