topic Re: Log files are not getting deleted automatically after logRetentionDuration internal in Data Engineering

Log files are not getting deleted automatically after logRetentionDuration internal

vinaykumar — Sat, 18 Feb 2023 19:43:28 GMT

Hi team

Log files are not getting deleted automatically after logRetentionDuration internal from delta log folder and after analysis , I see checkpoint files are not getting created after 10 commits .

Below table properties using

spark.sql( f"""

ALTER TABLE Table_name

SET TBLPROPERTIES (

delta.logRetentionDuration = 'interval 1 hours',

delta.deletedFileRetentionDuration = 'interval 1 hours',

delta.checkpointRetentionDuration = '1 hours'

)

"""

No checkpoint.parquet file getting generated in delta log

Re: Log files are not getting deleted automatically after logRetentionDuration internal

Lakshay — Mon, 20 Feb 2023 11:40:29 GMT

Hi @vinay kumar , Could you please run "SHOW TBLPROPERTIES <table name>" command and attach a screenshot here?

Re: Log files are not getting deleted automatically after logRetentionDuration internal

vinaykumar — Mon, 20 Feb 2023 12:04:23 GMT

Please find TBLPROPERTIES

Re: Log files are not getting deleted automatically after logRetentionDuration internal

Lakshay — Mon, 20 Feb 2023 12:16:48 GMT

Hi @vinay kumar , could you please answer the below questions as well

What is the latest version of the delta table?
Are there any checkpoint files created?
Are you doing the commits using the job cluster?

Re: Log files are not getting deleted automatically after logRetentionDuration internal

vinaykumar — Mon, 20 Feb 2023 12:19:48 GMT

What is the latest version of the delta table? -

Are there any checkpoint files created? Checkpoint is not getting created after 10 commits
Are you doing the commits using the job cluster? using all purpose cluster

Re: Log files are not getting deleted automatically after logRetentionDuration internal

Lakshay — Mon, 20 Feb 2023 12:48:46 GMT

Hi @vinay kumar , It is difficult to tell what is causing this issue. I advise you to raise a support case with us to have a closer look. You can raise a support case using the link: https://help.databricks.com/

Re: Log files are not getting deleted automatically after logRetentionDuration internal

Anonymous — Fri, 21 Apr 2023 09:26:48 GMT

Hi @vinay kumar

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Log files are not getting deleted automatically after logRetentionDuration internal

alex307 — Tue, 04 Nov 2025 14:23:54 GMT

Any body get any solution?

Re: Log files are not getting deleted automatically after logRetentionDuration internal

alex421 — Sun, 21 Dec 2025 12:51:27 GMT

Same question.

Re: Log files are not getting deleted automatically after logRetentionDuration internal

K_Anudeep — Mon, 22 Dec 2025 09:20:33 GMT

Hello @alex421 , could you post a new question with the table details and also provide the output of desc history in CSV format and also the files present in the delta log. I need to check the details and answer