Log files are not getting deleted automatically after logRetentionDuration internal

[vinaykumar]

Hi team

Log files are not getting deleted automatically after logRetentionDuration internal from delta log folder and after analysis , I see checkpoint files are not getting created after 10 commits .

Below table properties using

spark.sql(    f"""

    ALTER TABLE Table_name

      SET TBLPROPERTIES (

        delta.logRetentionDuration = 'interval 1 hours',

        delta.deletedFileRetentionDuration = 'interval 1 hours',

        delta.checkpointRetentionDuration = '1 hours'

      )

    """

No checkpoint.parquet file getting generated in delta log

[Lakshay]

Hi @vinay kumar​ , Could you please run "SHOW TBLPROPERTIES <table name>" command and attach a screenshot here?

[vinaykumar]

Please find TBLPROPERTIES 

[Lakshay]

Hi @vinay kumar​ , could you please answer the below questions as well

1. What is the latest version of the delta table?
2. Are there any checkpoint files created?
3. Are you doing the commits using the job cluster?

[vinaykumar]

1. What is the latest version of the delta table? -

1. Are there any checkpoint files created? Checkpoint is not getting created after 10 commits
2. Are you doing the commits using the job cluster? using all purpose cluster

[Lakshay]

Hi @vinay kumar​ , It is difficult to tell what is causing this issue. I advise you to raise a support case with us to have a closer look. You can raise a support case using the link: https://help.databricks.com/

[Anonymous]

Hi @vinay kumar​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[alex307]

Any body get any solution?

[alex421]

Same question.

[K_Anudeep]

Hello @alex421 , could you post a new question with the table details and also provide the output of desc history in CSV format and also the files present in the delta log. I need to  check the details and  answer