Deleted File Retention Duration in Cluster not work

William_Scardua
Valued Contributor

‎08-19-2024 03:32 PM - edited ‎08-19-2024 03:51 PM

Hi Guys,

I tried to set the retention period in my Cluster, but it`s not work

Cluster:

Screenshot 2024-08-19 at 19.51.25.png

Notebook:

Screenshot 2024-08-19 at 19.29.59.png

Does not remove physical files

Do you have any ideas? thanks

0 Kudos

Sidhant07
Databricks Employee
Databricks Employee

‎12-09-2024 12:13 AM - edited ‎12-09-2024 12:14 AM

Hi,

It seems you are setting the retention period to 0 hours.

You need to first use this config spark.databricks.delta.retentionDurationCheck.enabled to false.

and then set the retentention duration config as you used.

You can do a DRY RUN to check if there are any files which are ready to be vacummed.

Reference link: https://docs.databricks.com/en/sql/language-manual/delta-vacuum.html

0 Kudos