Deleted File Retention Duration in Cluster not work

William_Scardua — Mon, 19 Aug 2024 22:51:50 GMT

Hi Guys,

I tried to set the retention period in my Cluster, but it`s not work

Cluster:

Notebook:

Does not remove physical files

Do you have any ideas? thanks

Re: Deleted File Retention Duration in Cluster not work

Sidhant07 — Mon, 09 Dec 2024 08:14:40 GMT

Hi,

It seems you are setting the retention period to 0 hours.

You need to first use this config spark.databricks.delta.retentionDurationCheck.enabled to false.

and then set the retentention duration config as you used.

You can do a DRY RUN to check if there are any files which are ready to be vacummed.