topic Re: How to set retention period for a delta table lower than the default period? Is it even possible? in Data Engineering

How to set retention period for a delta table lower than the default period? Is it even possible?

manasa — Tue, 15 Mar 2022 15:33:42 GMT

I am trying to set retention period for a delta by using following commands.

deltaTable = DeltaTable.forPath(spark,delta_path)

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")

deltaTable.logRetentionDuration = "interval 1 days"

deltaTable.deletedFileRetentionDuration = "interval 1 days"

These commands are not working for me, I mean, they aren't removing any files for the given interval..where am I going wrong?

Hubert-Dudek — Tue, 15 Mar 2022 15:56:16 GMT

There are two ways:

1) Please set in cluster (Clusters -> edit -> Spark -> Spark config):

spark.databricks.delta.retentionDurationCheck.enabled false

2) or just before DeltaTable.forPath set (I think you need to change order in your code):

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")

manasa — Thu, 17 Mar 2022 06:37:56 GMT

Hi @Hubert Dudek , thanks for you response!

manasa — Thu, 17 Mar 2022 06:39:10 GMT

Hi @Kaniz Fatma , Yes I was able to solve the issue! Thanks