How to set retention period for a delta table lower than the default period? Is it even possible?

[manasa]

I am trying to set retention period for a delta by using following commands.

deltaTable = DeltaTable.forPath(spark,delta_path)

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")

deltaTable.logRetentionDuration = "interval 1 days"

deltaTable.deletedFileRetentionDuration = "interval 1 days"

These commands are not working for me, I mean, they aren't removing any files for the given interval..where am I going wrong?

[Hubert-Dudek · Accepted solution]

There are two ways:

1) Please set in cluster (Clusters -> edit -> Spark -> Spark config):

spark.databricks.delta.retentionDurationCheck.enabled false

 

2) or just before DeltaTable.forPath set (I think you need to change order in your code):

spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")

[Kaniz]

Hi @Manasa Kalluri​ , It seems @Hubert Dudek​ has given a comprehensive solution. Were you able to solve your problem?

[manasa]

Hi @Kaniz Fatma​ , Yes I was able to solve the issue! Thanks

[manasa]

Hi @Hubert Dudek​ , thanks for you response!

[Kaniz]

Hi @Manasa Kalluri​ , Thank you for the update. Would you like to mark @Hubert Dudek​ 's answer as "Best" which would help our community members hereafter 😊 ?