- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2022 08:33 AM
I am trying to set retention period for a delta by using following commands.
deltaTable = DeltaTable.forPath(spark,delta_path)
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
deltaTable.logRetentionDuration = "interval 1 days"
deltaTable.deletedFileRetentionDuration = "interval 1 days"
These commands are not working for me, I mean, they aren't removing any files for the given interval..where am I going wrong?
- Labels:
-
Delta table
-
Retention Period
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2022 08:56 AM
There are two ways:
1) Please set in cluster (Clusters -> edit -> Spark -> Spark config):
spark.databricks.delta.retentionDurationCheck.enabled false
2) or just before DeltaTable.forPath set (I think you need to change order in your code):
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2022 08:56 AM
There are two ways:
1) Please set in cluster (Clusters -> edit -> Spark -> Spark config):
spark.databricks.delta.retentionDurationCheck.enabled false
2) or just before DeltaTable.forPath set (I think you need to change order in your code):
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-16-2022 11:39 PM
Hi @Kaniz Fatma , Yes I was able to solve the issue! Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-16-2022 11:37 PM
Hi @Hubert Dudek , thanks for you response!

