โ03-15-2022 08:33 AM
I am trying to set retention period for a delta by using following commands.
deltaTable = DeltaTable.forPath(spark,delta_path)
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
deltaTable.logRetentionDuration = "interval 1 days"
deltaTable.deletedFileRetentionDuration = "interval 1 days"
These commands are not working for me, I mean, they aren't removing any files for the given interval..where am I going wrong?
โ03-15-2022 08:56 AM
There are two ways:
1) Please set in cluster (Clusters -> edit -> Spark -> Spark config):
spark.databricks.delta.retentionDurationCheck.enabled false
2) or just before DeltaTable.forPath set (I think you need to change order in your code):
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
โ03-15-2022 08:56 AM
There are two ways:
1) Please set in cluster (Clusters -> edit -> Spark -> Spark config):
spark.databricks.delta.retentionDurationCheck.enabled false
2) or just before DeltaTable.forPath set (I think you need to change order in your code):
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
โ03-16-2022 11:39 PM
Hi @Kaniz Fatmaโ , Yes I was able to solve the issue! Thanks
โ03-16-2022 11:37 PM
Hi @Hubert Dudekโ , thanks for you response!
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group