topic Triggering clean-ups in Spark to handle accumulated metadata in Data Engineering https://community.databricks.com/t5/data-engineering/triggering-clean-ups-in-spark-to-handle-accumulated-metadata/m-p/19631#M13185 Fri, 25 Jun 2021 15:48:50 GMT User16826994223 2021-06-25T15:48:50Z Triggering clean-ups in Spark to handle accumulated metadata https://community.databricks.com/t5/data-engineering/triggering-clean-ups-in-spark-to-handle-accumulated-metadata/m-p/19631#M13185 Fri, 25 Jun 2021 15:48:50 GMT https://community.databricks.com/t5/data-engineering/triggering-clean-ups-in-spark-to-handle-accumulated-metadata/m-p/19631#M13185 User16826994223 2021-06-25T15:48:50Z Re: Triggering clean-ups in Spark to handle accumulated metadata https://community.databricks.com/t5/data-engineering/triggering-clean-ups-in-spark-to-handle-accumulated-metadata/m-p/19632#M13186 <P> setting the parameter ‘spark.cleaner.ttl’ or by dividing the long running jobs into different batches and writing the intermediary results to the disk.</P> Fri, 25 Jun 2021 15:49:20 GMT https://community.databricks.com/t5/data-engineering/triggering-clean-ups-in-spark-to-handle-accumulated-metadata/m-p/19632#M13186 User16826994223 2021-06-25T15:49:20Z