- 6479 Views
- 7 replies
- 8 kudos
Clean up _delta_log files
Hello experts. We are trying to clarify how to clean up the large amount of files that are being accumulated in the _delta_log folder (json, crc and checkpoint files). We went through the related posts in the forum and followed the below:SET spark.da...
- 6479 Views
- 7 replies
- 8 kudos
- 8 kudos
What you’re seeing is expected behavior — the _delta_log folder always keeps a history of JSON commit files, checkpoint files, and CRCs. Even if you lower delta.logRetentionDuration and run VACUUM, cleanup won’t happen immediately. A couple of points...
- 8 kudos
- 5902 Views
- 5 replies
- 3 kudos
Resolved! AutoOptimize, OPTIMIZE command and Vacuum command : Order, production implementation best practices
So databricks gives us great toolkit in the form optimization and vacuum. But, in terms of operationaling them, I am really confused on the best practice.Should we enable "optimized writes" by setting the following at a workspace level?spark.conf.set...
- 5902 Views
- 5 replies
- 3 kudos
- 3 kudos
@AKSHAY PALLERLA​ Just checking in to see if you got a solution to the issue you shared above. Let us know!Thanks to @Werner Stinckens​ for jumping in, as always!
- 3 kudos