Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 06:19 AM
I can see my streaming jobs running optimize jobs more frequently, Is there any property so I can control autoOptimize duration
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 06:23 AM
The autoOptimize is not performed on a time basis. It's an event-based trigger. Once the delta table/partition has 50 (default value of spark.databricks.delta.autoCompact.minNumFiles) files, auto-compaction is triggered.
To reduce the frequency, increase the value of the aforementioned configuration. optionally you can completely turn off the auto-compaction and run a regular optimize command on a daily basis.