Is there any way to control the autoOptimize interval?

[brickster_2018]

I can see my streaming jobs running optimize jobs more frequently, Is there any property so I can control autoOptimize duration

[brickster_2018 · Accepted solution]

The autoOptimize is not performed on a time basis. It's an event-based trigger. Once the delta table/partition has 50 (default value of spark.databricks.delta.autoCompact.minNumFiles) files, auto-compaction is triggered.

To reduce the frequency, increase the value of the aforementioned configuration. optionally you can completely turn off the auto-compaction and run a regular optimize command on a daily basis.