ok no problem.

Auto optimize exists in fact as two operations. You have optimized writes (delta.autoOptimize.optimizeWrite)

which aims to write files of 128 MB. This is an approximate size and can vary depending on dataset characteristics. Often 128 MB will not be possible.

So then there is also auto compaction (delta.autoOptimize.autoCompact).

After an individual write, databricks checks if files can further be compacted, and runs an optimize job (with 128 MB file sizes instead of the 1 GB file size used in the standard

OPTIMIZE) to further compact files for partitions that have the most number of small files.

These optimizations come with a cost of course (shuffle f.e.). However, the net outcome is often positive because you write smaller files which are still large enough for good query performance.

The increase in throughput is as follows: let's say you want to write about 1000MB.

In a classic optimize example, this would create a single partition of 1000MB. A single partition means 1 task executed by one worker.

If you would write that 1000MB in 128MB partitions, you could parallelize the write into 4 or 5 tasks, hence more throughput.