difference between optimize and auto optimize and Optimize in delta

User16826994223
Databricks Employee
Databricks Employee

What would be good for me , if I should use Optimize every time or should I be using auto-optimize?

brickster_2018
Databricks Employee
Databricks Employee

  • Optimize:  Bin-packing/Compaction. Idempotent and Incremental
  • Optimize + Z-Order: Helps in Data Skipping; Use Range Partitioning
  • Optimize write: Improve the write operation to the Delta table. optimization is performed before the write/during the write.
  • Auto-compaction: Post-commit action Triggered when more than 50 files/directory

Auto-optimize = Optimize Write + Auto-compaction

What is suitable is determined by the use cases.

Reference: