cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

What is the difference between OPTIMIZE and Auto Optimize?

User16783854657
New Contributor III

I see that Delta Lake has an OPTIMIZE command and also table properties for Auto Optimize. What are the differences between these and when should I use one over the other?

2 REPLIES 2

User16783854657
New Contributor III

The OPTIMIZE command is a SQL command that can be run regularly or Ad Hoc. What it does is pack small files into larger files. Additionally, you can specify predicates to only run the command on a subset of a table, and also specify that you want to ZORDER on specific columns of the table.

Auto Optimize is a table property that consists of two parts: Optimized Writes and Auto Compaction. Once enabled on a table, all writes to that table will be carried out according to the config. Optimized Writes lowers the number of files output per write, whereas Auto Compaction will perform a more selective version of the OPTIMIZE SQL command after each write, and will bin pack partitions that have a large number of small files per partition (50 small files at time of writing).

If you want to ZORDER a table, you must use the OPTIMIZE SQL command. Otherwise, you can follow guidelines for Auto Optimize found here

User16869510359
Esteemed Contributor
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.