cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

What is the difference between OPTIMIZE and Auto Optimize?

User16783854657
New Contributor III

I see that Delta Lake has an OPTIMIZE command and also table properties for Auto Optimize. What are the differences between these and when should I use one over the other?

2 REPLIES 2

User16783854657
New Contributor III

The OPTIMIZE command is a SQL command that can be run regularly or Ad Hoc. What it does is pack small files into larger files. Additionally, you can specify predicates to only run the command on a subset of a table, and also specify that you want to ZORDER on specific columns of the table.

Auto Optimize is a table property that consists of two parts: Optimized Writes and Auto Compaction. Once enabled on a table, all writes to that table will be carried out according to the config. Optimized Writes lowers the number of files output per write, whereas Auto Compaction will perform a more selective version of the OPTIMIZE SQL command after each write, and will bin pack partitions that have a large number of small files per partition (50 small files at time of writing).

If you want to ZORDER a table, you must use the OPTIMIZE SQL command. Otherwise, you can follow guidelines for Auto Optimize found here

brickster_2018
Esteemed Contributor

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group