https://community.databricks.com/t5/machine-learning/difference-between-optimize-and-auto-optimize-and-optimize-in/m-p/23374#M1324 <P></P><UL><LI>Optimize:&nbsp;&nbsp;Bin-packing/Compaction. Idempotent and Incremental</LI><LI>Optimize + Z-Order: Helps in Data Skipping; Use Range Partitioning</LI><LI>Optimize write: Improve the write operation to the Delta table. optimization is performed before the write/during the write. </LI><LI>Auto-compaction: Post-commit action Triggered when more than 50 files/directory</LI></UL><P></P><P>Auto-optimize = Optimize Write + Auto-compaction</P><P></P><P>What is suitable is determined by the use cases. </P><P>Reference:</P><UL><LI> <A href="https://docs.databricks.com/delta/optimizations/auto-optimize.html" target="test_blank">https://docs.databricks.com/delta/optimizations/auto-optimize.html</A></LI><LI><A href="https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping" target="test_blank">https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping</A></LI></UL><P></P> Tue, 22 Jun 2021 23:07:54 GMT brickster_2018 2021-06-22T23:07:54Z https://community.databricks.com/t5/machine-learning/difference-between-optimize-and-auto-optimize-and-optimize-in/m-p/23373#M1323 <P>What would be good for me , if I should use Optimize every time or should I be using auto-optimize?</P> Thu, 17 Jun 2021 08:27:30 GMT https://community.databricks.com/t5/machine-learning/difference-between-optimize-and-auto-optimize-and-optimize-in/m-p/23373#M1323 User16826994223 2021-06-17T08:27:30Z https://community.databricks.com/t5/machine-learning/difference-between-optimize-and-auto-optimize-and-optimize-in/m-p/23374#M1324 <P></P><UL><LI>Optimize:&nbsp;&nbsp;Bin-packing/Compaction. Idempotent and Incremental</LI><LI>Optimize + Z-Order: Helps in Data Skipping; Use Range Partitioning</LI><LI>Optimize write: Improve the write operation to the Delta table. optimization is performed before the write/during the write. </LI><LI>Auto-compaction: Post-commit action Triggered when more than 50 files/directory</LI></UL><P></P><P>Auto-optimize = Optimize Write + Auto-compaction</P><P></P><P>What is suitable is determined by the use cases. </P><P>Reference:</P><UL><LI> <A href="https://docs.databricks.com/delta/optimizations/auto-optimize.html" target="test_blank">https://docs.databricks.com/delta/optimizations/auto-optimize.html</A></LI><LI><A href="https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping" target="test_blank">https://docs.databricks.com/delta/optimizations/file-mgmt.html#data-skipping</A></LI></UL><P></P> Tue, 22 Jun 2021 23:07:54 GMT https://community.databricks.com/t5/machine-learning/difference-between-optimize-and-auto-optimize-and-optimize-in/m-p/23374#M1324 brickster_2018 2021-06-22T23:07:54Z