https://community.databricks.com/t5/data-engineering/what-s-the-best-instance-type-to-run-optimize-bin-packing-and-z/m-p/26774#M18786 <P>I've been doing some research on optimizing data storage while implementing delta, however, I'm not sure which instance type would be best for this.</P> Fri, 21 May 2021 18:40:49 GMT User16790091296 2021-05-21T18:40:49Z https://community.databricks.com/t5/data-engineering/what-s-the-best-instance-type-to-run-optimize-bin-packing-and-z/m-p/26774#M18786 <P>I've been doing some research on optimizing data storage while implementing delta, however, I'm not sure which instance type would be best for this.</P> Fri, 21 May 2021 18:40:49 GMT https://community.databricks.com/t5/data-engineering/what-s-the-best-instance-type-to-run-optimize-bin-packing-and-z/m-p/26774#M18786 User16790091296 2021-05-21T18:40:49Z https://community.databricks.com/t5/data-engineering/what-s-the-best-instance-type-to-run-optimize-bin-packing-and-z/m-p/26775#M18787 <P>OPTIMIZE as you alluded has two operations , Bin-packing and <B>multi-dimensional clustering ( zorder)</B></P><UL><LI>Bin-packing optimization is&nbsp;<I>idempotent</I>, meaning that if it is run twice on the same dataset, the second run has no effect</LI><LI>Z-Ordering is&nbsp;<I>not idempotent</I>&nbsp;but aims to be an incremental operation. if no new data was added to a partition that was just Z-Ordered, another Z-Ordering of that partition will not have any effect.</LI></UL><P></P><P>With that in context, you'd want to check if your optimize job is CPU bound or memory bound. Anecdotal evidence shows that ESv3 series in Azure and r5a/r5d instances in AWS gives good cpu/mem ratio and is good for optimize - but really <B>YMMV</B></P> Fri, 18 Jun 2021 05:26:45 GMT https://community.databricks.com/t5/data-engineering/what-s-the-best-instance-type-to-run-optimize-bin-packing-and-z/m-p/26775#M18787 sajith_appukutt 2021-06-18T05:26:45Z