topic What type of cluster configuration should one use to run Optimize on a Delta Table in Data Engineering https://community.databricks.com/t5/data-engineering/what-type-of-cluster-configuration-should-one-use-to-run/m-p/20937#M14185 Thu, 24 Jun 2021 04:13:35 GMT aladda 2021-06-24T04:13:35Z What type of cluster configuration should one use to run Optimize on a Delta Table https://community.databricks.com/t5/data-engineering/what-type-of-cluster-configuration-should-one-use-to-run/m-p/20937#M14185 Thu, 24 Jun 2021 04:13:35 GMT https://community.databricks.com/t5/data-engineering/what-type-of-cluster-configuration-should-one-use-to-run/m-p/20937#M14185 aladda 2021-06-24T04:13:35Z Re: What type of cluster configuration should one use to run Optimize on a Delta Table https://community.databricks.com/t5/data-engineering/what-type-of-cluster-configuration-should-one-use-to-run/m-p/20938#M14186 <P>Optimize merges small files into larger ones and can involve shuffling and creation of large in-memory partitions. Thus its recommended to use a memory optimized executor configuration to prevent spilling to disk. IN additional use of autoscaling will also help with adapting to spikes in data volume</P><P></P><P>It is also recommended that you use a separate jobs cluster if possible to run optimize so as to no impact the performance of other jobs</P> Thu, 24 Jun 2021 04:15:49 GMT https://community.databricks.com/t5/data-engineering/what-type-of-cluster-configuration-should-one-use-to-run/m-p/20938#M14186 aladda 2021-06-24T04:15:49Z