https://community.databricks.com/t5/data-engineering/merge-take-too-long/m-p/87122#M37385 <P>Hi all,</P><P>I performed a merge process on approximately 19 million rows using two i3.4xlarge workers. However, the process took around 20 minutes to complete. How can I further optimize this process? I have already implemented the OPTIMIZE command and used a <A href="https://docs.databricks.com/en/delta/clustering.html" target="_self">liquid cluster setup</A>.</P> Mon, 02 Sep 2024 02:56:57 GMT abduldjafar 2024-09-02T02:56:57Z https://community.databricks.com/t5/data-engineering/merge-take-too-long/m-p/87122#M37385 <P>Hi all,</P><P>I performed a merge process on approximately 19 million rows using two i3.4xlarge workers. However, the process took around 20 minutes to complete. How can I further optimize this process? I have already implemented the OPTIMIZE command and used a <A href="https://docs.databricks.com/en/delta/clustering.html" target="_self">liquid cluster setup</A>.</P> Mon, 02 Sep 2024 02:56:57 GMT https://community.databricks.com/t5/data-engineering/merge-take-too-long/m-p/87122#M37385 abduldjafar 2024-09-02T02:56:57Z https://community.databricks.com/t5/data-engineering/merge-take-too-long/m-p/98330#M39696 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/118579">@abduldjafar</a>&nbsp;Use this general doc to optimize your workload based on your job analysis</P> <P><A href="https://www.databricks.com/discover/pages/optimize-data-workloads-guide" target="_blank">https://www.databricks.com/discover/pages/optimize-data-workloads-guide</A></P> Mon, 11 Nov 2024 13:08:31 GMT https://community.databricks.com/t5/data-engineering/merge-take-too-long/m-p/98330#M39696 MuthuLakshmi 2024-11-11T13:08:31Z