Merge take too long

abduldjafar
New Contributor

Hi all,

I performed a merge process on approximately 19 million rows using two i3.4xlarge workers. However, the process took around 20 minutes to complete. How can I further optimize this process? I have already implemented the OPTIMIZE command and used a liquid cluster setup.

MuthuLakshmi
Databricks Employee
Databricks Employee

@abduldjafar Use this general doc to optimize your workload based on your job analysis

https://www.databricks.com/discover/pages/optimize-data-workloads-guide