Merge take too long - Databricks Community - 87122

Register to join the community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi all,

I performed a merge process on approximately 19 million rows using two i3.4xlarge workers. However, the process took around 20 minutes to complete. How can I further optimize this process? I have already implemented the OPTIMIZE command and used a liquid cluster setup.

1 REPLY 1

@abduldjafar Use this general doc to optimize your workload based on your job analysis

https://www.databricks.com/discover/pages/optimize-data-workloads-guide

li.media.uploader-dialog.title

li.media.uploader-dialog.tabs.upload li.media.uploader-dialog.tabs.upload
li.media.uploader-dialog.tabs.web li.media.uploader-dialog.tabs.web
li.media.uploader-dialog.tabs.saved li.media.uploader-dialog.tabs.saved

li.media.album-selector.label

li.media.album-selector.label

li.media.album-selector.title

li.media.album-selector.new.title

li.media.image-upload.drag

li.media.image-upload.drop

li.media.image-upload.mobile.instruction

li.media.image-upload.no-flash

You must be signed in to add attachments

never-displayed

Announcements

Exciting Opportunity to Collaborate with Us!

Intelligent Data Warehousing: AI/BI for Self-service Analytics

Share Your Thoughts on Databricks & Get Rewarded!

Get Started With Lakehouse Architecture | Pass a quiz to earn your certificate completion.

Virtual Learning Festival: 9 April - 30 April

upload