cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

GOLD table slowed down at MERGE INTO

JRT5933
New Contributor III

Howdy - I recently took a table FACT_TENDER and made it into a medalliona tyle TABLE to test performance since I suspected medallion would be quicker.

Key differences:

  • Both tables use bronze data
  • original has all logic in one long notebook
    • MERGE INTO that updates/inserts records takes roughly 13-minutes
  • Medallion table that reads in SILVER table and performs two JOINS
    • MERGE INTO tha updates/inserts records takes about 2 hours...

The SILVER table is quick for the medallion build, so I am at a loss here... I have tried optimziing for range joins, filtering out data, and ordering by but none of these have worked. Any thoughts? I can provide more detail here.

1 ACCEPTED SOLUTION

Accepted Solutions

JRT5933
New Contributor III

I ended up instituing true and tried PARTITIONING and PRUNING methods to boost performance, which has succeeded.

View solution in original post

4 REPLIES 4

Ajay-Pandey
Esteemed Contributor III

Hi @Jaime Tirado​ ,

Please refer below blog that might help you-

How to improve performance of Delta Lake MERGE INTO queries using partition pruning - Databricks

Ajay Kumar Pandey

yes by referring this blog, you can have a much better understanding

Rishabh Pandey

JRT5933
New Contributor III

I have seen this article and it is not particularly helpful to my case. I have a DELTA table so I cannot add a partition.

JRT5933
New Contributor III

I ended up instituing true and tried PARTITIONING and PRUNING methods to boost performance, which has succeeded.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group