cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

GOLD table slowed down at MERGE INTO

JRT5933
New Contributor III

Howdy - I recently took a table FACT_TENDER and made it into a medalliona tyle TABLE to test performance since I suspected medallion would be quicker.

Key differences:

  • Both tables use bronze data
  • original has all logic in one long notebook
    • MERGE INTO that updates/inserts records takes roughly 13-minutes
  • Medallion table that reads in SILVER table and performs two JOINS
    • MERGE INTO tha updates/inserts records takes about 2 hours...

The SILVER table is quick for the medallion build, so I am at a loss here... I have tried optimziing for range joins, filtering out data, and ordering by but none of these have worked. Any thoughts? I can provide more detail here.

1 ACCEPTED SOLUTION

Accepted Solutions

JRT5933
New Contributor III

I ended up instituing true and tried PARTITIONING and PRUNING methods to boost performance, which has succeeded.

View solution in original post

4 REPLIES 4

Ajay-Pandey
Esteemed Contributor III

Hi @Jaime Tirado​ ,

Please refer below blog that might help you-

How to improve performance of Delta Lake MERGE INTO queries using partition pruning - Databricks

Rishabh264
Honored Contributor II

yes by referring this blog, you can have a much better understanding

Rishabh Pandey

JRT5933
New Contributor III

I have seen this article and it is not particularly helpful to my case. I have a DELTA table so I cannot add a partition.

JRT5933
New Contributor III

I ended up instituing true and tried PARTITIONING and PRUNING methods to boost performance, which has succeeded.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!