GOLD table slowed down at MERGE INTO

[JRT5933]

Howdy - I recently took a table FACT_TENDER and made it into a medalliona tyle TABLE to test performance since I suspected medallion would be quicker.

Key differences:

• Both tables use bronze data
• original has all logic in one long notebook
  • MERGE INTO that updates/inserts records takes roughly 13-minutes
• Medallion table that reads in SILVER table and performs two JOINS
  • MERGE INTO tha updates/inserts records takes about 2 hours...

The SILVER table is quick for the medallion build, so I am at a loss here... I have tried optimziing for range joins, filtering out data, and ordering by but none of these have worked. Any thoughts? I can provide more detail here.

[Ajay-Pandey]

Hi @Jaime Tirado​ ,

Please refer below blog that might help you-

How to improve performance of Delta Lake MERGE INTO queries using partition pruning - Databricks

[Rishabh-Pandey]

yes by referring this blog, you can have a much better understanding

[JRT5933]

I have seen this article and it is not particularly helpful to my case. I have a DELTA table so I cannot add a partition.

[JRT5933 · Accepted solution]

I ended up instituing true and tried PARTITIONING and PRUNING methods to boost performance, which has succeeded.