topic GOLD table slowed down at MERGE INTO in Data Engineering

GOLD table slowed down at MERGE INTO

JRT5933 — Tue, 21 Feb 2023 19:01:03 GMT

Howdy - I recently took a table FACT_TENDER and made it into a medalliona tyle TABLE to test performance since I suspected medallion would be quicker.

Key differences:

Both tables use bronze data
original has all logic in one long notebook
- MERGE INTO that updates/inserts records takes roughly 13-minutes
Medallion table that reads in SILVER table and performs two JOINS
- MERGE INTO tha updates/inserts records takes about 2 hours...

The SILVER table is quick for the medallion build, so I am at a loss here... I have tried optimziing for range joins, filtering out data, and ordering by but none of these have worked. Any thoughts? I can provide more detail here.

Re: GOLD table slowed down at MERGE INTO

Ajay-Pandey — Wed, 22 Feb 2023 07:13:45 GMT

Hi @Jaime Tirado ,

Please refer below blog that might help you-

How to improve performance of Delta Lake MERGE INTO queries using partition pruning - Databricks

Re: GOLD table slowed down at MERGE INTO

Rishabh-Pandey — Wed, 22 Feb 2023 10:06:49 GMT

yes by referring this blog, you can have a much better understanding

Re: GOLD table slowed down at MERGE INTO

JRT5933 — Wed, 22 Feb 2023 13:55:15 GMT

I have seen this article and it is not particularly helpful to my case. I have a DELTA table so I cannot add a partition.

Re: GOLD table slowed down at MERGE INTO

JRT5933 — Wed, 22 Feb 2023 15:30:37 GMT

I ended up instituing true and tried PARTITIONING and PRUNING methods to boost performance, which has succeeded.