- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-21-2023 11:01 AM
Howdy - I recently took a table FACT_TENDER and made it into a medalliona tyle TABLE to test performance since I suspected medallion would be quicker.
Key differences:
- Both tables use bronze data
- original has all logic in one long notebook
- MERGE INTO that updates/inserts records takes roughly 13-minutes
- Medallion table that reads in SILVER table and performs two JOINS
- MERGE INTO tha updates/inserts records takes about 2 hours...
The SILVER table is quick for the medallion build, so I am at a loss here... I have tried optimziing for range joins, filtering out data, and ordering by but none of these have worked. Any thoughts? I can provide more detail here.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-21-2023 11:13 PM
Hi @Jaime Tirado ,
Please refer below blog that might help you-
How to improve performance of Delta Lake MERGE INTO queries using partition pruning - Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2023 02:06 AM
yes by referring this blog, you can have a much better understanding
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2023 05:55 AM
I have seen this article and it is not particularly helpful to my case. I have a DELTA table so I cannot add a partition.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2023 07:30 AM
I ended up instituing true and tried PARTITIONING and PRUNING methods to boost performance, which has succeeded.
