- 223 Views
- 4 replies
- 15 kudos
From 40 Minutes to 8 minutes: Why We Dropped MERGE in Our SAP BW to Databricks Gold Layer
Infra keeps getting faster, but the fix for a slow notebook is usually a config line, not a bigger cluster.One Spark Config, 32 Minutes Saved: Replacing MERGE with Dynamic Partition OverwriteBy @Phani_sannala , co-authored with @sridharplv Our gold n...
- 223 Views
- 4 replies
- 15 kudos
- 447 Views
- 0 replies
- 0 kudos
Why Your Delta MERGE is 5x Slower Than an Overwrite (And How to Fix It)
Hey everyone,We’ve all been there: a Delta Lake MERGE job that should take 20 minutes drags on for 90 minutes, while a full overwrite of the same table finishes in under 20. When an overwrite outpaces a selective merge, it's a massive red flag that y...
- 447 Views
- 0 replies
- 0 kudos
- 1027 Views
- 0 replies
- 3 kudos
Understanding Delta Table Partition Size Distribution Using the Delta Log
When working with externally managed Delta tables and traditional partitioning strategies (for example by day, week, or month), one common challenge is:How large are my partitions actually?Before deciding whether to partition by day vs. week vs. mont...
- 1027 Views
- 0 replies
- 3 kudos
- 2737 Views
- 2 replies
- 3 kudos
Resolved! 🚀 Boost Databricks Performance ✅ Lazy Evaluation and Spark DataFrame Caching ☕
Published new video in my recently created youtube channel about one of my favorite topics: performance. So, here is a new video whose goal is to explain clearly what lazy evaluation is and how to use caching to boost performance. https://studio.y...
- 2737 Views
- 2 replies
- 3 kudos
- 3 kudos
Not sure if previous link works fine, here is the correct one https://youtu.be/pLGAr1VXQSQ?si=QaKzaNfzrNl_0Tv9
- 3 kudos