- 1228 Views
- 5 replies
- 16 kudos
From 40 Minutes to 8 minutes: Why We Dropped MERGE in Our SAP BW to Databricks Gold Layer
Infra keeps getting faster, but the fix for a slow notebook is usually a config line, not a bigger cluster.One Spark Config, 32 Minutes Saved: Replacing MERGE with Dynamic Partition OverwriteBy @Phani_sannala , co-authored with @sridharplv Our gold n...
- 1228 Views
- 5 replies
- 16 kudos
- 16 kudos
The zombie row bug is the real argument here reframing this from "MERGE is slow" to "MERGE is silently wrong for full-slice delivery" is what makes it land. Only thing I'd stress: dynamic partition overwrite is exactly as safe as your completeness ch...
- 16 kudos
- 1176 Views
- 0 replies
- 0 kudos
How a Partitioning Mistake Turned a 12-Minute Databricks Job Into a 2-Hour Nightmare
Hello Databricks Community!I recently published a detailed breakdown on Medium about a real-world optimization nightmare we faced, and I wanted to share the core lessons learned with this group.We had a highly efficient Delta table pipeline handling ...
- 1176 Views
- 0 replies
- 0 kudos
- 718 Views
- 0 replies
- 0 kudos
Gold Layer Design on Databricks — MERGE vs Overwrite, Partitioning, SCD Type 2 from SAP
Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...
- 718 Views
- 0 replies
- 0 kudos
- 1334 Views
- 0 replies
- 3 kudos
Understanding Delta Table Partition Size Distribution Using the Delta Log
When working with externally managed Delta tables and traditional partitioning strategies (for example by day, week, or month), one common challenge is:How large are my partitions actually?Before deciding whether to partition by day vs. week vs. mont...
- 1334 Views
- 0 replies
- 3 kudos