- 538 Views
- 5 replies
- 16 kudos
From 40 Minutes to 8 minutes: Why We Dropped MERGE in Our SAP BW to Databricks Gold Layer
Infra keeps getting faster, but the fix for a slow notebook is usually a config line, not a bigger cluster.One Spark Config, 32 Minutes Saved: Replacing MERGE with Dynamic Partition OverwriteBy @Phani_sannala , co-authored with @sridharplv Our gold n...
- 538 Views
- 5 replies
- 16 kudos
- 16 kudos
The zombie row bug is the real argument here reframing this from "MERGE is slow" to "MERGE is silently wrong for full-slice delivery" is what makes it land. Only thing I'd stress: dynamic partition overwrite is exactly as safe as your completeness ch...
- 16 kudos
- 507 Views
- 0 replies
- 0 kudos
Gold Layer Design on Databricks — MERGE vs Overwrite, Partitioning, SCD Type 2 from SAP
Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...
- 507 Views
- 0 replies
- 0 kudos
- 316 Views
- 0 replies
- 1 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...
- 316 Views
- 0 replies
- 1 kudos