- 304 Views
- 4 replies
- 15 kudos
From 40 Minutes to 8 minutes: Why We Dropped MERGE in Our SAP BW to Databricks Gold Layer
Infra keeps getting faster, but the fix for a slow notebook is usually a config line, not a bigger cluster.One Spark Config, 32 Minutes Saved: Replacing MERGE with Dynamic Partition OverwriteBy @Phani_sannala , co-authored with @sridharplv Our gold n...
- 304 Views
- 4 replies
- 15 kudos
- 966 Views
- 0 replies
- 0 kudos
How a Partitioning Mistake Turned a 12-Minute Databricks Job Into a 2-Hour Nightmare
Hello Databricks Community!I recently published a detailed breakdown on Medium about a real-world optimization nightmare we faced, and I wanted to share the core lessons learned with this group.We had a highly efficient Delta table pipeline handling ...
- 966 Views
- 0 replies
- 0 kudos
- 469 Views
- 0 replies
- 0 kudos
Gold Layer Design on Databricks — MERGE vs Overwrite, Partitioning, SCD Type 2 from SAP
Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...
- 469 Views
- 0 replies
- 0 kudos
- 1031 Views
- 0 replies
- 3 kudos
Understanding Delta Table Partition Size Distribution Using the Delta Log
When working with externally managed Delta tables and traditional partitioning strategies (for example by day, week, or month), one common challenge is:How large are my partitions actually?Before deciding whether to partition by day vs. week vs. mont...
- 1031 Views
- 0 replies
- 3 kudos