- 158 Views
- 1 replies
- 3 kudos
I Built an AI-Powered Data Pipeline Generator for Databricks — Here Is What Happened
Building robust Medallion architectures takes time. Writing the same boilerplate for Auto Loader, streaming tables, and SCD Type 2 merges across different projects is a bottleneck.So, I ran an experiment: What happens if you let AI write your Spark D...
- 158 Views
- 1 replies
- 3 kudos
- 3 kudos
excellent work
- 3 kudos
- 285 Views
- 0 replies
- 1 kudos
LakeOps - A Lakehouse Operation Observability App
The Problem LakeOps SolvesData teams focused on cloud-based analytics solutions often experience operational bottlenecks. Engineers spend countless hours writing SQL joins across system logs monitoring and reporting pipeline failures, while engineeri...
- 285 Views
- 0 replies
- 1 kudos
- 650 Views
- 1 replies
- 3 kudos
Building an End-to-End Store Order Forecasting System on Databricks: From Zero to Automated Pipeline
Why I Built ThisAfter nearly 15 years working in enterprise supply chain and data engineering — most recently as a Staff Engineer at a large national grocery retailer — I have seen firsthand how retail organizations struggle with one persistent probl...
- 650 Views
- 1 replies
- 3 kudos
- 3 kudos
Really solid project—especially the focus on turning ML forecasts into practical order recommendations. The data quality and decision-engine approach makes it genuinely useful.
- 3 kudos
- 772 Views
- 5 replies
- 16 kudos
From 40 Minutes to 8 minutes: Why We Dropped MERGE in Our SAP BW to Databricks Gold Layer
Infra keeps getting faster, but the fix for a slow notebook is usually a config line, not a bigger cluster.One Spark Config, 32 Minutes Saved: Replacing MERGE with Dynamic Partition OverwriteBy @Phani_sannala , co-authored with @sridharplv Our gold n...
- 772 Views
- 5 replies
- 16 kudos
- 16 kudos
The zombie row bug is the real argument here reframing this from "MERGE is slow" to "MERGE is silently wrong for full-slice delivery" is what makes it land. Only thing I'd stress: dynamic partition overwrite is exactly as safe as your completeness ch...
- 16 kudos
- 2101 Views
- 3 replies
- 9 kudos
Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram
Medallion Architecture in Practice: The Design Decisions Nobody Puts in the DiagramEvery Lakehouse conversation eventually shows the same three boxes: Bronze, Silver, Gold. It's a great mental model — but on a real enterprise migration, the diagram i...
- 2101 Views
- 3 replies
- 9 kudos
- 9 kudos
Great work, man! Even though I don't know much about this field, your article made me curious and motivated me to read more about it. Thanks for sharing such valuable insights.keep posting
- 9 kudos
- 355 Views
- 0 replies
- 1 kudos
Why We Used Two Bronze Tables Instead of One — And Why It Mattered
Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...
- 355 Views
- 0 replies
- 1 kudos
- 13463 Views
- 4 replies
- 6 kudos
The Medallion Architecture: Why Data Layers Matter for Modern Organisations
In today’s data-driven world, organisations are drowning in information. From customer transactions and IoT sensor readings to social media interactions and operational logs, the volume and variety of data continue to grow exponentially. Yet many org...
- 13463 Views
- 4 replies
- 6 kudos
- 6 kudos
Thank you, @Louis_Frolio ! My next post is about Data Governance with Unity Catalog, stay tuned!!
- 6 kudos
- 5430 Views
- 0 replies
- 1 kudos
How Databricks Empowers Scalable Data Products Through Medallion Mesh Architecture?
Unlock the Power of Your Data: Solving Fragmentation and Governance Challenges!In today’s fast-paced, data-driven enterprises, fragmented data and governance issues create roadblocks to decision-making and innovation. Traditional architectures strugg...
- 5430 Views
- 0 replies
- 1 kudos