Databricks Community

savlahanish27

Part 4 of my enterprise data platform series is up - this one cover why we added a fifth layer to the standard medallion architecture.We connected BI tools to the Gold layer and immediately hit four problems Gold alone couldn't solve:Schema breaks wh...

savlahanish27

Nobody told the analytics team they couldn't query the raw customer table. So, they did.Full names, email addresses, phone numbers - exported to a CSV for "a quick look." No alert fired. No one flagged it. We found out three weeks later during a comp...

savlahanish27

Part 3 of my series on building an enterprise data platform on Databricks is up - this one cover Gold layer design.The short version: Gold isn't just aggregated Silver. Silver maps to your source system. Gold maps to the business questions your consu...

savlahanish27

Part 2 of my series on building an enterprise data platform on Databricks — this one's about Silver.Part 1 covered why we ran two ingestion paths in parallel (GoldenGate CDC + JDBC batch) and kept them as separate bronze tables. If you missed it:http...

savlahanish27

Part 1 of a 5-part series on building an enterprise data platform on Databricks.When migrating a large retail conglomerate's SAP HANA platform to Databricks, we needed both historicalcompleteness and near-real-time freshness from day one.That require...

savlahanish27

The short answer is yes, but with an important condition.Delta Lake's versioning and Change Data Feed make keyless incremental loads possible by tracking changes at the storage layer rather than the data layer. Every write to a Delta table increments...

savlahanish27

Great point on modelling standards. The question we keep coming back to in practice is where the medallion layers draw the line - bronze as raw fidelity, silver as governed truth, gold as business ready. That boundary is a modelling decision, not a t...

savlahanish27

Granting only MODIFY on the table allows the group to insert, update, and delete rows. MODIFY grants the ability to change the contents of a table without altering its schema or metadata - it does not include schema-altering privileges like ALTER or ...

savlahanish27

The core problem you're facing is that Delta Lake doesn't enforce foreign key constraints, so most datagen tools generate each table independently and your joins produce no meaningful overlap.The solution is to generate a shared key pool first - a si...

Databricks Community

User Stats

User Activity

Medallion Architecture Has 3 Layers. We Built 5. Here's Why — Views Layer Design on Databricks

Your data is clean. But who's accessing it, and how? Governing your Lakehouse with Unity Catalog

Gold Layer Design on Databricks — MERGE vs Overwrite, Partitioning, SCD Type 2 from SAP

The Hardest Part of Our SAP Migration Wasn't the Data. It Was Timing

Why We Used Two Bronze Tables Instead of One — And Why It Mattered

Re: Incremental Load Without any keys

Re: AI Is Becoming a Data Problem Before It Becomes a Model Problem

Re: Can change a delta table, but not its schema

Re: Best way to generate fake data using underlying schema