Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram
Every Lakehouse conversation eventually shows the same three boxes: Bronze, Silver, Gold. It's a great mental model — but on a real enterprise migration, the diagram is the easy part. The decisions that actually determine whether the architecture holds up are the ones that never make it onto the slide.
I recently worked on a migration of a large enterprise analytics estate — legacy MPP warehouse (Greenplum) plus a brittle Talend ETL stack — onto a Databricks Lakehouse on AWS. Here are the four decisions that mattered most, written up in case they save someone else a design cycle.
1. Bronze must stay "boringly faithful" — resist the urge to clean early
The biggest temptation on any migration is to fix obviously bad data on the way into Bronze. Don't. If Bronze applies any business logic, you've coupled your raw layer to a specific interpretation of correctness — and the moment that interpretation changes (it will), you have no clean copy to reprocess from.
What we standardized on:
- Source structure preserved as-is, no type coercion, no filtering
- Four audit/lineage columns bolted on top: _source_system, _ingested_at, _batch_id, _file_name
- Schema evolution enabled (mergeSchema) so upstream column changes don't break the load
(spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "parquet")
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")
.option("cloudFiles.schemaLocation", schema_path)
.load(landing_path)
.withColumn("_source_system", lit(source_name))
.withColumn("_ingested_at", current_timestamp())
.withColumn("_batch_id", lit(batch_id))
.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(availableNow=True)
.toTable("bronze.raw_table"))The payoff: when a downstream bug is discovered in Silver logic three weeks later, you replay Bronze and reprocess — you don't re-extract from a source system that may have already moved on.
2. Gold is a governance boundary, not a storage boundary
This is the one that surprises people. "Gold" doesn't have to mean "physically materialized table." On this project, most of Gold was implemented as governed views over Silver, not copies:
- No physical duplication → no second copy to keep in sync, no extra storage cost
- Always reflects the latest Silver state
- Simpler to govern — one copy of the data, one set of grants
We only materialized a Gold table (with OPTIMIZE/ZORDER applied) when a specific consumer had a concurrency or latency SLA that a view genuinely couldn't meet. That was the exception, not the default.
The decision rule we used: default to a curated view; materialize only when you can point to a measured concurrency/latency requirement that justifies the extra storage and refresh complexity.
3. Silver is where the real engineering effort lives — and MERGE is the mechanism
Bronze is fidelity. Gold is a contract. Silver is where you actually earn your keep:
- Standardize types and naming
- Deduplicate on natural/business keys
- Enforce data quality rules (nulls, referential integrity, ranges)
- Join and enrich against reference/dimension data
- Apply domain-specific business rules
The mechanism that makes all of this idempotent is Delta's MERGE:
MERGE INTO silver.entity_table AS target
USING staged_updates AS source
ON target.business_key = source.business_key
AND target.effective_date = source.effective_date
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
One subtlety worth calling out: if your domain has legitimately repeating keys (e.g., effective-dated HR or reference data), your dedup/merge key must include the version/effective-date column. Keying only on the natural identifier will silently collapse valid history into a single row — a bug that's easy to introduce and painful to find after the fact.
Takeaways
- Keep Bronze dumb and durable; put correctness in Silver where it can be revised
- Don't assume Gold means "another copy" — default to views, materialize only with a measured reason
- MERGE-based up-serts are what make reprocessing safe; get your key design right, especially for versioned/effective-dated data
Curious how others have handled the Gold view-vs-table decision on their own Lakehouse migrations — would love to compare notes in the replies.
