Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram

TriambakR — Thu, 16 Jul 2026 15:43:26 GMT

Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram

Every Lakehouse conversation eventually shows the same three boxes: Bronze, Silver, Gold. It's a great mental model — but on a real enterprise migration, the diagram is the easy part. The decisions that actually determine whether the architecture holds up are the ones that never make it onto the slide.

I recently worked on a migration of a large enterprise analytics estate — legacy MPP warehouse (Greenplum) plus a brittle Talend ETL stack — onto a Databricks Lakehouse on AWS. Here are the four decisions that mattered most, written up in case they save someone else a design cycle.

1. Bronze must stay "boringly faithful" — resist the urge to clean early

The biggest temptation on any migration is to fix obviously bad data on the way into Bronze. Don't. If Bronze applies any business logic, you've coupled your raw layer to a specific interpretation of correctness — and the moment that interpretation changes (it will), you have no clean copy to reprocess from.

What we standardized on:

Source structure preserved as-is, no type coercion, no filtering
Four audit/lineage columns bolted on top: _source_system, _ingested_at, _batch_id, _file_name
Schema evolution enabled (mergeSchema) so upstream column changes don't break the load

(spark.readStream
  .format("cloudFiles")
  .option("cloudFiles.format", "parquet")
  .option("cloudFiles.schemaEvolutionMode", "addNewColumns")
  .option("cloudFiles.schemaLocation", schema_path)
  .load(landing_path)
  .withColumn("_source_system", lit(source_name))
  .withColumn("_ingested_at", current_timestamp())
  .withColumn("_batch_id", lit(batch_id))
  .writeStream
  .option("checkpointLocation", checkpoint_path)
  .trigger(availableNow=True)
  .toTable("bronze.raw_table"))

The payoff: when a downstream bug is discovered in Silver logic three weeks later, you replay Bronze and reprocess — you don't re-extract from a source system that may have already moved on.

2. Gold is a governance boundary, not a storage boundary

This is the one that surprises people. "Gold" doesn't have to mean "physically materialized table." On this project, most of Gold was implemented as governed views over Silver, not copies:

No physical duplication → no second copy to keep in sync, no extra storage cost
Always reflects the latest Silver state
Simpler to govern — one copy of the data, one set of grants

We only materialized a Gold table (with OPTIMIZE/ZORDER applied) when a specific consumer had a concurrency or latency SLA that a view genuinely couldn't meet. That was the exception, not the default.

The decision rule we used: default to a curated view; materialize only when you can point to a measured concurrency/latency requirement that justifies the extra storage and refresh complexity.

3. Silver is where the real engineering effort lives — and MERGE is the mechanism

Bronze is fidelity. Gold is a contract. Silver is where you actually earn your keep:

Standardize types and naming
Deduplicate on natural/business keys
Enforce data quality rules (nulls, referential integrity, ranges)
Join and enrich against reference/dimension data
Apply domain-specific business rules

The mechanism that makes all of this idempotent is Delta's MERGE:

MERGE INTO silver.entity_table AS target
USING staged_updates AS source
ON target.business_key = source.business_key
   AND target.effective_date = source.effective_date
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

One subtlety worth calling out: if your domain has legitimately repeating keys (e.g., effective-dated HR or reference data), your dedup/merge key must include the version/effective-date column. Keying only on the natural identifier will silently collapse valid history into a single row — a bug that's easy to introduce and painful to find after the fact.

Takeaways

Keep Bronze dumb and durable; put correctness in Silver where it can be revised
Don't assume Gold means "another copy" — default to views, materialize only with a measured reason
MERGE-based up-serts are what make reprocessing safe; get your key design right, especially for versioned/effective-dated data

Curious how others have handled the Gold view-vs-table decision on their own Lakehouse migrations — would love to compare notes in the replies.

Re: Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram

sasidharan_gs — Thu, 16 Jul 2026 16:24:14 GMT

I want to thank you for re-kindling my passion for data. This was a great overview of the medallion architecture!

Re: Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram

ashna_jindal — Thu, 16 Jul 2026 16:32:04 GMT

Insightful and well articulated

Re: Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram

Sbm_dracarys — Thu, 16 Jul 2026 17:11:01 GMT

Great work, man! Even though I don't know much about this field, your article made me curious and motivated me to read more about it. Thanks for sharing such valuable insights.keep posting

topic Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram in Community Articles

Medallion Architecture in Practice: The Design Decisions Nobody Puts in the Diagram