Why We Used Delta Live Tables for Exactly One Pipeline (and Not for Silver/Gold)
"Why isn't DLT used everywhere?" is one of the first questions I get when reviewing this architecture with other engineers — so I figured it was worth writing up properly, because the answer is a genuinely useful design pattern, not a gap.
The setup
On a recent enterprise Lakehouse program, one domain's ingestion looked like this: a report-based HR data feed (pulled via a managed connector, not a live transactional API), landing into Bronze, then flowing through standard Silver/Gold transformation. For that HR domain, we used a Delta Live Tables pipeline scoped only to the Bronze ingestion step. Silver and Gold for that same domain were implemented as standard Databricks Workflows (multi-task jobs), deployed and version-controlled through Databricks Asset Bundles (DAB) — the same CI/CD framework used for every other asset in that project.
Worth flagging since it's a separate detail people sometimes assume travels with this pattern: this particular project did not use Apache Airflow at all. Airflow was the orchestration standard on a separate, earlier platform migration (the broader legacy-warehouse-to-Lakehouse program), not on this HR pipeline. Two different engagements, two different orchestration footprints — DAB-deployed Databricks Workflows here, Airflow there.
Where DLT earned its keep
- A single, well-bounded ingestion contract. One report source, one target Bronze table, one schema to reason about. DLT's declarative model (you describe the target table, DLT resolves the dependency graph) pays off most when the graph is small and stable.
- Built-in data quality expectations at the point of landing. EXPECT constraints let us fail/drop/warn on malformed records right at ingestion, without hand-rolling validation code.
- Managed compute and autoscaling for the ingest job, with less orchestration code to own for that one pipeline.
import dlt
@dlt.table(
comment="Raw HR report data landed from managed connector"
)
@dlt.expect_or_drop("valid_employee_id", "employee_id IS NOT NULL")
@dlt.expect("valid_effective_date", "effective_date IS NOT NULL")
def bronze_hr_raw():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "parquet")
.load(landing_path)
)Why we didn't extend it to Silver/Gold
- Silver/Gold dependency chains here were straightforward, explicit task sequences — standardize, dedupe, join to reference data, publish — which a regular multi-task Databricks Workflow expresses cleanly without needing DLT's automatic dependency resolution.
- Deployment consistency mattered more than pipeline-engine consistency. Everything in this project — jobs, cluster policies, Unity Catalog grants — was defined and promoted through the same DAB bundle across dev/test/prod. Keeping Silver/Gold as plain Workflow tasks meant one deployment pattern for the whole project; the DLT pipeline (defined as its own resource type within the bundle) was the deliberate exception, scoped to where it earned its complexity.
- Built-in expectations mattered most at the ingestion boundary, where malformed source data first enters the platform. Once data reached Silver, quality rules were more naturally expressed as explicit SQL/PySpark assertions inside a Workflow task than as a second DLT pipeline stitched onto the first.
A platform constraint worth knowing before you commit to DLT
One trade-off that doesn't show up until you're already committed: DLT pipelines don't let you pin a specific Databricks Runtime version. DLT pipelines run on a Databricks-managed release channel (Current or Preview) that Databricks updates on its own rolling schedule — you don't select a DBR version the way you do for a regular job or all-purpose cluster.
That matters if your organization's compliance or stability baseline pins workloads to a certified LTS release (say, 13.3 LTS) across the workspace. Regular Databricks Workflows and clusters can honor that pin. A DLT pipeline can't — it's on its own channel regardless of what the rest of your fleet is running. If platform/compliance stakeholders expect every workload to sit on the same certified runtime, DLT needs to be called out as an explicit, approved exception up front, not discovered as a surprise during a security review. Worth surfacing this early in any conversation about adopting DLT more broadly on a platform with strict runtime governance.
The general principle
Consistency across a project is about a shared deployment and governance model — one bundle, one set of per-target configs, one promotion path — not about forcing every pipeline through the same engine. DLT was the right-sized tool for one bounded ingestion contract with a real data-quality need at the landing point. Standard DAB-deployed Workflows handled the rest, and Airflow (on a separate program entirely) handled a different platform's cross-domain orchestration needs.
If you're deciding whether to reach for DLT on a pipeline, the questions I'd actually ask are:
- Is this ingestion contract genuinely self-contained, or does it eventually need to join against things owned by other teams/pipelines?
- Do I need declarative expectations at the landing point specifically, or is validation just as easily done as an explicit step in an existing job?
- Does my platform have a pinned-runtime compliance requirement that DLT's managed channel would violate?
If the answer to #3 is yes, get that conversation started before you build, not after.
Happy to dig into specifics — what's driven your own DLT-vs-Workflow decisions, and has the runtime-pinning constraint come up for anyone else?
