Designing an Effective "Quarantine" Pattern for Failed DLT Expectations

[Khasim_1]

 Hi everyone,

I’m building out a production Medallion Architecture using Delta Live Tables (DLT), and I’m refining our strategy for handling data that fails Expectations.

While expect_or_drop and expect_or_fail are straightforward, I’m struggling to find the "Gold Standard" for a Quarantine pattern—where we capture the bad data for manual review without stopping the entire pipeline or losing the original lineage.

1. Are you creating a separate "Invalid" DLT table for every Silver table, or have you found a way to centralize all "Expectation Failures" into a single governed view?
2. How do you handle the "re-processing" lifecycle? Once a data steward corrects the source data, what is your preferred method for re-injecting that record into the DLT flow without triggering a full refresh?
3. Have you found that too many "Expectations" significantly impact the performance of DLT at the petabyte scale, or is the overhead negligible?

I’d love to see how other teams are closing the loop on data quality rather than just dropping records into the void!

[Islam_hoti · Accepted solution]

Hi,

Good questions. A few things that have worked.

On the quarantine pattern, the version most people land on first is two flows from the same source, one with the expectations and one with the inverted condition. It works, but it reads the source twice, which at your scale is the expensive part. The cheaper shape is a single silver table that keeps every row and carries the verdict as data: a boolean is_quarantined column plus a failed_rules array built from the same rule dictionary you feed to expect_all. Then you put two views on top, one filtered to valid and one to invalid, and downstream reads the valid view. One scan, full lineage preserved, and the bad rows sit next to the good ones rather than in a parallel universe.

For centralising across many silver tables, the blocker is schema heterogeneity. The approach that avoids a table per table is one central quarantine table with a fixed envelope schema: source_table, event_time, failed_rules as an array of strings, and the original row serialised into a single payload column. With VARIANT you get to keep it queryable rather than as opaque JSON. Every silver table writes into that one table via its own flow, and stewards work from one place.

On reprocessing, the piece you are probably missing is that a streaming table can have more than one flow writing into it. So alongside your main flow from bronze, define a second append flow sourced from a corrections table that your stewards write to. Corrected records land in the target through a normal append, no full refresh, no touching the main flow's checkpoint. If corrections need to supersede earlier versions of the same key rather than sit beside them, route them through AUTO CDC instead of a plain append so the upsert semantics do the work.

On performance, expectations themselves are cheap. They compile into the same plan and are evaluated in the pass you were already doing, so a long rule list costs far less than people expect. The overhead that actually shows up is structural rather than per rule: a second read of the source for quarantine, and any rule that reaches for a regex over a wide string column or a non trivial UDF. I would not prune rules for performance reasons before checking whether the pipeline is doing two scans where one would do.

One naming note in case you hit it in the docs: DLT is now Lakeflow Declarative Pipelines and APPLY CHANGES INTO is now AUTO CDC INTO. The older names still work, but the current documentation uses the new ones.