Databricks Data Quality - Agent Proposes, DQX Disposes

KeerthiItigimat — Wed, 23 Sep 2026 02:40:25 GMT

Every serious data platform grows a dead-letter table, and it always becomes a graveyard. Here is how we turned one into a self-healing queue — without letting a language model anywhere near the warehouse.

Every data platform that takes quality seriously ends up with a dead-letter table. You run pre-commit checks, and the rows that fail don’t go into Silver — they go into quarantine. It is the responsible thing to do. It is also, in practice, where data goes to die.

Nobody wants to open the quarantine table. The rows that land there are, by definition, the annoying ones: a code with a stray alphabetic prefix, a timestamp in the wrong format, a duplicate that got emitted twice, a foreign key that doesn’t quite line up because one side trims whitespace and the other doesn’t. Fixing them is fiddly, one-off work, and the backlog only grows.

We built a system that closes the loop. An LLM agent triages quarantined rows and proposes fixes. Those fixes are re-validated deterministically against the exact checks that quarantined the row in the first place. Only the rows that pass are merged back into Silver. Everything else escalates to a human review queue instead of looping forever.

The interesting part isn’t the LLM. It’s the constraints around it.

01
Two design principles

Everything in the system falls out of two rules we decided not to break.

“The agent proposes. DQX disposes.”
PRINCIPLE ONE

The agent never writes to Silver or Gold. Not once. Every agent-curated row is re-run through the same data-quality engine — we use DQX, the Databricks Labs quality framework — that quarantined it. If the “fix” doesn’t actually make the row pass the checks, it doesn’t get in. This keeps a hallucinated correction from silently corrupting the warehouse, and it makes the whole thing auditable: the gate is code, not vibes.

“Documented fixes beat guessed fixes.”
PRINCIPLE TWO

If a rule owner has already written down exactly how a violation should be remediated — “primary-key collision on customer_id: keep the row with the latest updated_at, drop the rest” — then that fix is applied deterministically and the LLM never sees the row. The agent is the fallback for rules nobody has written a playbook for yet. It is not the default path.

That second principle matters more than it sounds. The cheapest, most reliable, most auditable model call is the one you never make.

Figure 1. The loop. A quarantined row is fixed by a documented playbook strategy or, failing that, by the agent — then re-validated against the identical DQX checks before anything touches Silver.

02
The shape of the pipeline

The whole thing runs as a Databricks Workflow. Because a For Each task can only wrap a single task, not a sub-graph, the per-table pipeline is a child job that the parent fans out to — one child run per quarantine table.

Figure 2. The Workflow DAG. Parent discovers and fans out; each child run walks the same seven-stage path. 06 runs on at least one success of reingest or a false gate, so every table produces an audit row either way.

Figure 3. A recreation of the parent job’s run graph in Databricks Workflows — discover, then a For Each fan-out to one child run per quarantine table. Table names and durations are illustrative.

Each stage writes only to a control table — a review queue with one row per quarantined row per pass. Nothing touches Silver until the final merge, and that merge only ever sees rows that have already passed re-validation.

03
Triage

Pull the unprocessed rows from one quarantine table, dedupe to the latest quarantined version per row, and flatten the failure metadata. DQX records failures as an array of structs — {name, message, …} — and triage flattens those names into a plain array<string> of violated rule names, then seeds the review queue with status pending.

It also captures the business key at this moment. Reconstructing a natural key later, after the row has been patched, is guesswork; capturing it at quarantine time is not.

04
Playbook remediation — the deterministic half

This is where documented fixes get applied. Each dataset has a YAML file — not in the codebase, but in a Unity Catalog Volume, so adding or changing a rule needs no deploy:

Entries run in priority order; the first one that matches a row wins. Three things we learned to bake in:

where predicates let one rule route by value. status = 'Duplicate' goes to dedupe; status = 'Invalid' has no entry, so it falls through to the agent. Same rule, different handling, driven by the data.
reject_rows is a terminal state, not a failure. Some rows genuinely can’t be recovered. Dropping them is a decision the rule owner made on purpose, and it is recorded as such — not retried, not escalated, not seen by the agent.
Dedup is configured only here. The agent never deduplicates. Choosing which of two near-identical rows survives is a business call with a documented tie-breaker, not something to infer.

The strategy functions — dedupe_keep_latest, standardize_value, fill_default, reject_rows — are code. A new kind of fix needs a deploy. A new instance of an existing kind is just a file upload. Behavior in code, policy in config: that split is what keeps the playbook maintainable by the people who own the rules rather than the people who own the pipeline.

05
Agent curation — the fallback

Only rows with no playbook match reach the agent. And here is the decision that makes this effective:

“One model call per distinct violation signature. Not per row.”
THE COST MODEL

If 40,000 rows all failed the same check on the same column in the same way, that is one call. The model sees the rule name, the check function, the message, a sample of failing values, and up to five passing examples per flagged column. It returns a single deterministic transform per column, chosen from a fixed vocabulary:

Spark then applies that transform to every row in the signature. The model does not see 40,000 rows, does not write 40,000 fixes, and does not get to invent an expression. It picks lower_trim for a casing problem or regexp_replace to strip a known prefix, and that is the entire surface area. Token cost is independent of quarantine volume.

Transforms can be chained — strip a prefix, then zero-pad — which covers most of the real formatting damage: typos, column shifts, date formats, casing, padding, regex-strippable junk. And we guard against no-ops: if the model proposes left_pad on a string already at the target length, that is not a fix, and it gets downgraded to an escalation.

06
The cross-table probe — no model at all

Referential-integrity checks — a row in table A must match a row in table B — are common, and the LLM is bad at them. So before the agent is involved, a deterministic probe runs: on a sample, normalize the compared column (trim, lower_trim, upper_trim) on both sides of the join and check whether the mismatch clears. If a normalization resolves, say, 95% of the sample, propose that transform. If it doesn’t, escalate.

A surprising fraction of “broken foreign keys” are just one side of a join carrying trailing whitespace.

07
The confidence gate

The agent attaches a confidence score to every fix decision. Between curation and application, a gate checks the minimum confidence across the batch against a threshold. Above it: proceed to auto-apply. Below it: skip straight to audit, and those rows wait for a human.

One low-confidence signature holds back the whole batch for that table. We would rather under-automate than merge a shaky fix.

08
Apply and re-validate

For rows that clear the gate: apply the patch, strip every bookkeeping column, and re-run the row through approved_checks_for(silver_table) — the same rules table that quarantined it. Not a copy of the checks. Not a checks file that might have drifted. The same source. “Re-run the exact checks that quarantined it” is only true if both sides read from one place.

Rows that pass are staged. Rows that fail get retry_count += 1 and go back into the queue. After three failed attempts a row auto-escalates regardless of confidence. Nothing loops forever.

09
Reingest

A dynamic MERGE INTO Silver, keyed on the business key that triage captured, columns intersected with the target schema. Every reingested row is tagged with lineage: agent_curated = true, the confidence score, the run id, the timestamp. Downstream consumers who don’t trust agent-curated data can filter it out. That is their call to make, and we give them the column to make it with.

Figure 4. The review queue — one row per quarantined row per pass. Terminal states in green/red, human work in amber.

Figure 5. Illustrative outcome split for a single run. The playbook clears the bulk; the agent handles a couple hundred; a couple dozen reach a human.

10
Dry runs

Because “an LLM will now edit rows that flow into our warehouse” is a sentence that makes people nervous, there are two ways to see what would happen without committing anything.

A read-only estimate that computes the deterministic half for real — match, apply strategy, re-validate — and reports the agent half as a count only. The model is never called. One row per table into a report table, nothing else.
A full dry run that runs the whole pipeline, calls the agent, writes the queue and the staging table — but holds the final MERGE and instead reports the exact insert/update split against Silver.

Run the first before you turn the system on. Run the second in staging when you want to see precisely what will land.

11
What this bought us

The quarantine table stopped being a graveyard. On the first real run, the large majority of quarantined rows resolved deterministically — dedup and structural rejects that rule owners had already documented. A few hundred went to the agent. A couple dozen genuinely needed a human, and those are the ones a person should be looking at.

The lesson we keep relearning: the LLM is the least important component. The value is in the scaffolding — the deterministic-first routing, the fixed transform vocabulary, the re-validation gate, the retry cap, the confidence threshold, the dry runs. The agent is a fallback that turns “a human triages every weird row” into “a human triages the genuinely ambiguous ones.”

Agent proposes. DQX disposes. Silver stays clean.

Re: Databricks Data Quality - Agent Proposes, DQX Disposes

ivanvyd — Thu, 24 Sep 2026 20:10:16 GMT

Thanks for sharing, @KeerthiItigimat, what a great breakdown. How do you handle rules changing while rows are waiting in quarantine? Does remediation keep the original checks, use the latest approved rules, or run both?

Re: Databricks Data Quality - Agent Proposes, DQX Disposes

ivanvyd — Thu, 24 Sep 2026 20:11:04 GMT

Do you also retain the matching rule definitions alongside DQX's `rule_set_fingerprint`, so a later review can see exactly which checks were applied? Curious.

Re: Databricks Data Quality - Agent Proposes, DQX Disposes

KeerthiItigimat — Thu, 24 Sep 2026 22:33:02 GMT

Thank you @ivanvyd . The remediation playbooks lets you decide how you want to handle this scenario. By default, it will try to apply the same rule that quarantined. If you want to apply the latest version of the rule, you have an option to override that in playbooks.

Re: Databricks Data Quality - Agent Proposes, DQX Disposes

KeerthiItigimat — Thu, 24 Sep 2026 22:51:21 GMT

The rule definition is available as part of the per row errors or warnings column. Additional metadta is stored along side rule fingerprint and is available for a later review.

topic Re: Databricks Data Quality - Agent Proposes, DQX Disposes in Community Articles

Databricks Data Quality - Agent Proposes, DQX Disposes

01Two design principles

02The shape of the pipeline

03Triage

04Playbook remediation — the deterministic half

05Agent curation — the fallback

06The cross-table probe — no model at all

07The confidence gate

08Apply and re-validate

09Reingest

10Dry runs

11What this bought us

Re: Databricks Data Quality - Agent Proposes, DQX Disposes

Re: Databricks Data Quality - Agent Proposes, DQX Disposes

Re: Databricks Data Quality - Agent Proposes, DQX Disposes

Re: Databricks Data Quality - Agent Proposes, DQX Disposes

01
Two design principles

02
The shape of the pipeline

03
Triage

04
Playbook remediation — the deterministic half

05
Agent curation — the fallback

06
The cross-table probe — no model at all

07
The confidence gate

08
Apply and re-validate

09
Reingest

10
Dry runs

11
What this bought us