DQX - Everything The Docs Don’t Say Yet

KeerthiItigimat — Sat, 19 Sep 2026 06:04:43 GMT

DQX Studio already has its own documentation for the rule editor, the approval workflow, the scheduler. This is what running it against real pipelines, real approvers, and a growing quarantine table made us add on top of it — and around it. It'll be out of date the next time we ship something, and that's fine.

DQX gives you check functions and an engine. DQX Studio — the UI and control plane we run on top of it — gives you a place to author, approve, schedule, and review those checks without writing a notebook. Studio itself is documented elsewhere; this isn’t that document.

This is the list of things that turned out to be missing once real people, not just a pipeline, depended on it. An analyst who wants to see the data before writing a filter. An approver who wants to know why a rule looks the way it does, six weeks later. A scheduler that has to decide, unattended, whether “the last hour” means anything for a given table. A quarantine table that needs an owner other than whoever happens to open it.

What follows, roughly in the order we hit them.

Figure 1. Three layers, one API contract. Studio's own docs cover the middle layer; this post is the outer one.

01
Preview the data before you write anything

Every rule starts with someone looking at the table. The preview panel in the rule editor runs a live sample against Unity Catalog under the caller’s own identity — on-behalf-of, so row filters and column masks apply the same way they would to any other query that person runs — with two things layered on:

A column picker to narrow a wide table down to what’s relevant before you start authoring a check against it.
A natural-language filter box. Type “orders from the last week with a null customer_id” and it’s handed to a Foundation Model with the table’s real column list and a hard constraint — use only these columns, always cap the row count, default to a plain SELECT * if the request doesn’t resolve to anything sensible. The model returns SQL, not an answer; if the call fails for any reason, the preview falls back to an ordinary unfiltered sample rather than erroring out.

Small feature, but it collapses the gap between “I can describe the bad rows” and “I can write a SQL predicate for them” — which is most of the distance between a domain expert and a rule.

02
Two ways to run a check, on purpose

Databricks Apps don’t run Spark. So every dry run has to leave the app process one way or another — and we deliberately built two different exits for it, because “fast” and “exactly how it’ll run in production” are different needs.

Run Locally executes against a small sample through a local, serverless Spark session — a few seconds, no job to wait on. Good for iterating on a rule while you’re still writing it.
Run as Workflow submits the same check to the actual task-runner job — the identical path a schedule will use in production, including the real sample size and Row Scope. Slower, but it’s not a simulation of the production path, it is the production path.

An author uses the first ten times while shaping a rule and the second once, right before submitting it for approval, to confirm it survives contact with the real execution environment.

03
Every rule remembers how it was made

A rule authored six weeks ago, by someone else, using a phrase like “flag orders where the region code doesn’t match our list” — an approver reviewing that later shouldn’t have to reverse-engineer intent from a YAML blob. Two things carry that context forward: a full audit trail (dq_quality_rules_history) recording every version, every status transition, and who made it, not just the current state; and, for AI-generated checks specifically, the original natural-language instruction travels with the rule as metadata rather than being discarded once the check function comes back. The prompt that produced a rule is part of the rule’s own record, not a one-time scratch input.

04
Global rules, for the check that isn’t about one table

Most checks are bound to a table — orders.customer_id is not null. Some aren’t really about a table at all; they’re about a shape: “any column named email should look like an email,” regardless of which of forty tables it shows up in. Global rules are authored once, table-independent, and apply wherever a matching column exists — without re-authoring the same check forty times or drifting slightly different each time someone copies it.

05
Cross-table checks, and what production found

DQX’s sql_query check is the right tool for anything that isn’t row-local — a duplicate key, a referential mismatch between two tables. It also has a real contract: an explicit condition column, and merge_columns to join violations back to specific input rows. An author writing “give me the rows that violate this” doesn’t know that contract, and shouldn’t have to.

what an author writes vs. what DQX's engine requires

The editor derives that wrap on save, and only trusts a column as a merge key once it’s checked against the target table’s real schema — a renamed join column looks safe in the query text and fails at execution time, so schema-text alone isn’t enough. A related fix went the other way: for a query that doesn’t actually depend on its home table’s rows (its own independent join), the engine was pre-sampling that input before the check ran, which could silently drop real violations whose key wasn’t in the sampled slice. The fix routes that shape through the same fast path a true cross-table check already used, instead of teaching the row-level engine a special case.

Underneath both: Unity Catalog metadata reaching a fresh serverless job can lag well behind a SQL warehouse seeing the same object, so the read path needed a retry budget sized off actual observed timelines, not a guess.

06
Alerting that isn’t just “check the app”

A validation result nobody sees is worth nothing. Two channels out:

Microsoft Teams webhooks, scoped per channel by trigger — all runs, scheduled-only, or manual-only — so a channel used for production monitoring doesn’t fill up with every dry run an author fires off while iterating.
Site24x7, or any uptime monitor, via a pair of read-only status endpoints — latest result by table, or by a specific run id — that return a plain 200 on a clean run and a 503 the moment error rows show up. External monitoring infrastructure gets to treat data quality the same way it treats an HTTP health check, without knowing anything about DQX.

07
Schedules that know what they’re sampling

“Validate the last hour” only means something if the scheduler knows which column is time, and on a table it’s never seen before, that’s a real guess. It tries typed timestamp/date columns first, then a name-priority list, and surfaces the candidates as editable tags — a schedule owner can pin the right one, remove one that’s wrong for their table, or leave it on automatic. Row Scope pairs with tag-based scope filtering, so one schedule can target “every approved rule labeled pii” across every table that has one, on either the full table or a time-windowed sample, instead of one schedule per table.

08
Closing the loop, not just quarantining

Everything above eventually produces failing rows, and Studio’s side of what happens to them is deliberately small: a per-dataset YAML runbook, edited or uploaded in the app, written straight to a Unity Catalog Volume. The app checks that it’s valid YAML and writes it — it doesn’t parse rule names, strategies, or parameters, so a new remediation strategy is a change to the consuming pipeline, never to Studio.

What reads those runbooks — an agentic re-ingestion system that tries the documented fix first and falls back to an LLM only for what nobody has written a playbook for — is its own build, written up separately:

Agent Proposes, DQX Disposes →

09
dqx-validator — the write side of the dead-letter pattern

Everything so far assumes rows are already in a Silver table by the time DQX sees them. dqx-validator is the piece that runs earlier — a standalone library an ingestion pipeline calls directly, mid-flight, to apply Studio’s own approved rules to a DataFrame before it lands, and quarantine whatever fails.

It loads status = 'approved' checks from the same rules tables Studio’s approval workflow writes to, resolved per-environment from a small table config. The loader is exposed as a single function, approved_checks_for(table_fqns), and the agentic re-ingestion side calls the exact same function — so the checks that quarantined a row and the checks that later re-validate a proposed fix are guaranteed to be the same object, not two copies that can drift.

Splitting the DataFrame is one call to DQX’s own apply_checks_by_metadata_and_split — no custom validation logic, just the public engine. The quarantine row carries the input schema unchanged plus bookkeeping columns (_error, _warning, _generated_at, _data_source, _rule_sources, _is_cross_table), and one more that matters more than it looks: _row_id, a hash of whichever business-key columns the caller names. That’s the field the entire review queue on the read side keys every quarantined row on. Name the wrong columns, or none, and the hash falls back to every input column — which still works, but logs a warning, because at that point two rows differing only in a bookkeeping timestamp look like two different violations instead of one.

Figure 2. Items 08 and 09 are two ends of the same loop — dqx-validator writes to quarantine, the remediation pipeline reads it back out, and both sides resolve rules the same way.

10
Ask it directly, instead of building another chart

Studio has an Insights dashboard for the questions we anticipated. It doesn’t have one for the question someone asks once, in passing — “which tables had a spike in errors last Tuesday” isn’t worth a dashboard tile, but it’s a completely reasonable thing to want to know. A Genie space scoped to Studio’s own schema — run history, quarantine records, metrics — turns that into a conversation instead of a feature request: ask in plain language, get an answer grounded in the real tables, no ticket filed to add a chart nobody will look at again.

11
What all of this has in common

None of the ten items above touch the DQX library. Every one of them is a caller of its public API, the same way a notebook is — the editor, the validator library, the agentic pipeline, all of it sits beside DQX rather than inside a fork of it. That was never an accident; it’s the only way any of this stays upgradeable as the library keeps moving.

Studio is documented. This is everything the documentation doesn’t say yet.

topic DQX - Everything The Docs Don’t Say Yet in Community Articles

DQX - Everything The Docs Don’t Say Yet

01Preview the data before you write anything

02Two ways to run a check, on purpose

03Every rule remembers how it was made

04Global rules, for the check that isn’t about one table

05Cross-table checks, and what production found

06Alerting that isn’t just “check the app”

07Schedules that know what they’re sampling

08Closing the loop, not just quarantining

09dqx-validator — the write side of the dead-letter pattern

10Ask it directly, instead of building another chart

11What all of this has in common