Document AI is a pipeline, not a model
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a month ago - last edited a month ago
When a financial document needs to be approved or rejected, there usually isn't one model making the decision.
In the check / payment-slip systems I worked on, the pipeline looked more like this:
1. Multimodal models + OCR to extract fields
2. YOLO to detect regions and crop signatures
3. Fine-tuned signature verification models, outperforming generic models on bank-specific documents
4. A business-rule engine for decisions the model should not be making
Some lessons were particularly clear:
- A model can read the amount. A rule should verify that the amount in numbers matches the amount in words.
- A model can detect a signature. A specialized model can verify whether it matches an authorized signer.
- Front/back consistency, required fields, thresholds, and institution-specific requirements belong in deterministic logic.
- For critical fields, specialized crops + specialized models can outperform an end-to-end VLM.
And when document formats differ significantly between institutions, having separate rule branches isn't necessarily technical debt. Sometimes it's simply the correct representation of the business.
The interesting part starts when you scale this.
On Databricks, I would treat document understanding, model serving, monitoring, and business rules as one governed production path?not as a notebook demo connected to a webhook.
If you're designing IDP on Databricks / Mosaic AI, I'm interested in how you're drawing the line between what the model should decide and what should remain deterministic.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
On your closing question, the rule I'd use for the line: anything with a checkable ground truth, either inside the document or in a system of record, goes to deterministic logic, and the model only proposes. Amount in figures vs words, date validity, account number checksums, required fields. The model extracts, a rule verifies, and a mismatch routes to a human instead of the model arguing its way out of it.
The model keeps what is genuinely ambiguous and has no oracle, like messy handwriting or whether a signature plausibly matches. For those I'd have it return a confidence and set the thresholds per field by risk, since a wrong amount should cost more review than a wrong memo line.
On the Databricks side, what makes this governable is writing each field's extracted value, confidence, the rule that fired and the final decision to one Delta table. Then every rule-vs-model disagreement becomes a labelled example you can use to retune thresholds or fine-tune the signature model, and an auditor can replay any decision later.
