Document AI
How to evaluate document extraction before you automate
Field-level precision and recall, confidence calibration and straight-through thresholds: a practical method for judging document AI on your own files.
2 min readLast reviewed
Vendors and models are often compared on headline accuracy figures. Those figures rarely transfer to your documents, your fields and your tolerance for error. Before automating a document process, measure extraction on a labelled sample of your own files. The method below is simple, repeatable and makes the business decision explicit.
1. Define the fields precisely
Write a definition for every target field: what counts as the value, its format, and what to record when it is absent. “Invoice date” sounds obvious until you meet documents with an issue date, a tax point date and a delivery date. Disagreements between human labellers usually reveal definitions that need tightening.
2. Build a representative sample
Sample documents in proportion to real volume, then deliberately add the difficult cases: poor scans, unusual layouts, multi-page tables, handwriting. Keep a held-out portion that is not used for tuning prompts or rules, so the final measurement is honest.
3. Measure per field, not per document
Document-level accuracy hides which fields are weak. For each field calculate precision (of the values extracted, how many were correct) and recall (of the values present, how many were found). A field can have high precision and low recall, which is often acceptable, or the reverse, which usually is not.
| Field | Precision | Recall | Business impact of error |
|---|---|---|---|
| Supplier tax ID | measured | measured | High: posting to wrong supplier |
| Invoice total | measured | measured | High: payment amount |
| PO number | measured | measured | Medium: matching delay |
| Line item descriptions | measured | measured | Low: reporting only |
4. Check whether confidence means anything
Automation decisions depend on confidence scores, so test whether they are calibrated. Group extractions by confidence band and check accuracy in each band. If 95%-confidence extractions are only right 80% of the time, thresholds based on them will let errors through.
5. Add validation before deciding thresholds
Business rules catch many errors that models miss: line items that do not sum to the total, dates in the future, tax IDs that fail a checksum, purchase orders that do not exist. Measure accuracy after validation, because that is what the downstream system will see.
6. Set the straight-through threshold as a business decision
Plot, for different confidence thresholds, the share of documents that would flow through without review against the error rate in that flow. Finance, operations and risk owners then choose the trade-off explicitly. Revisit the decision as reviewer corrections accumulate and the model or rules improve.