How to Build a Document Classification System (Playbook)

Document classification looks like a solved problem until its errors surface three steps downstream: an invoice processed as a statement, a legal notice filed as correspondence, a claim form routed to the wrong pipeline. Because classification sits upstream of extraction and routing, its accuracy per class determines everything after it. This playbook covers building a document classification system that holds up, following FISTA's AI enablement practice. Context is in document processing ai and how to build a document ai system.

What does the system do?

Stage	Function
Intake	Receive files; split multi-document files; deduplicate
Preprocessing	OCR and text extraction; layout features; metadata
Classification	Class or classes with confidence
Routing	To extraction pipelines, queues, or archives by class
Human review	Low-confidence and ambiguous documents
Feedback	Reviewer labels to the labeled set
Monitoring	Per-class accuracy, distribution drift, new types

Step 1: Design the taxonomy with consumers

Work with the teams that consume classes (extraction pipelines, routing rules, archival policies) to define classes that are distinguishable, useful, and complete, with an explicit "unknown" class. Decide whether multi-label is needed. Document definitions and examples per class; the taxonomy is the specification and is versioned. See how to write an ai spec.

Step 2: Assemble labeled data

Collect representative documents per class across sources, layouts, and quality levels, including hard negatives and look-alikes. Label with guidelines and measure inter-annotator agreement. Reserve a held-out set stratified by class. Data practice is in what is data labeling and ai training data checklist.

Step 3: Preprocess for signal

Split multi-document files at detected boundaries; apply OCR where needed; extract text, layout features (headers, tables, logos, form structure), and metadata (source, filename, sender). Layout signals often distinguish classes text alone cannot. Preprocessing patterns are in how to build an ocr pipeline with llms.

Step 4: Choose the classification approach by evaluation

Evaluate candidates on the held-out set: a language model prompted with the taxonomy definitions and examples, producing structured output with confidence; a trained classifier on text and layout features; and a hybrid where a trained classifier handles high-volume stable classes and the language model handles low-confidence or rare cases. Compare per-class accuracy, latency, and cost. Concepts are in prompt engineering vs fine-tuning and when to build a custom ai model.

Step 5: Set confidence thresholds per class

Derive thresholds per class from evaluation, weighted by downstream error cost: classes that trigger regulated or expensive processing get higher thresholds. Below-threshold documents route to human review with candidate classes shown. Calibrate confidence so thresholds mean what they say. See what is structured output.

Step 6: Design human review

Reviewers see the document, candidate classes with confidence, and taxonomy definitions, and assign the class with one click or flag a new type. Their labels feed the labeled set and reveal taxonomy gaps. Queue design is in how to build a human review queue.

Step 7: Evaluate per class

Report accuracy, precision, and recall per class on the held-out set, confusion patterns between classes, unknown-class handling, and multi-label correctness where applicable. Gate taxonomy, prompt, or model changes on the suite in CI. Method is in the AI evaluation and testing whitepaper.

Step 8: Deploy and monitor

Deploy behind an API used by intake pipelines; monitor class distribution, confidence distribution, review rates, and reviewer disagreement; detect new document types from review flags; and manage taxonomy changes through versioning with evaluation. Drift concepts are in what is model drift and the AI observability whitepaper.

Worked example: a mortgage operations intake

A mortgage servicer receives borrower and third-party documents by mail scan, email, and portal: pay stubs, bank statements, tax forms, insurance declarations, hardship letters, legal notices, and correspondence. The taxonomy is designed with operations and legal, with legal notices given the highest threshold because misfiling them carries regulatory risk. Multi-page scans are split by detected boundaries. A trained classifier on text and layout handles the high-volume financial documents; a language model with taxonomy definitions handles hardship letters and correspondence where language matters; anything below threshold or flagged as unknown goes to review. Per-class evaluation shows strong performance on financial documents and initial confusion between two letter types, resolved by sharpening definitions and adding examples. Monitoring later detects a new state-specific disclosure form appearing in the unknown class, which is added through change control with its own labeled examples.

What does it cost to run?

Cost scales with document volume; trained classifiers are very cheap per document, and language-model classification is reserved for the cases that need it. Value is measured in downstream error reduction and manual sorting time. Drivers are in document ai cost.

What are the common mistakes?

• Taxonomies with overlapping classes nobody can distinguish.
• Optimizing average accuracy while a rare, critical class is misrouted.
• No unknown class, so new document types are forced into wrong ones.
• Ignoring multi-document files.
• One approach for all classes regardless of volume and stability.
• No monitoring of class drift.

How does classification connect to downstream pipelines?

Each class maps to a downstream handler: an extraction schema, a routing rule, an archival policy, or a review queue. Publish the class as structured metadata with confidence so handlers can apply their own thresholds, and version the mapping alongside the taxonomy so a class change never silently breaks a pipeline.

How do you handle new document types?

Route anything the classifier cannot place confidently to a holding queue, review it weekly, and decide whether it is noise, a variant of an existing class, or a new class that needs labels and a taxonomy change. Adding a class means new labeled examples, an evaluation update, and a communication to the teams that consume the classes, so treat it as a controlled change rather than a quick edit.

How FISTA Solutions builds document classification systems

FISTA Solutions builds document classification systems to this playbook: consumer-designed taxonomies, labeled data with agreement measurement, layout-aware preprocessing, approach selection by per-class evaluation, calibrated per-class thresholds, human review that grows the labeled set, CI-gated evaluation, and drift monitoring. The AI enablement practice delivers the system as part of document platforms, AI agents consume its outputs in workflows, and forward deployed engineers embed with your operations team to design the taxonomy. The record behind the work is 150+ projects with 99.9% uptime.

To scope a document classification system, message FISTA on WhatsApp, or read how to build an ai data extraction pipeline for the downstream stage.