How to Build an OCR Pipeline with LLMs (Playbook)

OCR used to mean turning pixels into a stream of characters and hoping downstream code could make sense of it. Language models changed the pipeline: layout can be reconstructed, errors corrected from context, and content interpreted into structured records. They also introduced a new risk, plausible text that was never on the page. This playbook covers building an OCR pipeline with LLMs that captures the gains and controls the risk, following FISTA's AI enablement practice. Context is in ocr vs llm document extraction and document processing ai.

What does the pipeline do?

Stage	Function
Capture	Scans, photos, faxes, PDFs with image layers
Preprocessing	Deskew, denoise, binarize, crop, orientation, quality scoring
Recognition	OCR engine and/or multimodal model with positions and confidence
Layout recovery	Reading order, blocks, tables, key-value regions
LLM post-processing	Context-based correction; structured interpretation constrained to source
Validation	Rules, cross-checks, reference lookups
Review	Low-confidence regions with image evidence
Evaluation	Character, word, layout, field, and hallucination metrics

Step 1: Define document types and outputs

Specify the document types, the outputs required (plain text, structured fields, tables), accuracy thresholds per output, and the consequence of errors. Define what the review interface must show. This is the specification and the evaluation basis. See how to write acceptance criteria for ai.

Step 2: Preprocess for quality

Score image quality and reject or flag unusable captures early; deskew, denoise, correct orientation, crop borders, and normalize resolution; handle multi-page and mixed-orientation files. Preprocessing determines the ceiling of everything downstream and is where mobile captures need the most work. Computer vision practice is in how to build a computer vision system.

Step 3: Recognize text with positions

Run an OCR engine to obtain characters, words, positions, and confidence, and evaluate multimodal models as an alternative or complement per document type. Positions are essential for layout recovery, review interfaces, and audit evidence. Choose engines per language and script by evaluation, and consider handwriting-capable models where forms are handwritten.

Step 4: Recover layout

Detect blocks, columns, reading order, tables with cell structure, key-value regions, and checkboxes. Tables deserve dedicated handling: cell boundaries, header rows, merged cells, and row continuity across pages. Preserve structure in the output rather than flattening to text. Layout quality is where most extraction errors originate.

Step 5: Apply LLM post-processing, constrained

Use a language model to correct recognition errors from context (domain vocabulary, formats, checksums), to map table headers and normalize values, and to interpret recovered text into the required schema. Constrain it: corrections must be plausible edits of recognized text, every output maps to a source region, missing content is flagged rather than filled, and outputs are structured. Measure hallucination rate explicitly. Concepts are in what is structured output and llm output validation.

Step 6: Validate and route to review

Run rules: formats, checksums, arithmetic consistency, allowed values, reference lookups. Combine recognition confidence, layout confidence, and validation outcomes into a per-field confidence, and route low-confidence regions to reviewers who see the image crop beside the value. Corrections feed evaluation. Review design is in how to build a human review queue.

Step 7: Evaluate at every level

Build a labeled set per document type across quality levels: ground-truth text, layout annotations for tables and reading order, and field values. Measure character and word error rates, table structure accuracy, reading-order accuracy, field-level accuracy, hallucination rate, and review rate. Compare engine and model configurations on the same set and gate changes in CI. Method is in the AI evaluation and testing whitepaper.

Step 8: Deploy and monitor

Deploy as a service consumed by extraction and classification pipelines; monitor quality-score distributions, confidence, review rates, and per-type accuracy on sampled production documents; and detect new layouts and sources. See the AI observability whitepaper.

Worked example: field inspection reports

A utility receives inspection reports as photographed handwritten forms and printed checklists from field technicians. Preprocessing scores image quality and prompts technicians to recapture blurred pages at the point of upload. A handwriting-capable recognition model handles the forms; a standard engine handles printed checklists; layout recovery detects the checklist table and reading order. Language-model post-processing corrects equipment identifiers against the asset register format and maps checklist rows to the schema, constrained to recognized text with every field linked to an image region. Validation checks identifier existence and required fields; low-confidence readings route to a reviewer who sees the crop. Evaluation on labeled reports tracks word error rate by capture quality and field accuracy per checklist item, and the hallucination metric stays near zero because the model cannot introduce identifiers absent from the recognized text.

What does it cost to run?

Cost scales with page volume; OCR engines are cheap, multimodal and language-model steps cost more and are applied selectively by document type and confidence. Value is measured in manual keying avoided and downstream error reduction. Drivers are in computer vision project cost and document ai cost.

What are the common mistakes?

• Skipping preprocessing and blaming the model for blurry scans.
• Flattening tables to text.
• Unconstrained LLM correction that invents content.
• Losing positions, so review and audit have no evidence.
• Evaluating word error rate only, never field accuracy or hallucination.
• One engine for every language, script, and document type.

How FISTA Solutions builds OCR pipelines

FISTA Solutions builds OCR pipelines to this playbook: specification-defined outputs and thresholds, quality-gated preprocessing, engine and model selection per document type by evaluation, dedicated layout and table recovery, constrained language-model post-processing with hallucination measurement, rule validation with targeted review, and multi-level evaluation in CI. The AI enablement practice delivers the pipeline as part of document platforms, AI agents consume its outputs, and forward deployed engineers embed with your operations team. The record behind the work is 150+ projects with 99.9% uptime.

To scope an OCR pipeline, message FISTA on WhatsApp, or read how to build an ai data extraction pipeline for the extraction stage it feeds.