How to Build a Contract Analysis System (Playbook)

Contract review AI succeeds or fails on one question: is there a written playbook? Systems asked to "find risks" without one produce unexplainable flags that lawyers ignore. Systems that review against explicit positions, and show the evidence, become trusted tools. This playbook covers building a contract analysis system that way, following FISTA's AI agents practice and the controls in the AI for legal operations whitepaper. The functional overview is in ai contract review.

What does the system do?

Function	Output	Reviewer
Ingestion and parsing	Structured contract text with sections and definitions	System
Clause extraction and classification	Clauses labeled by type with locations	System, sampled
Deviation detection	Flags with contract language, playbook position, severity	Lawyer
Key-term extraction	Parties, dates, values, renewal terms, governing law	Lawyer verifies key fields
Summary and routing	Review summary; routing by severity and matter	Lawyer
Audit	Record of flags, decisions, and edits	Compliance

Step 1: Write the playbook as a specification

Work with the legal team to encode, for each clause type in scope: the preferred position, acceptable fallbacks, unacceptable terms, severity for deviations, and any hard rules (for example, any limitation-of-liability deviation is always highest severity). Define what happens with unrecognized clause types: they are surfaced, not classified. Define prohibited behavior: no drafting of alternative language unless requested, no processing of privileged-marked documents. This document is the specification and the source of the evaluation set. Method is in the spec-driven development for AI whitepaper.

Step 2: Build structure-preserving ingestion

Contracts arrive as PDFs, scanned images, and word-processing files. Parsing must preserve section hierarchy, numbering, cross-references, definitions, and tables, because clause meaning depends on them. Use layout-aware parsing with OCR where needed and capture metadata: counterparty, contract type, matter, date, jurisdiction. Attach matter-level access control at ingestion. Parsing patterns are in how to build an ocr pipeline with llms.

Step 3: Extract and classify clauses

Segment the contract into clauses and classify each against the playbook's clause taxonomy. Combine retrieval of playbook definitions and examples with model classification, and produce a confidence score per clause. Low-confidence classifications and unrecognized clauses route to a reviewer rather than proceeding to deviation detection. Evaluate classification accuracy per clause type on the labeled set.

Step 4: Detect deviations and assign severity

For each classified clause, compare the contract language to the playbook's positions:

1. Retrieve the playbook entry for the clause type.
2. Determine whether the language matches the preferred position, an acceptable fallback, or neither.
3. Apply hard rules that override model judgment.
4. Assign severity per the playbook.
5. Produce the flag with the contract language, the playbook position, and a short explanation, side by side.

Severity assignment is rule-driven; the model's role is language comparison, and its output is evidence for a lawyer rather than a decision.

Step 5: Extract key terms

Extract structured fields: parties, effective and expiry dates, renewal and notice terms, payment terms, governing law, liability caps, and any playbook-defined fields. Each field carries a confidence score and a source location. Key fields are verified by the reviewer before they populate the contract management system. See how to build an ai data extraction pipeline.

Step 6: Design the lawyer review workflow

The review interface determines whether review is real. Show the contract with flags anchored to their locations, the playbook position beside the contract language, severity and explanation, and one-click actions: accept, reject, escalate, request alternative language. Capture every decision and edit for audit and evaluation. Route by severity and matter to the right lawyer. Review queue design is in how to build a human review queue.

Step 7: Evaluate by clause-level recall

Build a labeled set of past contracts of the type in scope, annotated by lawyers with clause types, deviations, and severities. Measure:

Metric	Why
Clause classification accuracy by type	Upstream errors propagate
Deviation recall by type	Missed deviations are the critical failure
Deviation precision	False flags erode trust
Severity accuracy	Routing depends on it
Key-term field accuracy	Downstream records depend on it
Unrecognized-clause handling	Must surface, not misclassify

Set thresholds with the legal team, weighted by consequence; recall on liability, indemnity, and IP clauses carries the highest bar. Wire the suite into CI. Method is in the AI evaluation and testing whitepaper.

Step 8: Implement privilege and confidentiality controls

• Matter-level access control mirrored from the document management system.
• Vendor terms prohibiting training on client content, or private deployment; see private llm vs public api.
• Client-specific restrictions from outside-counsel guidelines encoded as policy.
• Access and output logging.
• Work-product labeling and storage per counsel's guidance.

Detail is in ai access control and ai data leakage prevention.

Step 9: Launch and expand

Launch on one contract type with lawyer review of every output and measure recall, precision, review time, and reviewer edits. Refine the playbook from edits; lawyers will discover positions they had never written down. Expand to further contract types with their own playbooks and labeled sets. The extension path is described in the worked example in the AI for legal operations whitepaper.

What does it cost to run?

Run cost scales with contract volume and length and is small relative to lawyer time; build cost is dominated by playbook encoding, labeled-set creation, and integration with document and contract management systems. The value measure is review time per contract and missed-deviation rate against the baseline. Drivers are in document ai cost.

What are the common mistakes?

• Asking the model to "find risks" without a playbook.
• Flattening contracts in parsing and losing cross-references.
• Measuring overall accuracy instead of recall on high-severity clauses.
• Review interfaces that hide source language, producing click-through approval.
• Retrieval that ignores ethical walls.
• Expanding to complex agreements before the first type is trusted.

This guide is general guidance, not legal advice; confirm contractual and regulatory obligations with counsel before deployment.

How FISTA Solutions builds contract analysis systems

FISTA Solutions builds contract analysis systems to this playbook: playbook-as-specification with the legal team, structure-preserving ingestion with matter-level access control, clause classification and rule-driven deviation detection, review workflows that show evidence, clause-level recall evaluation in CI, and privilege controls designed with counsel. The AI agents practice delivers the system, AI enablement the document and retrieval platform, and forward deployed engineers embed with your lawyers to encode the playbook. The record behind the work is 150+ projects with 99.9% uptime.

To scope a contract analysis system for one contract type, message FISTA on WhatsApp, or read ai for legal teams for the functional overview.