AI Evaluation and Testing Strategy: A Whitepaper

Traditional software is tested by asserting that a given input produces a given output. AI systems cannot be tested that way, because the same input can produce different outputs and because correctness is often a matter of degree. What replaces testing is evaluation: measuring how often, and how well, the system meets its specification across a representative set of cases. This whitepaper sets out a complete evaluation and testing strategy for LLM applications and agents.

Why does evaluation replace testing for AI systems?

A model's behavior can only be characterized statistically. You cannot inspect its logic, and any single output tells you little about the next. Correctness must therefore be defined externally in a specification and measured empirically against a dataset. This has practical consequences: quality is a number with a confidence interval, changes are regression risks, and the dataset is as important an asset as the code. FISTA's foundational explainer is LLM evaluation explained; the spec-side method is in the spec-driven development for AI whitepaper.

What are the components of an evaluation strategy?

Component	Purpose
Specification	Defines correct behavior and generates test categories
Golden dataset	Labeled reference cases covering the spec
Metrics	Task-appropriate measures of quality
Automated scoring	Deterministic checks and calibrated model graders
Human review	Labels, calibration, and judgment on hard cases
Safety and adversarial tests	Injection, leakage, policy violations, unsafe actions
Regression gates	Release blocked on threshold breaches
Production sampling	Continuous measurement and drift detection
Feedback loop	Production failures added to the golden set

How is a golden dataset built?

The golden dataset is derived from the spec. Each business rule, output constraint, prohibited action, and escalation threshold generates categories of cases. Construction steps:

1. Enumerate categories from the spec: normal cases per rule, boundary cases, prohibited-action attempts, escalation triggers, out-of-scope inputs.
2. Source real inputs from historical data where possible, anonymized as required; synthesize only to fill gaps, and label synthetic cases as such.
3. Label with domain experts, using written guidelines so labels are consistent; measure inter-annotator agreement and resolve disagreements.
4. Attach expected outputs or rubrics: exact outputs for structured tasks, reference answers and grading rubrics for free-text tasks.
5. Version the dataset and record its provenance; changes go through review.
6. Stratify so that reporting can show quality per category, not just overall.

Coverage matters more than size. Detail is in what is a golden dataset and AI training data checklist.

Which metrics fit which tasks?

Task type	Primary metrics	Notes
Classification and routing	Accuracy, precision, recall, F1 per class	Report per class; watch minority classes
Structured extraction	Field-level exact match, schema validity, completeness	Partial credit per field
Retrieval	Recall at k, precision at k, MRR	Evaluate separately from generation
Grounded generation	Groundedness, citation accuracy, answer correctness, refusal correctness	Reference answers plus rubric
Free-text generation	Rubric scores on accuracy, completeness, tone, policy compliance	Human-calibrated judge
Agent tasks	Task success rate, action correctness, unsafe-action rate, steps and cost per task	Evaluate trajectories, not just final output
Safety	Injection success rate, leakage rate, policy-violation rate	Adversarial suite

Retrieval and generation must be measured separately in RAG systems because they fail differently; see the enterprise RAG reference architecture. Agent evaluation is covered in how to build an agent evaluation harness.

How should automated scoring and LLM-as-a-judge be used?

Deterministic checks (schema validation, exact match, rule assertions) are cheap and reliable and should be applied wherever the task allows. For free-text outputs, LLM-as-a-judge scales scoring but introduces a second probabilistic component that must itself be validated:

• Write explicit rubrics with criteria and score anchors; vague instructions produce noisy judges.
• Calibrate the judge against a human-labeled subset and report agreement; if agreement is low, fix the rubric or the judge before trusting it.
• Prefer pairwise or reference-based judging over absolute scoring where possible; it is more stable.
• Watch for known biases: position bias, verbosity bias, self-preference when the judge shares a model family with the system.
• Re-calibrate periodically and whenever the judge model changes.

Concepts are covered in what is LLM-as-a-judge and what is an eval in AI.

What is the role of human review?

Humans do three things automated scoring cannot: label the golden set, calibrate the automated graders, and judge the hard and high-consequence cases. A sustainable review program samples production outputs on a schedule proportional to consequence, uses the same guidelines as the golden set, and feeds disagreements back into rubric and dataset improvements. Review is also where domain owners maintain their ownership of the definition of correct. The operational design is in how to build a human review queue.

How is safety tested?

Safety tests live in the same suite as quality tests and run on every change:

• Prompt injection: direct and indirect payloads through every content channel the system reads; success measured as instruction override, data exfiltration, or unauthorized action.
• Data leakage: attempts to extract sensitive content from context, memory, or other users' data.
• Policy violations: cases designed to elicit prohibited content, commitments, or advice.
• Unsafe actions for agents: attempts to trigger out-of-scope tools, exceed limits, or bypass gates.
• Robustness: malformed inputs, adversarial formatting, and edge-case data.

Red-team exercises extend the suite with novel attacks, and every finding becomes a permanent case. See the AI agent security architecture whitepaper and AI red teaming guide.

How do regression gates work?

Every change to prompts, models, retrieval configuration, tools, or data triggers the full suite in CI. Thresholds per metric and per category are set from the spec; a breach blocks release. Provider model updates are treated as changes: pin versions, run the suite on the new version in a staging configuration, and adopt only when it passes. Results are stored so trends are visible across versions. This is the mechanism that turns evaluation from a launch activity into an operating discipline; the philosophy is described in verification-led engineering.

How is quality measured in production?

Production evaluation has three strands:

1. Sampled automated scoring of live traffic using the same graders, with results tracked over time and alerts on drops.
2. Scheduled human review of stratified samples, weighted toward high-consequence categories and low-confidence outputs.
3. Drift indicators: shifts in input distribution, confidence distribution, escalation rates, and user feedback that signal the golden set no longer represents reality.

Production failures are triaged and, where representative, added to the golden dataset so the suite grows with reality. The monitoring architecture is in the AI observability whitepaper and how to monitor AI in production.

How should evaluation results be reported?

Report per category and per version, with confidence intervals where sample sizes allow. Overall averages hide the minority categories where systems fail. A useful report shows current quality by category against thresholds, trend across versions, safety-suite results, production sample results, and open findings. Business owners read the category view; engineering reads the version view; governance reads the safety and trend view. Guidance on measurement for leadership is in how to measure AI success.

What are the common evaluation mistakes?

• Tuning on the test set. Iterating prompts against the golden set until it passes, then discovering the long tail. Keep a held-out set.
• Twenty hand-picked examples. Not a dataset; a demo script.
• Uncalibrated judges. Trusting a model grader nobody checked against humans.
• Overall averages only. Missing the category where the system is dangerous.
• Launch-only evaluation. No regression gates; silent degradation on model updates.
• No feedback loop. Production failures fixed individually and never added to the suite.
• Safety as a separate audit. Tested once, then never again as the system changes.

What does an evaluation program cost, and what does it save?

Building the golden set and harness is a real cost, dominated by expert labeling time. It is also the cost most often cut, and the cut is false economy: it is repaid by eliminating implementation churn, preventing production incidents, enabling safe model migrations, and making autonomy increases defensible. In TCO terms, evaluation is a build-phase investment that lowers oversight and error-handling costs for the life of the system; see the AI total cost of ownership whitepaper.

What is a minimum viable evaluation program?

For a team starting from nothing, the minimum that still works is a specification with categories, a golden set of a few hundred labeled cases stratified by category, deterministic checks wherever the task allows, one calibrated model grader for free-text outputs, a small adversarial suite, a CI job that runs it all and fails on threshold breaches, and a weekly human review of production samples. Everything beyond that improves precision and coverage; nothing less provides control.

How FISTA Solutions runs evaluation

FISTA Solutions builds the evaluation harness before tuning a single prompt. Our forward deployed engineers derive the golden dataset from the spec with your domain experts, choose metrics per task, calibrate automated graders against human labels, and wire regression gates into the pipeline. The AI enablement practice delivers the shared evaluation service that every subsequent workflow reuses, and the AI agents practice evaluates trajectories, actions, and safety as first-class metrics. The discipline is central to a record of 150+ projects with 99.9% uptime.

To assess how an existing AI system is evaluated, or to build a harness for a new one, message FISTA on WhatsApp. For a compact version of this whitepaper, use the AI evaluation checklist.