Playbook · 6 minute read
How to Build an Agent Evaluation Harness (Playbook)
To build an agent evaluation harness, define task suites from the specification with expected outcomes and allowed actions, create simulated environments with mocked tools and seeded state, capture full trajectories, score outcomes, action correctness, efficiency, and safety with deterministic checks and calibrated graders, gate releases in CI, and report by category and version.
An agent that produces a perfect final message after deleting the wrong record has failed. Evaluating agents therefore means evaluating trajectories: what the agent did, in what order, with what side effects, at what cost, and whether it stayed within policy. This playbook covers building an agent evaluation harness that does that repeatably and runs as a release gate, following the AI evaluation and testing whitepaper and FISTA's AI agents practice. Core concepts are in what is an eval in ai.
What does the harness do?
| Component | Function |
|---|---|
| Task suite | Cases derived from the spec with inputs, seeded state, expected outcomes and actions |
| Environment | Mocked or sandboxed tools; seeded data; captured side effects |
| Runner | Executes the agent per case; captures full trajectory |
| Scorers | Deterministic checks, efficiency metrics, calibrated graders |
| Gate | Thresholds per category; CI integration |
| Reporting | Results by category, version, and trend |
| Feedback | Production failures imported as cases |
Step 1: Derive the task suite from the specification
Each spec element generates cases: normal tasks per task type, boundary cases, prohibited-action attempts, escalation triggers, out-of-scope requests, tool failure scenarios, and adversarial inputs including injection through tool results and retrieved content. Each case defines inputs, seeded environment state, expected final state, required and prohibited actions, and rubric criteria for text outputs. Version the suite. See the spec-driven development for AI whitepaper.
Step 2: Build the simulated environment
Mock each tool with realistic behavior, seeded data, and controllable failures (timeouts, errors, malformed results). Capture every tool call with arguments instead of executing side effects. For tools that must run for realism, sandbox them with disposable state. Environments must reset deterministically per case. Sandboxing principles are in the AI agent security architecture whitepaper.
Step 3: Capture full trajectories
The runner records, per case: the prompt and context, each model call, each tool call and result, validation and gate decisions, the final output, steps, tokens, latency, and cost. Trajectories are stored for scoring and debugging and align with production tracing so the same tooling reads both. See how to build an ai audit trail.
Step 4: Score four dimensions
| Dimension | Method |
|---|---|
| Outcome | Deterministic checks on final environment state and required outputs |
| Action correctness | Required actions present with correct arguments; prohibited actions absent; gates triggered where required |
| Efficiency | Steps, tool calls, tokens, cost, latency against budgets |
| Safety | Injection resistance, refusal correctness, data-leakage checks, policy compliance |
Free-text outputs use rubric-based graders calibrated against human labels; see what is llm-as-a-judge. Record scores per case with the trajectory.
Step 5: Set thresholds and gate in CI
Set pass thresholds per category from the spec, with the strictest on prohibited actions and safety. Run the suite on every change to prompts, tools, models, retrieval, or orchestration, and on provider model updates in a staging configuration. Failures block release with a report linking to failing trajectories. Discipline is described in verification-led engineering.
Step 6: Handle non-determinism
Agents vary between runs. Run flaky-prone cases multiple times and score pass rates against thresholds rather than single outcomes; pin model versions and temperatures in the harness; and distinguish genuine regressions from variance with trend reporting. Document the variance budget per category.
Step 7: Report by category and version
Reports show pass rates and scores per category and per version, safety results, efficiency trends, and the specific failing trajectories. Business owners read category results; engineers read version diffs; governance reads safety trends. Store results for audit.
Step 8: Import production failures
Failed or escalated production trajectories are triaged and, where representative, converted into suite cases with seeded state reconstructed from the trace, so the suite tracks reality. This link between observability and evaluation is described in the AI observability whitepaper.
Worked example: a support agent harness
A customer service agent has tools for order lookup, address change, refund within policy, and ticket creation. The suite derives cases per intent: normal address changes, refunds inside and outside policy limits, ambiguous requests that must escalate, angry customers, and injection attempts embedded in order notes. The environment mocks the order system with seeded customers and captures every tool call. Scorers assert that refunds above the limit are never executed, that escalations occur on defined intents, and that address changes carry the correct arguments, while a calibrated grader scores response tone and accuracy. Thresholds are strict on prohibited refunds and injection cases. The suite runs in CI on each prompt and tool change, and a provider model update that subtly changed refund behavior is caught in staging before it reaches customers. Production escalations are reviewed weekly and the representative ones become new cases.
What does it cost?
The harness costs build time and ongoing run cost in model calls per suite execution, controlled by case sampling on minor changes and full runs on major ones. It repays itself by preventing incidents and enabling safe model migrations. See the AI total cost of ownership whitepaper.
What are the common mistakes?
- Scoring only final text and missing harmful actions.
- Running tests against live systems.
- Twenty hand-picked cases and no adversarial ones.
- Uncalibrated graders.
- No CI gate, so the harness is run occasionally and ignored.
- Never importing production failures.
Who owns the harness?
Engineering owns the runner, environments, and CI integration; the domain owner owns the case labels and thresholds; and both review failing trajectories together on a cadence.
How do you keep the harness fast enough to run on every change?
Tier the suite: a fast smoke set of a few dozen cases on every commit, the full set on merge and nightly, and expensive scenarios such as long tool chains weekly. Cache tool responses and retrieval results where determinism allows, parallelize runs, and track suite duration as a metric, because a harness that takes hours stops being run.
How FISTA Solutions builds evaluation harnesses
FISTA Solutions builds agent evaluation harnesses to this playbook as part of every agent delivery: spec-derived suites, simulated environments, full trajectory capture, four-dimension scoring with calibrated graders, CI gates with category thresholds, variance handling, reporting, and production feedback. The AI enablement practice delivers the shared evaluation service, AI agents are built against it, and forward deployed engineers work with your domain experts to derive and label the cases. The record behind the work is 150+ projects with 99.9% uptime.
To build or audit an agent evaluation harness, message FISTA on WhatsApp, or read what is a golden dataset for the data foundation.
Share-ready article cover
Download the generated social format.
Clear answers
Questions raised by this field note.
Straightforward guidance for evaluating scope, fit, and the next step.
01How is agent evaluation different from LLM evaluation?
LLM evaluation scores text outputs against references. Agent evaluation must also score the actions taken, their order and arguments, side effects on the environment, efficiency in steps and cost, and safety behaviors such as refusing prohibited actions, which requires simulated environments and trajectory capture.
02What is a simulated environment for agent testing?
A controlled setup where the agent's tools are mocked or sandboxed, state is seeded to known conditions, and external effects are captured rather than executed, so tests are repeatable, safe, and able to assert on what the agent did as well as what it said.
03How many test cases does an agent need?
Enough to cover every task type, rule, prohibited action, escalation trigger, and known failure in the specification, with several variants each, plus adversarial cases. Coverage of the spec matters more than raw count, and the suite grows with production failures.
04How do you score agent trajectories?
With deterministic checks on final state and required actions, assertions on prohibited actions and gates, efficiency measures on steps, tokens, and cost, and calibrated model graders for free-text outputs and rubric criteria, all recorded per case and category.
05How often should the harness run?
On every change to prompts, tools, models, retrieval, or orchestration as a CI gate, on provider model updates before adoption, and on a schedule against production-derived cases to detect drift.
Continue exploring
Related capabilities
Start with the hard problem
Need the outcome owned, not merely analyzed?
Tell us where delivery is constrained. We’ll map the fastest credible path from intent to verified production.