Cost · 4 minute read
LLM Cost per Task Benchmarking: Comparing Models Honestly
LLM cost per task benchmarking measures what each candidate model costs to complete one unit of your workload at the required quality, including tokens across all steps and retries, tool and retrieval costs, and the human time to finish cases the model gets wrong. It replaces per-token price comparison with a number that predicts your bill.
Model pricing pages invite a comparison that misleads: dollars per million tokens, side by side. A workload does not consume tokens in the abstract. It consumes the tokens a particular model needs to complete a task, including the retries when output fails validation, the extra steps an agent takes, the context the model needs to reach the quality bar, and the human time spent on the cases it gets wrong. Cost per completed task at equal quality is the number that predicts the bill, and it is benchmarkable. This guide shows how. It applies the AI agent unit economics whitepaper to model comparison and feeds the routing decisions in how to design a model routing strategy.
What is the unit?
One completed task of the workload at the required quality: one document extracted correctly, one ticket resolved, one summary that passes the rubric. Cost per token, per call, and per step are inputs; cost per completed task is the result.
What are the components?
| Component | Measured how |
|---|---|
| Input tokens | Prompt, context, retrieved content, tool results, per step, summed over the task |
| Output tokens | Generated text and tool calls, per step |
| Retries | Extra calls after validation failures or errors |
| Steps | For agents, the number of model calls per task |
| Caching | Savings from cached context where the provider supports it |
| Tools and retrieval | Per-call costs of vector search, database queries, APIs |
| Infrastructure | Runtime and serving cost allocated per task; significant for self-hosted models |
| Human finishing | Failure rate × human time to complete or correct × loaded cost |
The last row is what makes "cheap" models expensive: a model with a lower price and a higher failure rate pushes cost into people.
How is the harness built?
- Start from the workload's golden set and correctness criteria.
- Use production-like prompts, context sizes, retrieval, and tools; benchmark the configuration that would ship.
- Run each candidate configuration over the set several times to capture variance.
- Record per task: tokens by type, steps, retries, tool calls, latency, quality score by category, pass or fail against the bar.
- Compute cost per task with your contracted prices, and human finishing cost from the failure rate.
- Report cost per task only for configurations that clear the bar in every category; report failures as failures.
The evaluation infrastructure is the same one described in the evaluation-driven development whitepaper.
How do you read the results?
| Configuration | Quality vs bar | Cost per task | Latency | Verdict |
|---|---|---|---|---|
| Frontier model | Clears | Highest | Slow | Baseline for quality |
| Efficient model | Clears | Low | Fast | Primary candidate |
| Cheapest model | Fails one category | Appears lowest | Fast | Not eligible; human finishing cost makes it highest |
| Cascade: efficient with escalation | Clears | Lowest among eligible | Fast for most cases | Often the winner |
Read quality first, then cost among eligible configurations, then latency. The cascade row is why configurations, not just models, must be benchmarked; the escalation rate determines its cost, and the harness measures it.
How do prompts and context change the answer?
Cost per task is sensitive to context discipline. Retrieving fewer, better chunks, trimming system prompts, and using cached context for repeated material can change a model's cost per task substantially without changing the model. Benchmark the disciplined configuration; a sloppy prompt makes every model look expensive. Caching in particular is worth measuring explicitly; see what is prompt caching.
What about self-hosted models?
Self-hosted open-weight models have fixed infrastructure cost that must be allocated per task by utilization. At low utilization, cost per task can exceed API pricing; at high, steady volume, it can fall well below. Benchmark with realistic utilization assumptions and include operating overhead. The strategic considerations are in open-weight models for regulated industries.
How should results feed decisions?
Routing rules, budgets, and business cases all draw on the benchmark. Attach the benchmark version and date to each routing rule, set per-workload budgets from it, and re-run on price changes, model updates, deprecations, and workload drift. The governance of routing is in the multi-model strategy whitepaper.
What are the common mistakes?
- Per-token comparison.
- Cost reported for models that fail the bar.
- Benchmarking toy prompts instead of the shipping configuration.
- Ignoring human finishing.
- Single-run results with no variance.
- Stale benchmarks driving routing.
How does FISTA Solutions help?
FISTA Solutions builds cost-per-task benchmarks for client workloads as part of its AI enablement practice, on the evaluation harness every AI agent FISTA delivers already carries, and turns the results into gateway routing rules and budgets with forward deployed engineers inside your platform team. FISTA is an official Anthropic partner and has delivered 150+ projects for 50+ companies across 12+ countries.
To benchmark one workload across candidates, message FISTA on WhatsApp, or read LLM token cost explained for the token-level fundamentals.
Share-ready article cover
Download the generated social format.
Clear answers
Questions raised by this field note.
Straightforward guidance for evaluating scope, fit, and the next step.
01Why not just compare per-token prices?
Because tokens per task vary by model: verbosity, retries after validation failures, steps an agent takes, and context each model needs all differ. A model with a lower per-token price can consume more tokens per task and need more human finishing, ending up more expensive at equal quality. Only a task-level benchmark shows it.
02What is included in cost per task?
Input and output tokens across every step and retry, cached-token savings where applicable, tool and retrieval call costs, infrastructure allocated per task, and the human time to finish or correct cases the model fails, weighted by the measured failure rate. All measured on your golden set at production-like settings.
03How do you compare at equal quality?
Set the workload's quality bar first, run every candidate on the golden set, and compare cost per task only among configurations that clear the bar in every category. A cheaper model below the bar is reported as failing, not as cheap, because its real cost includes the errors and the human work to fix them.
04How often should benchmarks be re-run?
Whenever a candidate model changes materially, prices change, a deprecation is announced, or the workload's inputs drift, and at least quarterly. The harness makes re-runs cheap, and rankings change often enough that a benchmark older than a quarter should not drive a routing decision.
Continue exploring
Related capabilities
Start with the hard problem
Need the outcome owned, not merely analyzed?
Tell us where delivery is constrained. We’ll map the fastest credible path from intent to verified production.