Glossary ¡ 4 minute read
What Is an LLM Router? Sending Each Request to the Right Model
An LLM router is a component that decides, per request, which model should handle it, based on task type, estimated difficulty, cost, latency requirements, or policy such as data residency. Routers range from static rules per feature to learned classifiers, and they cut cost by sending routine requests to smaller models while reserving capable models for hard ones.
Using the strongest model for every request is simple and expensive. Most workloads mix routine requests a small model handles well with hard ones that need a capable model, and the price gap between them is large. An LLM router exploits that mix by sending each request to the cheapest model that meets quality. This explainer covers routing strategies, evaluation, and pitfalls, drawing on FISTA Solutions' AI enablement practice. The infrastructure it lives in is described in what is an ai gateway and the optimization program in the ai cost optimization checklist.
What is an LLM router?
An LLM router is a decision component that selects a model, and sometimes a prompt variant or provider, for each request according to defined criteria: task type, predicted difficulty, quality requirements, cost and latency budgets, and policy constraints. It sits in the request path, usually within the gateway, and its decisions are logged with the outcome so routing can be tuned on evidence. Token economics are in llm token cost explained.
What routing strategies exist?
| Strategy | How it decides | Strengths | Limits |
|---|---|---|---|
| Static rules | Per feature or endpoint configuration | Simple, predictable | Ignores per-request variation |
| Task classification | Classifies the request type and maps to a model | Matches capability to task | Needs a classifier and labeled data |
| Difficulty prediction | Predicts whether a small model will suffice | Fine-grained savings | Prediction errors cost quality |
| Cascade | Cheap model first; escalate on low confidence | No prediction needed | Added latency on escalation |
| Policy routing | Residency, vendor, compliance constraints | Enforces rules | Not about quality or cost |
| Load and health | Routes around outages and rate limits | Resilience | Must be combined with the above |
Failover behavior is in what is a fallback model.
How does routing cut cost without hurting quality?
By identifying the share of requests a smaller model answers as well as the large one and sending only those. The identification is the hard part: it needs either a classifier trained on labeled requests, a confidence signal from the small model, or task boundaries that separate routine from hard work. Savings equal that share times the price gap, minus router overhead. Model choice trade-offs are in the AI total cost of ownership whitepaper.
How do cascades work?
The request goes to a cheap model first; a checker, such as the model's own confidence, a validator, or a judge model, decides whether the answer is good enough; if not, the request escalates to a stronger model. Cascades avoid predicting difficulty in advance at the cost of extra latency and spend on escalated requests. They suit workloads where most requests are easy and latency budgets allow a retry. Judge-based checking is in what is llm as a judge.
How should a router be evaluated?
Run the golden dataset through the router and compare quality per category and overall with the single-strong-model baseline, while measuring cost and latency. Report quality by route, because aggregate quality can hide a category that the small model fails. Re-evaluate when models, prompts, or traffic change. Evaluation practice is in what is an eval in ai and the harness in how to build an agent evaluation harness.
What are the pitfalls?
Routing on cost alone and discovering quality loss in production; classifiers trained on old traffic that drift as usage changes; cascades that escalate so often the savings vanish; policy constraints applied after quality routing so a request lands on a disallowed model; and routers that cannot explain their decisions. Log every routing decision with outcome, and review routes monthly. Drift monitoring is in ai evaluation vs ai monitoring.
How does routing interact with prompts and caching?
Different models may need different prompt variants, so routers often select a prompt alongside a model, versioned together. Caching sits before routing so repeated requests never reach a model at all. Prompt management is in how to build a prompt management system and caching in what is prompt caching.
What does routing look like in practice?
A support platform classifies incoming requests into categories. Order status and policy lookups route to a small model with retrieval; billing disputes and multi-step troubleshooting route to a capable model; requests from regulated regions route only to approved providers; and provider errors fail over automatically. Weekly evaluation shows quality per category matching the strong-model baseline while spend falls substantially. The routing configuration lives in the gateway and changes without deployments. Support agent construction is in how to build an ai customer service agent.
How FISTA Solutions implements routing
FISTA Solutions starts with a strong-model baseline and a golden dataset, measures which categories smaller models handle at parity, implements routing in the gateway with policy constraints applied first, logs every decision with outcomes, and re-evaluates on every model or prompt change. The AI enablement practice delivers the platform, AI agents benefit from routed calls, and forward deployed engineers embed with client platform teams. The record behind the approach is 150+ projects with 99.9% uptime.
To cut model spend without losing quality, message FISTA on WhatsApp, or read ai inference cost for the cost structure routing optimizes.
Share-ready article cover
Download the generated social format.
Clear answers
Questions raised by this field note.
Straightforward guidance for evaluating scope, fit, and the next step.
01What is an LLM router in simple terms?
A dispatcher for AI requests. Instead of every request going to one model, the router looks at what the request is and sends easy ones to fast, cheap models and hard ones to stronger, costlier models. Done well, quality stays the same and the bill drops.
02What routing strategies exist?
Static rules per feature or endpoint; classification of the task type; predicted difficulty from the request; cascades that try a cheap model and escalate when confidence is low; and policy routing by data residency, vendor allowances, or latency budgets. Most production routers combine several.
03How much can routing save?
It depends on the share of requests a smaller model can handle at acceptable quality and the price gap between models. Workloads with many routine requests see large savings; workloads that are uniformly hard see little. Measure on your own traffic before promising numbers.
04How do you evaluate a router?
Run the golden dataset through the router and score quality per route and overall against the single strong-model baseline, while tracking cost and latency. A router that saves money but drops quality on a category has failed on that category.
05Where does the router live?
Usually in the AI gateway, so routing rules change as configuration without touching applications, and so routing decisions are logged with cost and quality data. Some frameworks also route inside the application for task-specific logic.
Continue exploring
Related capabilities
Start with the hard problem
Need the outcome owned, not merely analyzed?
Tell us where delivery is constrained. Weâll map the fastest credible path from intent to verified production.