What Is Model Serving? Running Models in Production Explained

Model serving is where a working model becomes a dependable service, and it is the part most underestimated in project planning. The gap between a model producing correct output in a notebook and one serving traffic within a latency budget at acceptable cost is substantial. This explainer covers what a serving layer does and the decisions around it. It complements what is inference optimization and gpu cost for ai, and reflects FISTA Solutions' approach in AI enablement delivery.

What does a serving layer handle?

Request intake and queuing. Batching, so that the hardware processes several requests together rather than idling between them. Memory management for weights and attention caches. Backpressure when demand exceeds capacity. Routing between model versions. Streaming responses. And metrics detailed enough to diagnose problems.

Each of these is straightforward alone and non-trivial in combination, which is why dedicated serving frameworks exist rather than everyone writing a wrapper around a model call.

Concern	Hosted API	Self-hosted
Operational burden	Provider's	Yours
Cost model	Per token	Per hour of capacity
Data residency control	Limited	Full
Model choice	Provider catalogue	Any open-weight model
Scaling	Automatic	Your capacity planning
Version control	Pinning where offered	Complete

When does self-hosting make sense?

When data residency or sector regulation requires it. When sustained volume makes reserved capacity cheaper than per-token pricing. When a specific open-weight or fine-tuned model is needed that no provider offers. When latency requires the model close to the application.

Absent one of those, hosted APIs are the correct default. The operational cost of running inference infrastructure well — capacity, upgrades, monitoring, on-call — is routinely underestimated, and a preference for control does not by itself justify it.

Why are cold starts different here?

Because model weights are large. Loading tens of gigabytes onto accelerators takes minutes, not the milliseconds a container start takes. Scale-to-zero, which is ordinary practice for stateless services, means the first request after an idle period waits for that load.

The practical consequence is that serving infrastructure keeps warm capacity, and warm capacity costs money whether or not it is serving traffic. Utilisation, not peak capacity, is the number that determines whether self-hosting is economical.

What makes GPU autoscaling hard?

Acquisition latency, regional availability, warm-up time, and unit cost. A conventional service scales in seconds on abundant commodity instances; accelerator capacity may take minutes to acquire and may not be available at all in a given region at a given moment.

Scaling therefore reacts on a timescale slower than traffic changes. Capacity planning — provisioning for expected peak with a buffer — matters more than elasticity, which inverts the intuition most teams bring from web infrastructure.

How should versioning work?

By pinning explicitly and upgrading deliberately. Provider-side model updates change behaviour, sometimes subtly, and a system whose evaluation was run against one version has no guarantee against the next.

The pattern that works: pin the version, run the evaluation suite against a candidate before switching, deploy the change like a code change, and keep the ability to roll back. Treating a model version as configuration rather than as a given is the core discipline. See ai evaluation checklist.

What should be monitored?

Request latency including time to first token, which matters more than total time for streaming interfaces. Tokens per second. Queue depth and wait time. Batch sizes achieved. Memory utilisation. Error and timeout rates by cause. Cost per request.

Request-level latency alone hides the common failure where a system is nominally healthy while queue time has quietly become the dominant term.

How does batching affect latency?

It trades individual latency for aggregate throughput. Larger batches use hardware more efficiently and make each request wait longer for the batch to form. Continuous batching, where requests join an in-flight batch rather than waiting for a new one, substantially improves that trade and is standard in modern serving stacks.

The configuration should follow the workload: interactive interfaces want small batches and low wait, bulk processing wants the opposite.

What about multi-model serving?

Common and awkward. Serving several models on shared hardware saves cost when each is individually under-utilised, but memory is the binding constraint and swapping weights is slow. Approaches that share a base model across multiple fine-tuned adapters are considerably more efficient than running separate full models, where the use case allows it.

How does this connect to routing?

Directly. A serving layer that can route between a small fast model and a large capable one, based on task complexity, is often the largest available cost lever. The routing decision belongs above the serving layer, but the serving layer must support both models being available with predictable latency for it to work.

How FISTA Solutions helps

FISTA Solutions makes hosted-versus-self-hosted decisions on evidence about volume, residency, and latency, plans accelerator capacity rather than relying on elasticity, pins and evaluates model versions before upgrades, configures batching to the workload, and monitors token-level metrics, through AI enablement, AI agents, and forward deployed engineers. The record behind the approach is 150+ projects for 50+ companies with 99.9% uptime.

To put a model into production reliably, message FISTA on WhatsApp, or read ai evaluation checklist.