Plain Explanation

Model serving means operating a model behind an API or endpoint so it can be used in a real product. A model working once in a notebook is different from a service answering thousands of concurrent users reliably. Serving combines inference, request handling, scaling, monitoring, and versioning into an operational system.

Examples & Analogies

• Restaurant operation: if the model is the recipe, serving is order intake, kitchen flow, delivery, and quality control.
• Chatbot API: receives user messages, runs LLM inference, and streams the answer back.
• Image classification service: queues uploaded images and processes them with batch inference.

At a Glance

Concept	Scope	Main concern
Inference	one model execution	latency, output quality
Model Serving	inference as a service	API, scaling, monitoring
Batch Inference	many inputs together	throughput, cost
MLOps	full model lifecycle	data, training, deploy, governance

Where and Why It Matters

Once an AI feature enters a product, model serving quality shapes the user experience. A strong model can still fail the product if cold starts are slow, GPU memory is exhausted, or rollback is missing. LLM serving especially depends on time to first token (TTFT), streaming, continuous batching, KV cache, rate limits, and prompt/token accounting.

Common Misconceptions

• “Putting the model in Docker is enough” → autoscaling, health checks, observability, and rollback are still needed.
• “A fast model means a fast service” → queue wait, batching, network, tokenizer, and postprocessing all affect latency.
• “High GPU utilization always means efficient serving” → bad tail latency or high error rates still indicate poor serving.
• “We can just swap in the new model” → versioning, canary traffic, and rollback plans are required.

How It Sounds in Conversation

• “Model quality is fine, but serving P99 latency violates the SLO.”
• “Changing batching improves GPU utilization but may hurt TTFT.”
• “Serve the new model behind canary traffic and define rollback conditions first.”
• “Cost per request needs token usage and GPU memory, not just request count.”

Related Reading

• Inference: the actual model execution inside a serving stack.
• GPU: the main compute resource for large-scale serving.
• KV Cache: controls LLM serving concurrency and memory.
• Quantization: reduces serving memory and cost.
• Autoscaling: adjusts replicas and resources to traffic.