LLM Inference and Serving Systems
LLM serving architecture: continuous batching, paged KV cache, quantization, prefix caching, autoscaling, failure modes, evals, and when not to self-host.
- Read time
- 16 min
- Level
- Advanced
- Updated
- 2026-08-25
- Components
- Continuous batcher · Paged KV cache · Quantization layer
An LLM inference serving system is the runtime that turns GPU (or other accelerator) capacity into token throughput and latency SLOs: a scheduler that continuously batches in-flight generations, a paged KV-cache so memory is not wasted on padding, optional quantization, prefix caching for repeated prompts, and autoscaling on queue depth rather than CPU. Model weights are the easy part; the serving stack is what makes a private deployment usable.
Key takeaways
- 01
Continuous batching, not static batches, is what makes interactive serving efficient: sequences join and leave the batch as they finish.
- 02
KV-cache memory, not FLOPs, is usually the binding constraint; paged attention and cache-aware scheduling are the levers.
- 03
Quantization (FP8, INT8, AWQ/GPTQ-class INT4) is a measured trade: run the eval suite at the serving dtype, not at the training dtype.
- 04
Autoscale on queue time and KV-cache occupancy, not on GPU utilization — a full GPU can still be latency-starved.
- 05
Pin model and engine versions; silent engine upgrades change tokenizer behavior, stop sequences, and structured-output reliability.
What an inference stack actually is
A serving stack is a request scheduler plus a model runtime. The scheduler owns admission, prefill vs decode, preemption, and fairness. The runtime owns kernels, KV-cache layout, tensor or pipeline parallelism, and sampling. In front sits a gateway: auth, request schema, streaming SSE/gRPC, timeout, and a load-shedding policy. Beside it sits a cache for prefixes and, optionally, a draft model for speculative decoding. Observability is not optional: queue time, prefill ms, time-to-first-token, tokens/s, KV occupancy, and abort rate.
Engines in this class — vLLM, TensorRT-LLM, TGI, SGLang, and vendor runtimes — differ in kernels and features, not in the architecture. If you cannot name where prefill, decode, KV pages, and the queue live in your deployment, you do not have a serving system yet; you have a Python script holding a GPU.
Data flow: prefill, decode, cache
A request arrives with prompt tokens, sampling params, and optional logit bias or grammar. Prefill runs the prompt through the model once and materializes KV pages. Decode then emits one token (or a speculative block) at a time, appending to those pages. Continuous batching mixes prefills and decodes from many requests in the same step so a finished sequence’s slots are reused immediately. Streaming tokens leave the gateway as they are produced; the client does not wait for EOS.
Prefix / prompt caching reuses KV pages when many requests share a long system prompt or a RAG prefix. It is the single largest win for agent and RAG workloads whose prompts are 80% identical. It is also a correctness hazard if two “same” prefixes differ by a tokenizer version or a trailing space — hash the exact token ids, not the string.
- Admit → tokenize → (prefix cache hit?) → prefill → decode loop → EOS/abort
- Continuous batch: sequences enter and exit every step
- KV pages allocated on demand; preemption when occupancy hits the cap
- Prefix cache keyed on token ids plus model/engine version
- Stream tokens; never buffer the full completion for interactive SLOs
KV cache, quantization, and parallelism
During decode, each live sequence holds a KV cache that grows with context length. That memory, not matrix multiplies, is why a GPU “at 100%” still queues. Paged KV layout (the idea behind PagedAttention and its descendants) allocates cache in blocks so you do not reserve max-length padding for every request. The scheduler must know occupancy: admit a new prefill only if pages will exist for the promised max tokens, or preempt the lowest-priority decode.
Quantization shrinks weights and sometimes KV. FP8 and INT8 are the usual production starting points on modern accelerators; 4-bit is a memory play that needs the eval suite at that dtype. Tensor parallelism splits layers across GPUs for large models; pipeline parallelism is for models that do not fit even split. Do not combine speculative decoding, aggressive quantization, and a new engine version in one change — you will not know which broke stop sequences.
Autoscaling, routing, and multi-model
Scale replicas on queueing delay and KV occupancy, with a floor that covers cold-start. GPU utilization is a trailing indicator and will tell you to scale down while p95 TTFT is already red. Keep a warm pool; model load plus CUDA graph capture is minutes, not seconds. Route short-context cheap models and long-context or high-quality models to different pools so a 128k RAG job cannot starve interactive chat.
Multi-LoRA serving belongs here when you have many adapters on one base: load adapters on demand, cap concurrent adapters by VRAM, and pin which adapter id a request may use. Merge-on-deploy is simpler if you have one adapter and a stable base. Either way the gateway, not the client, chooses the adapter.
Failure modes
Head-of-line blocking: a few long prefills occupy the batch and interactive TTFT explodes. OOM from KV growth mid-decode, especially with unbounded max_tokens. Tokenizer mismatch between training, eval, and serve. Prefix-cache collisions or stale pages after a prompt template change. Structured-output grammars that deadlock the sampler. Load shedding that returns 200 with an empty stream. An autoscaler that chases GPU% and flaps.
Silent quality drift is the failure evals exist for. An engine upgrade that changes sampling defaults, a switch from FP16 to INT4, or a stop-token bug that truncates JSON will not page you. Pin engine, image, model hash, tokenizer, and dtype. Canary a replica on the golden set before the rest of the fleet moves.
- Long prefills starving decode (need prefill caps / chunked prefill)
- KV OOM and unplanned preemption
- Tokenizer or stop-sequence mismatch
- Quantization accepted without the eval suite at that dtype
- Autoscaling on GPU% instead of queue time
Evals, and when not to self-host
Serving evals are two suites. Quality: the same golden set you run for prompts, at the production dtype, engine, and sampling params — including JSON/schema tasks. Performance: TTFT p50/p95, inter-token latency, throughput at a stated concurrency, abort rate, and KV occupancy under a replay of production traces. A tokens/s number from a single long generation is marketing, not capacity planning.
Do not self-host if your volume does not fill a GPU, if you cannot staff on-call for engines, or if a provider’s zero-retention endpoint already meets residency. Do self-host when data cannot leave the VPC, when you need a pinned engine for regulated evals, or when prefix-heavy agent traffic makes a dedicated fleet cheaper than token APIs. The architecture above is what you operate in that case — not “run vLLM and hope.”