How you actually run an LLM in production: why generic web servers can’t serve LLMs efficiently (decode is memory-bandwidth-bound, not compute-bound), the two ideas that fixed it (continuous batching + PagedAttention), the 2026 engine landscape (vLLM, SGLang, TensorRT-LLM — and TGI now in maintenance), the throughput levers you’ll be asked about (prefix caching, chunked prefill, speculative decoding, quantization, prefill/decode disaggregation), GPU sizing math, the provider-API vs self-host decision, and the gateway/fallback layer. For the cost/latency numbers this all feeds, see 15 - Cost, Latency & Performance; for the API layer in front of it, FastAPI Senior Prep.
1. Why specialized LLM serving exists: decode is memory-bandwidth-bound
Generation runs in two phases with opposite hardware profiles, and getting this distinction right is the whole reason vLLM/SGLang/TensorRT-LLM exist.
Phase
What happens
Bottleneck
Prefill
process the whole prompt in parallel, build the KV cache
compute-bound — big matmuls saturate the GPU’s FLOPs
Decode
generate one token at a time, each step reads all weights + the KV cache from VRAM
memory-bandwidth-bound — tiny matmul per step, GPU sits idle waiting on memory
The decode loop’s arithmetic is trivial — one new token against the whole model. The cost is moving the weights and KV cache through memory every single step. A 70B model in FP16 is ~140 GB of weights; even at ~3 TB/s HBM bandwidth that caps you near ~20 tokens/s for a single request, and the GPU’s compute units are mostly idle.
The core consequence
Because decode wastes compute, the path to throughput is doing many requests’ decode steps together so one weight-read serves dozens of sequences. That is batching — and naive batching (wait for a full batch, run it to completion, return together) is terrible for LLMs because sequences finish at wildly different lengths. Specialized serving exists to batch well. A plain Flask/FastAPI loop calling model.generate() per request leaves 10–24x throughput on the table.
2. Continuous batching (Orca) — the first big idea
Static batching groups N requests, runs them together, and the whole batch waits for the slowest sequence. Short replies sit idle while a long one finishes; new arrivals wait for the next batch. GPU utilization tanks.
Continuous batching (a.k.a. iteration-level / in-flight batching, from the Orca paper, OSDI ‘22) schedules at the token step, not the request:
the scheduler runs one decode step across all active sequences
a sequence that hits its stop token leaves the batch immediately and its slot is freed
a newly arrived request joins on the very next step (its prefill is interleaved)
So the batch composition changes every iteration — no head-of-line blocking, no idle slots.
How the modern engines mix it: vLLM’s V1 scheduler can put prefill and decode in the same step (older V0 could only do one or the other per step), which is what chunked prefill (§5) builds on.
Interview one-liner
“Static batching wastes the GPU because the batch waits for the slowest sequence. Continuous batching schedules per token step, evicts finished sequences immediately, and admits new ones mid-flight — that’s the 10x+ throughput win, before any other trick.”
3. PagedAttention (vLLM) — the second big idea
Continuous batching needs somewhere to put each sequence’s KV cache. The pre-vLLM approach pre-allocated one contiguous chunk per request sized to max_tokens. Two failures:
Internal fragmentation — a request that stops early wastes the rest of its reserved block.
External fragmentation — free memory exists but not as one contiguous run, so a new request can’t fit. Reported KV waste was 60–80%.
PagedAttention borrows OS virtual memory: split the KV cache into fixed-size blocks (pages), keep a per-sequence block table mapping logical → physical blocks. Blocks need not be contiguous; you allocate on demand. Result: near-zero KV waste, so you fit 2–4x more concurrent sequences on the same VRAM — which directly raises the achievable batch size and thus throughput.
Bonus: blocks can be shared via copy-on-write — multiple sequences from the same prompt (parallel sampling, beam, shared system prompt) point at the same physical KV blocks until they diverge. This is the foundation of prefix caching (§5).
Why it matters as a pair
Continuous batching answers “how do I keep the GPU busy across requests?”; PagedAttention answers “how do I fit enough of them in VRAM to do that?” Together they’re why vLLM beat naive HF generate by an order of magnitude and reset the whole field.
4. The engine landscape (2026)
This section is HIGH-CHURN — verify versions before quoting
Engine status, defaults, and APIs move monthly. Below is current as of mid-2026; treat exact flags as “check the docs.”
Engine
Sweet spot
Notes
vLLM
the default open-source production server
PagedAttention origin; V1 engine is now the default architecture; OpenAI-compatible server; huge model coverage; broad hardware (NVIDIA/AMD/others)
SGLang
shared-prefix / high-concurrency workloads
RadixAttention caches KV in a radix tree keyed at token level → automatic, aggressive prefix reuse (paper reports up to ~6x on prefix-heavy loads); now in the PyTorch ecosystem; also does PD-disaggregation, spec decoding, structured output
TensorRT-LLM
max throughput on NVIDIA, latency-critical
compiles per-model/per-GPU engines; fastest on NVIDIA but heavy build step and least portable; often fronted by NVIDIA Triton / Dynamo
Hugging Face TGI
⚠️ maintenance mode (Dec 2025)
only minor bug fixes accepted now; HF points new deployments to vLLM or SGLang (and llama.cpp / MLX locally). Do not start a greenfield deploy on TGI.
Ollama / llama.cpp
local / dev / CPU+small GPU
llama.cpp is the GGUF/quantized engine; Ollama is the friendly wrapper around it. Great for laptops and demos, not high-concurrency serving.
MLX
local on Apple Silicon
Apple’s array framework; the right local engine on Macs
The TGI deprecation is an interview signal
If you cite TGI as a current production recommendation, you’re out of date. Since Dec 11 2025 TGI is in maintenance mode; HF themselves now recommend vLLM / SGLang (and llama.cpp / MLX for local). The “movement” TGI started — optimized engines on top of transformers — continues in those projects. Default answer for “what do we serve open-weight models on?” in 2026 is vLLM, with SGLang when the workload is prefix-heavy and TensorRT-LLM when you need the last drop of NVIDIA throughput.
5. Advanced throughput levers
These stack on top of continuous batching + PagedAttention. Know what each one trades.
Lever
What it does
Trade-off / caveat
Prefix caching
reuse KV of identical prompt prefixes (system prompt, tools, few-shot, RAG boilerplate) across requests — skip recomputing prefill
huge win when prefixes repeat; ~nothing when every prompt is unique. SGLang’s RadixAttention is the aggressive form; vLLM has automatic prefix caching too
Chunked prefill
split a long prompt’s prefill into chunks and interleave them with ongoing decode steps in the same batch
smooths the latency spike a big prefill causes for other users (less decode stall); default in vLLM V1
Speculative decoding
a cheap draft model proposes N tokens; the target model verifies all N in one forward pass, keeping the longest correct prefix
cuts per-token latency with identical output distribution; gain depends on draft acceptance rate; extra VRAM for the draft model
Quantization
serve weights/KV in FP8 / INT8 / INT4 / AWQ / GPTQ instead of FP16
shrinks VRAM and raises bandwidth-bound throughput; risks quality loss — eval before/after (see 13 - Evaluation & Evals if present). FP8 on Hopper/Blackwell is near-lossless and common
Prefill/decode disaggregation (PD)
run prefill (compute-bound) and decode (bandwidth-bound) on separate GPU pools so each scales independently and neither stalls the other
now an industry-standard pattern (vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo); adds KV-transfer cost and orchestration complexity — justified at scale, overkill for one box
How to talk about these
They’re not free speedups — each trades memory, complexity, or quality for throughput/latency. The senior move is matching the lever to the workload: prefix caching for shared system prompts and RAG; chunked prefill when long prompts hurt others’ latency; spec decoding for latency-sensitive single-stream; quantization when VRAM-bound; PD-disaggregation only at multi-GPU scale.
6. GPU sizing math (back-of-envelope)
You will be asked “what GPU(s) does this model need?” Reason it out, don’t memorize a table.
add ~30–50% overhead → ~18–21 GB before serving much context
→ fits a 24 GB card (e.g. one mid/high-end GPU); leaves only a little for KV, so concurrency/context is tight
Worked example — 70B model, FP16:
weights: 70e9 × 2 B ≈ 140 GB → does not fit one 80 GB GPU
need tensor parallelism across 2× 80 GB (160 GB) — and KV cache + overhead still eat into that, so 2 GPUs is the floor, not comfortable headroom
quantize to FP8 → weights ≈ 70 GB → now fits a single 80 GB GPU with room for KV
The lever that decides fit
KV cache scales with context length × concurrency and can rival the weights at long context / high batch. If you’re VRAM-bound the two big knobs are: quantize the weights (FP16→FP8 roughly halves them) and cap context/batch (or use GQA models, which have far smaller KV). Always reserve 30–50% overhead — running a GPU at 100% VRAM OOMs the moment a long prompt arrives.
7. Provider API vs self-host — the decision framework
Dimension
Provider API (OpenAI/Anthropic/Google/Bedrock)
Self-host (vLLM/SGLang on your GPUs)
Time to ship
minutes — no infra
days–weeks — GPUs, engine, autoscaling, on-call
Frontier quality
yes (latest GPT/Claude/Gemini)
only open-weight models
Unit cost at scale
per-token; cheap at low volume, expensive at high steady volume
high fixed GPU cost; cheaper per token once GPUs are saturated
Data control / residency
data leaves your boundary (check terms/zero-retention)
data stays in your VPC — needed for regulated/sensitive
Customization
limited (prompt, some fine-tune)
full — any open model, LoRA, quantization, custom kernels
Ops burden
none
you own capacity, OOMs, upgrades, GPU supply
Rate limits
provider’s quotas
your own capacity
Decision rule of thumb
Start on a provider API (fastest to value, frontier quality). Move to self-host when one of these bites: (1) steady high volume where per-token cost > amortized GPU cost, (2) data residency / compliance forbids sending data out, (3) deep customization (fine-tuned/open model, custom decoding), or (4) latency/availability guarantees the provider won’t give. Many shops run hybrid — provider for frontier/spiky traffic, self-host for the bulk steady workload — behind one gateway (§9).
Serverless GPU + cold starts
Serverless GPU platforms (Modal, RunPod, Replicate, Bedrock-style on-demand) let you pay only for inference and scale to zero — attractive for spiky/low-volume self-host. The catch is the cold start: loading tens of GB of weights into VRAM can take tens of seconds to minutes on the first request after scale-down. Mitigate with min-replicas > 0 (warm pool) for latency-sensitive paths, snapshotting, or keeping a small always-on tier. Scale-to-zero is great for batch/async, risky for interactive UX.
8. Structured-output serving
Serving engines can constrain generation to a grammar/JSON schema so the output is guaranteed parseable (grammar-/FSM-constrained decoding via Outlines/XGrammar backends).
vLLM moved off the guided_* API — VERIFY your version
vLLM unified its constrained-decoding params under a single structured_outputs config. The legacy fields — guided_json, guided_regex, guided_choice, guided_grammar, guided_whitespace_pattern, structural_tag, guided_decoding_backend — were deprecated and then removed (around v0.12.0). Migrate:
# OLD (removed): extra_body={"guided_json": schema}# NEW: structured_outputs configfrom vllm.sampling_params import StructuredOutputsParamsStructuredOutputsParams(json=schema) # also: regex=, choice=, grammar=# or over the OpenAI-compatible API, prefer the standard:response_format={"type": "json_schema", "json_schema": {...}}
Check the exact field names and removal version against the docs for the vLLM build you run — this has churned across releases.
For the schema/validation/retry pattern around all this (Pydantic, repair loops), see 03 - Prompt & Context Engineering if covered there, and the API layer in FastAPI Senior Prep.
9. Gateways, rate limits, and fallbacks
In front of one-or-many model backends you usually want a gateway (LiteLLM is the common open-source one; commercial options exist) that gives you one OpenAI-compatible endpoint, key/budget management, routing, retries, and fallbacks (primary model → secondary on error/timeout/rate-limit).
A gateway is another hop and a single point of failure
LiteLLM (and any gateway) adds latency and becomes a SPOF — if it’s down, all your model traffic is down. Run it HA: multiple replicas behind a load balancer, externalized state (shared Redis/Postgres, not in-process), health checks, and a tested bypass/failover path. Don’t put a single un-replicated proxy on your critical inference path.
Patterns to know:
Retries with backoff + jitter on 429/5xx — but cap them; infinite retries amplify an outage.
Fallback chain — e.g. primary self-host → provider API on overflow, or strong model → cheaper model on rate-limit. Watch for quality/behavior drift between fallback targets.
Per-key rate limits & budgets so one tenant can’t starve others.
Timeouts everywhere — long-tail LLM latency will hang your request pool otherwise (this is an FastAPI Senior Prep concern: async, connection pools, timeouts).
Load balancing across replicas/regions keyed on least-load or prefix-affinity (route same-prefix traffic to the same replica to hit its prefix cache).
10. Caveats & risk mitigation
Don’t serve LLMs on a generic web stack.model.generate() per request behind Flask/FastAPI gives ~no batching and wastes the GPU. Use a real engine (vLLM/SGLang) and put your FastAPI app in front of it as the API/orchestration layer — not as the model runner.
OOM is a step-function failure, not graceful. VRAM at 100% means the next long prompt or concurrency spike kills the process. Reserve 30–50% overhead, set max model context and max-num-seqs explicitly, and load-test at peak context length.
Quantization needs an eval gate. FP8 is usually near-lossless; INT4/aggressive quant can quietly degrade reasoning/long-context quality. Always benchmark the quantized model on your tasks before shipping.
Engine churn breaks deployments. vLLM removed guided_*; TGI went to maintenance; defaults flip between versions. Pin engine versions, read release notes on upgrade, and keep config centralized — don’t scatter engine flags.
Cold starts wreck interactive latency. Scale-to-zero serverless is fine for async/batch; for user-facing paths keep a warm minimum replica.
The gateway is a SPOF — covered in §9; HA it or skip it.
Cost is dominated by GPU saturation, not raw price. A cheap GPU at 10% utilization is more expensive per token than a pricey one at 90%. Throughput tuning (§2–5) is cost optimization — tie to 15 - Cost, Latency & Performance.
11. In practice — a vLLM serve sketch
Minimal containerized OpenAI-compatible server, then a FastAPI gateway in front:
# Dockerfile — vLLM OpenAI-compatible serverFROM vllm/vllm-openai:latest# Model + serving config baked as defaults (override at runtime)ENV HF_HOME=/modelsEXPOSE 8000# Tune for your GPU: tensor-parallel across N GPUs, cap context & concurrency,# enable prefix caching; quantize if VRAM-bound.ENTRYPOINT ["python", "-m", "vllm.entrypoints.openai.api_server", \ "--model", "meta-llama/Llama-3.1-8B-Instruct", \ "--tensor-parallel-size", "1", \ "--max-model-len", "8192", \ "--gpu-memory-utilization", "0.90", \ "--enable-prefix-caching"]# (chunked prefill is on by default in V1; add --quantization fp8 if needed)
# Calling it — it speaks the OpenAI API, so the standard client works.from openai import OpenAIclient = OpenAI(base_url="http://vllm:8000/v1", api_key="not-used")resp = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "..."}], temperature=0, max_tokens=512, # structured output: prefer the standard response_format over legacy guided_* (removed) response_format={"type": "json_schema", "json_schema": {"name": "out", "schema": {...}}},)
Where FastAPI fits
The FastAPI app is your gateway/orchestration tier — auth, rate limits, request shaping, fallback routing, streaming passthrough, timeouts — calling vLLM over its OpenAI endpoint. Keep async + bounded connection pools + per-call timeouts so long-tail latency can’t exhaust your workers. Details in FastAPI Senior Prep.