Scope
The engineering levers that make an LLM feature affordable and fast enough to ship: token economics, prompt caching (the highest-ROI lever), KV cache / PagedAttention, Batch APIs, model routing & cascades, semantic caching, quantization, speculative decoding, context compression, streaming, and the latency SLOs (TTFT, ITL, throughput, goodput) you actually get paged on. The through-line: optimize cost-per-successful-task, not benchmark score or raw $/token. All dollar/percent figures here are high-churn — re-pull live numbers before quoting them.
Related: Gen AI Engineer, 03 - Prompt & Context Engineering, 16 - Observability & LLMOps Lifecycle, 17 - Deployment & Serving
1. The concept: two phases, two cost structures
An LLM request runs in two phases, and conflating them is the root of most bad cost/latency intuition.
| Phase | What happens | Compute shape | Cost driver |
|---|---|---|---|
| Prefill (process the prompt) | All input tokens processed in parallel in one forward pass | Compute-bound, highly parallel → cheap per token | Sets TTFT (time to first token) |
| Decode (generate the answer) | Tokens emitted one at a time, each depends on the last | Memory-bandwidth-bound, sequential → slow, expensive per token | Sets ITL and total latency |
The single most useful mental model on this page
Prefill is parallel and cheap. Decode is sequential and slow. You can feed a model 50K input tokens almost as fast as 5K (one parallel pass), but every output token is a separate sequential step. That is why output tokens are priced higher than input tokens, and why “make the output shorter” is often a bigger latency win than “make the prompt shorter.”
This asymmetry explains nearly every lever below: caching attacks redundant prefill, speculative decoding attacks sequential decode, batching amortizes both.
2. Token economics: input ≠ output
Providers bill separate rates per million tokens, and the rates are not symmetric.
| Token class | Typical relative price | Why |
|---|---|---|
| Input (prompt) | baseline (cheapest) | processed in one parallel prefill pass |
| Output (completion) | several× input (often 3–5×) | each token is a sequential decode step |
| Cached input | a fraction of input (see §3) | prefill work was already done and stored |
| Reasoning/thinking tokens | billed as output | hidden tokens still cost — and they count against max_output_tokens |
Reasoning tokens are an invisible bill
On reasoning models the hidden chain-of-thought is billed at output rates and can dwarf the visible answer. A “short” reply can cost 10× its visible length. Budget and cap them (see 02 - Decoding, Sampling & Reasoning Controls on effort budgets).
Durable rule, not a number: output dominates cost on generative workloads; input dominates on long-context/RAG workloads. Profile your input:output ratio before optimizing — the right lever differs entirely between the two.
Numbers churn weekly
Per-token prices, cache discounts, and batch SLAs change with every model release and price war. Treat every figure in this note as illustrative. Re-pull from the provider’s live pricing page before putting a number in a budget or a contract.
3. Prompt caching — teach this first, it’s the highest-ROI lever
If you do one thing for cost, do this. Most production prompts are mostly static — a long system prompt, tool definitions, few-shot examples, a retrieved document — followed by a small variable user turn. Prompt caching stores the prefill state of that static prefix so it isn’t recomputed every call.
It wins on both axes at once: cached tokens are billed at a steep discount and skip prefill compute, so TTFT drops too.
[ system prompt ][ tool defs ][ few-shot ][ retrieved doc ] [ user turn ]
└─────────────────── cache this static prefix ──────────────┘ └─ varies ─┘
The one rule that makes caching work: prefix stability
Caching is prefix-based. The cache hits only if the beginning of the prompt is byte-identical to a previous request. Change one token near the top — a timestamp, a reordered tool, a per-user greeting — and everything after it misses. Put static content first, volatile content last. This ordering discipline is the whole game.
Per-provider behavior (shapes are durable; discounts/TTLs are churn — verify live):
| Provider | Activation | Cached-read discount* | Notes |
|---|---|---|---|
| OpenAI | automatic above a token threshold | partial (historically ~50%; newest flagships moved toward ~90%) | no code change; you just keep prefixes stable |
| Anthropic (Claude) | explicit cache_control markers (up to 4 breakpoints) | steep (~90%) | you pay a write premium (1.25× for short TTL, 2× for long TTL); markers can sit mid-prompt so a RAG doc caches separately |
| Google (Gemini) | implicit + explicit context caching | steep | explicit caching has its own storage pricing |
*Directional only as of mid-2026 — providers have been converging toward ~90% cached-read discounts on flagship models. Confirm before quoting.
Practical caching playbook
- Sort the prompt: static → volatile. System prompt and tools at the very top.
- On Anthropic, set
cache_controlat the boundary of the stable block; mind the write premium — only worth it if the prefix is reused before the TTL expires.- Watch the cache-hit rate as a first-class metric (see 16 - Observability & LLMOps Lifecycle). A “caching” feature with a 5% hit rate is a misconfigured prefix.
- Short TTLs (minutes) suit bursty chat; long TTLs suit stable system prompts reused all day.
Pros: largest single cost cut for prompt-heavy/RAG/agent workloads; also lowers TTFT; minimal code. Cons: fragile to prefix changes; write premium can lose money if prefixes rarely repeat; TTL expiry silently reverts you to full price.
4. KV cache & PagedAttention — the same idea, one layer down
§3 is caching across requests. The KV cache is caching within a single generation: during decode the model stores the key/value tensors for every prior token so it doesn’t re-attend from scratch each step. It’s what makes decode tractable — and it’s a memory hog that grows linearly with sequence length × batch size, becoming the binding constraint on how many concurrent requests a GPU can hold.
PagedAttention (vLLM) manages that KV memory like an OS manages RAM — in non-contiguous pages instead of one giant pre-reserved block. This kills internal fragmentation, lets the server pack far more concurrent sequences onto a GPU, and enables prefix sharing across requests. Net effect: higher throughput and lower cost per token at the serving layer.
Where this lives
KV-cache/PagedAttention tuning is a serving-layer concern (you’re running vLLM/TGI/SGLang yourself). Provider prompt caching (§3) is the managed-API version of the same instinct. Full treatment in 17 - Deployment & Serving.
5. Batch APIs — ~50% off if you can wait
For any workload that doesn’t need a real-time answer — nightly evals, bulk classification/extraction, embedding backfills, synthetic data, document enrichment — submit it as a batch job.
The deal (shape is durable; verify exact terms)
~50% discount on input+output, in exchange for an asynchronous ~24-hour SLA. OpenAI, Anthropic, and Google all offer functionally equivalent batch endpoints with the same per-request schema as the sync API. As of mid-2026 the ~50% / 24h shape held across all three — confirm current numbers and job-size limits.
Pros: trivial ~2× cost cut, no model/quality change, near drop-in (same params). Cons: not for anything user-facing or interactive; results arrive whenever within the window (often faster, but no guarantee); you must build job submission/polling/result-reconciliation plumbing.
Risk mitigation: route by latency requirement at the application layer — interactive traffic to sync, everything tolerant of delay to batch. Tag batch jobs so a failed/partial batch is re-runnable idempotently.
6. Model routing & cascades — cheap model first, escalate on doubt
Not every query needs your most expensive model. A router sends each request to the cheapest model that can handle it; a cascade runs a cheap model first and escalates to a stronger one only when confidence is low.
| Pattern | How | Reference work |
|---|---|---|
| Router | classify difficulty up front (small classifier / heuristic), dispatch to the right tier | RouteLLM |
| Cascade | cheap model answers; a scorer judges confidence; escalate the low-confidence tail to the strong model | FrugalGPT |
The economics: if the cheap model handles 70–80% of traffic acceptably, blended cost-per-task collapses even though the hard tail still hits the expensive model. This is one of the biggest real-world savers because most production traffic is easy.
The router is itself a system you must eval
A router that misroutes hard queries to the weak model silently degrades quality to save pennies — exactly the failure that tanks
cost-per-successful-taskwhile looking great on the cost dashboard. The confidence/escalation threshold is the critical knob: tune it against an eval set (13 - Evaluation & Evals), and watch escalation rate as a live metric. Also budget the router’s own latency/cost — a heavy classifier on every request can erase the savings.
Pros: large blended savings; lets you reserve frontier models for the queries that need them. Cons: added complexity + a new failure mode; threshold needs ongoing tuning as traffic and models drift.
7. Semantic caching — answer reuse, NOT prefix reuse
Distinct from §3. Prompt caching reuses prefill compute for byte-identical prefixes. Semantic caching (e.g. GPTCache) reuses the final answer when a new query is semantically similar to a past one: embed the query, vector-search prior queries, and if similarity clears a threshold, return the stored answer with zero model call.
When it fits — FAQs, repetitive support questions, common how-tos — it’s the cheapest possible response (an embedding lookup, no generation).
FALSE-HIT risk is the whole danger
Two queries can be near in embedding space but demand different answers (“refund policy for 2023” vs “2024”; “how do I enable X” vs “disable X”). A loose threshold serves a confidently wrong cached answer. Mitigations:
- Conservative similarity threshold — bias toward cache misses; a miss costs a model call, a false hit costs trust.
- Never cache personalized or time-sensitive answers — anything keyed to a user, account, balance, date, or live data. Scope the cache to genuinely stable, generic knowledge.
- Add a TTL and an invalidation path for when the underlying truth changes.
- Monitor false-hit rate with periodic human/LLM-judge audits.
Pros: can eliminate the model call entirely for repetitive traffic; big latency + cost win on FAQ-shaped workloads. Cons: correctness hazard from false hits; useless for high-variety prompts; another store to operate and invalidate.
8. Quantization — smaller weights, less VRAM (self-hosted)
Quantization stores weights/activations in lower precision (FP8, INT8, INT4) instead of FP16/BF16, cutting VRAM and memory bandwidth — which raises throughput and lets bigger models fit on smaller/cheaper GPUs.
| Format | Trade-off | Use for |
|---|---|---|
| FP8 | minimal quality loss; increasingly the default serving precision on modern GPUs (native hardware support) | general production serving |
| INT8 / AWQ / GPTQ | bigger VRAM savings, small quality cost on most tasks | cost-sensitive serving, larger models on limited VRAM |
| INT4 | largest savings, measurable degradation on hard tasks | latency/VRAM-critical, tolerant tasks |
Don't quantize aggressively for math / reasoning / code
Low-bit quantization (esp. INT4/AWQ) tends to degrade multi-step reasoning, arithmetic, and precise code more than it degrades chat/summarization — the errors are subtle and only show under eval, not in a quick eyeball test. FP8 is a safe default; treat INT4 as opt-in for tolerant workloads and always A/B against the unquantized model on a task-specific eval before shipping.
Pros: fits bigger models on cheaper hardware; higher throughput; lower $/token. Cons: quality cliffs that are workload-specific and easy to miss; managed-API users don’t control this (the provider already picked).
9. Speculative decoding — beat the sequential decode bottleneck
Decode is slow because it’s sequential (§1). Speculative decoding breaks the chain: a small draft model proposes several tokens ahead, and the big target model verifies them all in one parallel forward pass, accepting the longest correct prefix. Accepted tokens come “for free” relative to running the big model token-by-token.
EAGLE (and successors) is the current strong approach, using a lightweight feature-level draft head for high acceptance rates.
It's a latency/throughput win, not a quality change
Speculative decoding is output-preserving — the target model verifies, so the final distribution matches what it would have produced alone. You get the same answer faster. (Contrast §6/§7/§8, which trade quality/correctness for cost.)
Pros: real decode-latency reduction with no quality loss; stacks with batching. Cons: serving-layer complexity (run + align a draft model); speedup depends on acceptance rate, which varies by domain; mostly a self-hosted lever (managed APIs may apply it transparently).
10. Context compression — fewer input tokens, but watch what you drop
When prompts are huge (long RAG context, fat histories), context compression (e.g. LLMLingua) prunes low-information tokens before the model sees them, cutting input cost and prefill latency while aiming to preserve meaning.
Compression can silently delete the one fact that mattered
The compressor decides what’s “low-information” — and it can be wrong, dropping the exact clause your answer depended on. The failure is invisible: the model produces a fluent, confident answer from an impoverished context. Always A/B compressed vs uncompressed on a real eval set and track task success, not token savings. If quality holds, ship it; if it dips, the savings aren’t real.
Pros: cuts input tokens on long-context workloads; lower TTFT. Cons: lossy by design; correctness regressions are hard to detect; adds a preprocessing step + its own latency. Often prompt caching (§3) is the better first move — it makes the static context nearly free without dropping anything.
11. Streaming — fixes perceived latency, not real cost or work
Streaming sends tokens to the client as they’re generated instead of waiting for the full completion. The user sees text appear at TTFT rather than after the whole (much longer) total latency.
Streaming changes perception, not economics
Same tokens, same total compute, same cost, same true end-to-end time. It only makes the wait feel short by showing progress early. Essential UX for chat; irrelevant for batch/back-end jobs where no human is watching. Don’t confuse “added streaming” with “made it faster.”
12. Metrics & SLOs — what you actually get paged on
You cannot manage latency you don’t measure. The standard serving vocabulary:
| Metric | Definition | What it captures |
|---|---|---|
| TTFT (Time To First Token) | request → first output token | prefill speed + queue wait; the responsiveness number users feel |
| ITL / TPOT (Inter-Token Latency / Time Per Output Token) | average gap between successive output tokens | decode speed; how fast text “flows” |
| Throughput | total tokens/sec across all concurrent requests | server efficiency / cost-per-token |
| Goodput | throughput of requests that meet their SLOs | the honest number — high throughput from requests that blew their TTFT budget is worthless |
Set SLOs as percentiles, on the right metric
Set TTFT and ITL targets at p95/p99, not averages — tail latency is what users complain about. There’s a fundamental TTFT ↔ throughput tension: large batches raise throughput (lower cost) but push individual TTFT up. Goodput is the metric that reconciles them — tune the serving config to maximize requests that hit SLO, not raw tokens/sec.
13. FinOps — attribute, cap, and tie cost to value
Aggregate spend is useless for decisions. You need cost broken down to where it’s incurred.
- Per-feature / per-tenant cost attribution. Tag every call with feature, user/tenant, and model. Without this you can’t find the one feature burning the budget or price a customer. Wire it into 16 - Observability & LLMOps Lifecycle.
- Runtime token caps. Enforce
max_output_tokens, context limits, and per-request/per-user budgets in code — a runaway agent loop or a pathological prompt can 100× a bill in minutes. Caps are the circuit breaker. - Reasoning-effort budgets. Default effort low; raise only where evals prove it pays (ties to 02 - Decoding, Sampling & Reasoning Controls).
- Pre-deploy cost estimation. Model input:output token ratio × volume × price before launch, not after the invoice.
The real production metric: cost-per-successful-task
Optimize cost per successful task, not $/token and not benchmark %. A model that’s 30% cheaper per token but fails 2× as often (forcing retries, escalations, human handoff, lost trust) is more expensive per resolved task and worse for users. Always: total cost ÷ tasks that actually succeeded (success defined by your eval — see 13 - Evaluation & Evals). Every lever in this note is justified only if it holds or improves that number.
14. Putting the levers in order
A pragmatic sequence — cheapest/safest wins first, riskiest last:
| Order | Lever | Effort | Quality risk |
|---|---|---|---|
| 1 | Prompt caching + prefix discipline (§3) | low | none |
| 2 | Shorten output / cap tokens / lower effort (§2) | low | low |
| 3 | Batch API for non-real-time work (§5) | low | none |
| 4 | Streaming for perceived latency (§11) | low | none |
| 5 | Model routing / cascade (§6) | medium | medium (eval the router) |
| 6 | Semantic caching for repetitive traffic (§7) | medium | high (false hits) |
| 7 | Quantization / spec-decoding / PagedAttention (§8,§9,§4) | high (self-host) | varies |
| 8 | Context compression (§10) | medium | high (silent drops) |
Self-hosted vs managed API
§4, §8, §9 are levers only if you run the serving stack (17 - Deployment & Serving). On managed APIs the provider already chose them; your levers are §3, §5, §6, §7, §10, plus prompt/output discipline.
15. Caveats & risk mitigation (summary)
- Every number here is high-churn. Prices, cache discounts, batch SLAs shift with each release. Re-pull from live provider pricing pages before quoting; centralize pricing in one config, never scatter magic numbers.
- Cost dashboards lie about quality. Routing, semantic caching, quantization, and compression can cut the cost line while quietly raising failure rate. Gate every one behind an eval on cost-per-successful-task.
- Caching is fragile. A reordered tool or injected timestamp near the prompt top silently drops your hit rate to ~0. Monitor hit rate as a metric, not an assumption.
- Latency is a distribution. Optimize p95/p99 and goodput, not averages. The TTFT↔throughput trade is real — pick the batch size that maximizes SLO-meeting requests.
- Caps are non-negotiable. Runtime token/budget limits are the only thing standing between a buggy loop and a five-figure surprise invoice.
16. Sources
- OpenAI — Prompt caching guide
- Anthropic — Prompt caching
- Anthropic — Pricing
- OpenAI — Batch API
- Anthropic — Message Batches
- Google — Gemini batch mode & context caching
- vLLM / PagedAttention paper — Efficient Memory Management for LLM Serving
- RouteLLM — learning to route LLMs
- FrugalGPT — using LLMs cheaply
- GPTCache — semantic cache for LLMs
- LLMLingua — prompt compression
- EAGLE — speculative decoding
- Speculative decoding (original)
- DistServe — TTFT/TPOT goodput, prefill/decode disaggregation