Sanket's Tutorials

01 - How LLMs Work

12 min read

Scope

The mental model a senior engineer needs to use and operate an LLM well — not a math course. How text becomes tokens (BPE), what an embedding actually is (and the two very different things people call “embeddings”), what the transformer/attention block does conceptually, why generation happens one token at a time, why the context window is a shared, quadratically-priced budget, and why the KV cache — not the weights — dominates inference memory. Then the failure modes that fall straight out of this architecture, and the decision tree for pretraining vs fine-tuning vs RAG vs in-context learning. For the knobs that control how tokens are sampled, see 02 - Decoding, Sampling & Reasoning Controls.

Related: Gen AI Engineer, 02 - Decoding, Sampling & Reasoning Controls, 06 - Embeddings & Vector Search, 17 - Deployment & Serving

1. The one-sentence model

An LLM is a function that, given a sequence of tokens, predicts a probability distribution over the next token — and you run it in a loop. Everything else (tokenizer, attention, KV cache, context window) is machinery that makes that loop fast, coherent, and long-range aware.

One token is sampled from the distribution, appended to the input, and the whole sequence is fed back in — autoregressive generation.

2. Tokenization: tokens ≠ words ≠ characters

Models don’t see text. They see token IDs — integers into a fixed vocabulary (typically ~100k–200k entries for modern models). The tokenizer that produces them is almost always Byte-Pair Encoding (BPE) or a byte-level variant.

How BPE works: start from raw bytes/characters, then repeatedly merge the most frequent adjacent pair into a new vocabulary entry. Frequent strings (the, ing, def) collapse to a single token; rare strings fragment into many.

InputRough tokenizationTokens
the["the"]1
strawberry["straw", "berry"]2
antidisestablishment["anti","dis","establish","ment"]4
你好 (CJK)several byte-level piecesmany
1234567often split oddly per-digit-groupvaries

Why an engineer must care about tokens

  • Cost and rate limits are per-token, not per-word. A rough English rule is ~0.75 words/token (~4 chars/token), but code, JSON, non-Latin scripts, and long numbers are far denser — they cost more than they look. Never size a prompt budget in characters.
  • Context limits are token limits. “200k context” is 200k tokens, not characters or words.
  • Tokenization is model-family-specific. Counting tokens for one model with another’s tokenizer is wrong. Use the provider’s own counter (e.g. Anthropic’s token-counting endpoint, OpenAI’s tiktoken).

3. Two things called “embeddings” — engineers conflate them

This is one of the most common confusions in interviews and design reviews. There are two different vector spaces and they are not interchangeable.

Internal token embeddingsRetrieval / sentence embeddings
WhatThe first layer of the LLM maps each token ID → a learned vectorA separate model maps a whole chunk of text → one vector
GranularityPer tokenPer sentence/paragraph/document
Where it livesInside the generative model, never exposedA standalone embedding model + a vector DB
PurposeGive the transformer something continuous to compute onSemantic search / RAG — find text by meaning
APINot directly accessibleA dedicated embeddings endpoint

They are different models with different vector spaces

The vector for the token "cat" inside a chat model is not the vector a retrieval embedding model produces for the sentence “I have a cat.” You cannot mix them, compare them, or feed one where the other is expected. When someone says “let’s embed the docs for RAG,” they mean retrieval embeddings — covered in 06 - Embeddings & Vector Search. When the transformer “embeds tokens,” that’s an internal detail you never touch.

4. Transformer & attention — the conceptual version

Inside the model, token embeddings flow through a stack of identical transformer blocks. The core operation is self-attention: for each token, the model decides which other tokens in the sequence to “look at” and how much.

Each token is projected into three vectors — Query (Q), Key (K), Value (V). Attention scores how well a token’s Query matches every other token’s Key, normalizes those scores, and uses them to take a weighted sum of Values:

  • QKᵀ = how relevant each token is to each other token (a full token-by-token grid).
  • /√d_k = scaling so the softmax doesn’t saturate.
  • softmax = turn scores into weights that sum to 1.
  • × V = blend the values according to those weights.

What this buys you, conceptually

Attention is how the model routes context: a pronoun can “attend to” the noun it refers to 500 tokens earlier; a closing brace can attend to its opener. Stacking many attention layers + feed-forward layers lets the model build increasingly abstract representations. The key cost fact: that token-by-token grid is O(n²) in sequence length n — see §6.

The original architecture: Attention Is All You Need (Vaswani et al., 2017), in the Sources.

5. Autoregressive decoding: one token at a time

Generation is a loop, not a single forward pass:

  1. Feed the full prompt → model outputs a distribution over the next token.
  2. Sample one token from it (how you sample = 02 - Decoding, Sampling & Reasoning Controls).
  3. Append that token to the sequence.
  4. Feed the whole thing back in → next distribution.
  5. Repeat until a stop token / length cap.

Consequences that fall straight out of this:

  • Output is serial. You can’t generate token 100 before token 99. This is why output tokens are slower and usually priced higher than input tokens, and why latency scales with output length (see 17 - Deployment & Serving).
  • Errors compound. A wrong early token conditions everything after it — the model will confidently continue down a bad path.
  • The prompt is processed in parallel (the “prefill”), but generation (the “decode”) is one step per token.

6. The context window: a shared budget with quadratic cost

The context window is the maximum number of tokens the model can attend to at once. Two facts engineers routinely get wrong:

Input and output share one budget

The window covers prompt + system + tools + retrieved docs + conversation history + the generated answer + (on reasoning models) hidden thinking tokens — all of it. A “200k context” model with a 190k-token prompt has ~10k left for the answer and any reasoning. Over-stuffing the input is the usual cause of truncated or empty completions.

Cost is roughly quadratic in context length

Attention’s O(n²) means doubling the context can ~quadruple the attention compute, and the KV cache (§7) grows linearly and must be held in GPU memory the whole time. Long context is not free real estate — it is the dominant driver of latency and serving cost. Fit only what’s needed; prefer retrieval over dumping (see 06 - Embeddings & Vector Search).

2026 landscape (high-churn — verify per model): frontier models advertise large windows — Claude and GPT-4.1-class models at ~1M tokens, Gemini 2.5 Pro at 2M, some open-weight models claiming far more. But see context rot in §8: advertised ≠ usable.

7. The KV cache: the real inference memory cost

Naively, generating token N would re-attend over all previous tokens from scratch every step — recomputing their Keys and Values each time. Instead, every K and V vector computed so far is cached so each new token only computes its own Q/K/V and attends against the stored cache.

The KV cache, not the weights, dominates per-request memory

Model weights are fixed and shared across all requests. The KV cache is per-request and grows with every token (∝ sequence length × layers × hidden size × number of concurrent requests). On long contexts and high concurrency it, not the weights, is what fills the GPU and caps how many users a box can serve. This is the central constraint of LLM serving.

Inference optimizations built around the KV cache

TechniqueWhat it doesWhy it matters
PagedAttention (vLLM)Stores the KV cache in fixed-size pages (like OS virtual memory) instead of one contiguous blockCuts memory fragmentation/waste to a few %, raising throughput 2–4×; enables high batch sizes
Prefix / prompt cachingReuses the KV pages of an identical prompt prefix (system prompt, tools, few-shot examples) across requestsSkips re-prefilling shared text → big latency + cost savings; providers expose this directly
Speculative decodingA small “draft” model proposes several tokens; the big model verifies them in one passSpeeds up decode without changing the output distribution

More on serving and how these surface as provider features: 17 - Deployment & Serving.

8. Failure modes rooted in the architecture

These aren’t bugs to patch — they’re consequences of “predict the next token.”

Hallucination is the architecture working as designed

The model is trained to produce fluent, probable continuations — not true ones. It has no built-in notion of “I don’t know.” A confident, well-formed, wrong answer is the same machinery as a right one. Mitigation: ground it. RAG, tool use, citations, and asking for “I don’t know” in the prompt reduce it; they don’t eliminate it. There is no temperature setting that makes a model truthful.

The "count the r's in strawberry" problem is tokenization, not stupidity

The model never sees the letters. strawberry arrives as ["straw","berry"]; the characters inside a token are invisible unless explicitly trained on. So character-level tasks (counting letters, reversing strings, exact substring math) are unreliable by construction. Mitigation: give it a tool (code execution / a function) for anything character- or arithmetic-exact, rather than trusting the raw model.

Context rot: usable context << advertised limit

Independent testing (Chroma, Context Rot, July 2025; Hong, Troynikov & Huber) across 18 frontier models found accuracy degrades non-uniformly as input grows — often 30–50% well before the documented limit. A 1M-token window does not reliably reason across 1M tokens; a 200k model can lose accuracy by ~50k tokens of input. Counterintuitively, coherent input sometimes degraded attention more than shuffled input, and items earlier in the sequence are recalled more reliably (position bias). Mitigation: don’t treat the window as a dumping ground. Retrieve and place the most relevant material near the top, keep prompts tight, and evaluate on your lengths — don’t trust the spec sheet. (See 06 - Embeddings & Vector Search and 02 - Decoding, Sampling & Reasoning Controls.)

9. Pros / cons

StrengthsLimits
Fluent, general, few-shot adaptable with no retrainingNo ground truth → hallucinates confidently
Strong at pattern/language tasks across domainsCharacter/exact-arithmetic tasks unreliable (tokenization)
Long context enables RAG, agents, long docsQuadratic attention + linear KV cache → context is expensive
Frozen weights → cacheable, shareable across requestsKnowledge frozen at training cutoff; needs RAG/tools to be current
Parallel prefill → fast time-to-first-tokenSerial decode → output latency scales with answer length

10. Decision tree: pretraining vs fine-tuning vs RAG vs in-context learning

Cheapest and fastest first; escalate only when the cheaper option provably fails.

ApproachChanges weights?Use whenCost / effortWatch out
In-context learning (prompting, few-shot, system instructions)NoBehaviour/format/tone; task is explainable in the prompt; first thing to tryLowest — minutesEats context budget; brittle to wording
RAG (retrieve facts, inject into prompt)NoThe gap is knowledge — private, fresh, or factual data the model can’t have memorizedLow–medium; build a retrieval pipelineRetrieval quality caps answer quality; adds latency; context rot if you over-stuff
Fine-tuning (LoRA/full)YesThe gap is consistent style/format/skill that prompting can’t reliably get; you have good labeled dataMedium–high; data + training + eval + hostingDoesn’t add knowledge reliably; can stale; must re-do on base-model upgrades
Pretraining (from scratch)Yes (all)Almost never for an application team — a new base model, new modality, or proprietary scaleHighest — millions, large infraRarely justified; assume the answer is “no”

The senior heuristic

Knowledge problem → RAG. Behaviour problem → prompt, then fine-tune. Never reach for fine-tuning to teach facts, or for pretraining at all, unless you’ve proven the cheaper layers fail. They compose: a typical production stack is a base model + RAG + careful prompting, with fine-tuning added only where it earns its keep. Decision detail lives in 06 - Embeddings & Vector Search (RAG) and the fine-tuning note.

11. Caveats & risk mitigation

  • Token counting is model-specific and load-bearing for cost. Centralize tokenization/counting; never estimate budgets from character counts. A wrong estimate truncates prompts or 400s the request.
  • Don’t conflate the two embedding spaces (§3) — a recurring architecture-review mistake that produces silently broken retrieval.
  • The context window is shared and quadratic — size prompts so the answer (and hidden reasoning tokens) fit; prefer retrieval to stuffing.
  • KV cache is the serving bottleneck — capacity planning for self-hosting must model KV memory at peak concurrency and context length, not just weight size. See 17 - Deployment & Serving.
  • Hallucination and character-tasks are structural — design around them (grounding, tools, evals), don’t expect a prompt to “fix” them.
  • Context rot means specs lie — benchmark recall/accuracy at your real input lengths before promising long-context behaviour.
  • Model/version churn is high (2026). Context sizes, exact model IDs, and which models accept which params change frequently — verify against current provider docs before quoting numbers in code or to stakeholders. Don’t hardcode assumptions about a specific model’s window.

12. In practice

# 1. Count tokens with the model's OWN tokenizer before sending — budgets are per-token.
#    (Anthropic exposes a token-counting endpoint; OpenAI uses tiktoken.)
#    Pseudocode — verify the current API for your provider/model.
n_tokens = count_tokens(model="<model-id>", text=prompt)
assert n_tokens + expected_output_tokens < context_window  # input + output share the budget
 
# 2. Reuse a stable prefix so the provider can prefix-cache the KV pages of system+tools.
SYSTEM = "...long, stable system prompt + tool defs..."   # keep this identical across calls
messages = [{"role": "system", "content": SYSTEM}, *dynamic_turns]
 
# 3. Knowledge gap? Retrieve and inject the few most-relevant chunks near the TOP
#    (position bias + context rot) instead of dumping the whole corpus.
context = retrieve(query, k=5)            # RAG, not a 1M-token dump

13. Sources

Backlinks

Graph View