Sanket's Tutorials

02 - Decoding, Sampling & Reasoning Controls

6 min read

Scope

The knobs that control how a model turns its probability predictions into text: temperature, top-p, top-k, penalties, stop sequences, seeds, logit bias, logprobs — and the 2026 shift on reasoning models from numeric sampling params to effort/thinking budgets. This is “tuning the LLM at inference time” (no training involved). For tuning the weights, see 12 - Fine-tuning & Model Adaptation.

Related: Gen AI Engineer, 01 - How LLMs Work, 03 - Prompt & Context Engineering, 15 - Cost, Latency & Performance

1. The concept: a model outputs a distribution, not a word

At each step an LLM produces a probability over every token in its vocabulary (the logits, after softmax). “Decoding” is the policy that picks the next token from that distribution. The simplest policy — always take the highest-probability token (greedy) — sounds safe but produces flat, repetitive, degenerate text (“the the the”, looping phrases). So we sample, and the parameters below shape how.

Watch what temperature does to that distribution — same predictions, different shape:

At low temperature one token dominates (near-deterministic); as temperature rises the distribution flattens and rarer tokens get a real chance (diverse, “creative”, eventually incoherent).

2. Temperature

Temperature T divides the logits before softmax: softmax(logits / T).

TEffectUse for
→ 0distribution sharpens to the top token (≈ greedy, near-deterministic)extraction, classification, code, structured output, math
~0.2–0.4focused, slight variationfactual Q&A, RAG answers, agents/tool-calling
~0.7–0.9balanced creativitychat, drafting, brainstorming
≥1.2flat, surprising, risks incoherenceideation, diverse sampling for eval

Mental model

Temperature doesn’t add knowledge or “make it smarter” — it only changes how much probability mass the rarer tokens get. Low T = “trust the top prediction.” High T = “take more risks.” For anything you’ll parse or that has a right answer, keep it low.

3. Top-p (nucleus) and top-k

These truncate the candidate set before sampling, so improbable tokens can’t be chosen at all:

ParamRuleNote
top-kkeep only the k highest-probability tokensfixed count; blunt when the distribution is very peaked or very flat
top-p (nucleus)keep the smallest set whose cumulative probability ≥ p (e.g. 0.9)adaptive — few tokens when confident, more when uncertain. The common default.
min-p (2024)keep tokens with prob ≥ p × max_probnewer; robust at high temperature

Don't tune temperature and top-p together

They interact and you’ll chase your tail. Pick one as your primary knob (most teams hold top_p≈1 and tune temperature, or hold temperature and tune top_p). Tuning both at once makes results hard to reason about and reproduce.

4. Penalties (OpenAI-family)

Discourage repetition by subtracting from the logits of already-seen tokens:

ParamEffectCaveat
frequency_penaltypenalty grows with how often a token appearedreduces verbatim loops
presence_penaltyflat penalty once a token appears at allnudges toward new topics
repetition_penalty(some APIs) multiplicative variant

Penalties break structured output

Values above ~0.3–0.5 will damage JSON, code, and anything with required repeated tokens (braces, indentation, common keywords). Keep penalties at/near 0 for structured or code generation. Anthropic has no direct penalty params — handle repetition with prompting/stop sequences instead.

5. The other controls

ParamWhat it doesGotcha
max_tokens (now max_completion_tokens / max_output_tokens)hard cap on output lengtha cap, not a target; on reasoning models hidden thinking tokens also spend it — set it generously or you truncate before the answer
stop sequencesend generation when a string appearsthe stop string is not included in output; useful to fence formats
seedrequest reproducibilitybest-effort only — pair with system_fingerprint; not bit-exact across batched serving
logit_biasadd/subtract a fixed bias to specific token IDsoperates on tokens not words; tokenizer-specific; easy to get wrong
logprobs / top_logprobsreturn per-token probabilitiesthe only way to see what your sampling settings actually did — great for debugging and confidence scoring
nreturn multiple samplesmultiplies cost; useful for self-consistency / pass^k

Why no beam search in chat APIs? It optimizes for high-probability sequences, which produces bland, generic text for open-ended generation — sampling wins for chat. Beam search lives on in some translation/constrained tasks, not the chat endpoints.

6. ⚠️ The 2026 shift: reasoning models are dropping these knobs

This is the highest-churn part of the topic — and a real interview signal.

Frontier "reasoning" models increasingly reject temperature/top_p

  • OpenAI o-series and GPT-5 don’t accept temperature/top_p/penalties/logit_bias — you control them with reasoning_effort (minimal/low/medium/high).
  • Anthropic Claude 4.x (Opus 4.7+) dropped temperature/top_p/top_k tuning (returns an error on non-default values); you use an extended-thinking / effort budget instead.
  • Google Gemini 2.5 uses thinkingConfig.thinkingBudget.

The mental shift: instead of “how random should each token be,” you now spend “how much should the model think.” Knob-tuning is migrating from per-token sampling to effort/budget abstractions. Classic sampling params still apply to non-reasoning / open-weight models — know both worlds and check what your specific model supports.

Don't bolt manual chain-of-thought onto a reasoning model

Telling a reasoning model to “think step by step” is redundant and can trigger overthinking that degrades accuracy and burns tokens. Use its native thinking/effort setting instead. (More in 03 - Prompt & Context Engineering.)

7. Pros / cons / when

GoalSettings (non-reasoning models)
Extraction / classification / routingtemperature=0, top_p=1, penalties 0
Structured output / codetemperature=0–0.2, penalties 0 (they corrupt structure)
RAG / factual answertemperature=0.1–0.3 (grounded, low drift)
Chat / assistanttemperature=0.5–0.8
Brainstorm / diverse draftstemperature=0.9–1.2, maybe n>1
Reasoning model (o-series/GPT-5/Claude 4.x)set effort/thinking budget, leave sampling at defaults

8. Caveats & risk mitigation

  • “Temperature 0 = deterministic” is a myth. Floating-point and batched-kernel non-determinism mean even T=0 isn’t bit-exact across runs/replicas. Don’t promise reproducibility; if you need it, snapshot inputs+outputs and assert on meaning (via evals), not exact strings. See 13 - Evaluation & Evals.
  • Provider drift. Param names and support differ (OpenAI vs Anthropic vs Gemini vs open-weight) and change per release. Centralize model config; don’t scatter magic numbers. A wrong param can 400 the request or be silently ignored.
  • High temperature ≠ smarter. It’s a common misconception that cranking temperature improves creativity quality — past a point it just adds incoherence. Raise it deliberately and eval the result.
  • Cost coupling. n, large max_tokens, and high reasoning effort all multiply token spend and latency — tie these to 15 - Cost, Latency & Performance budgets.
  • Debug with logprobs. If output is surprising, pull logprobs to see whether the model was confident (a knowledge/prompt problem) or uncertain (a sampling problem) — they need different fixes.

9. In practice

# Non-reasoning model: deterministic extraction
resp = client.chat.completions.create(
    model="...",
    messages=[...],
    temperature=0,            # focused
    top_p=1,                  # tune ONE knob, not both
    max_completion_tokens=512,
    stop=["\n\n"],
    seed=42,                  # best-effort reproducibility
    logprobs=True,            # see the model's confidence
)
 
# Reasoning model: no temperature — spend "thinking" instead
resp = client.responses.create(
    model="<reasoning-model>",
    input=[...],
    reasoning={"effort": "high"},   # the modern knob
    max_output_tokens=4096,         # leave room: hidden thinking tokens count
)

10. Sources

Backlinks

Graph View