13 - Evaluation & Evals

Scope

How you know an LLM system is good enough to ship — and stays good as models, prompts, and data churn underneath it. Covers offline vs online eval, golden/eval sets, LLM-as-judge (and how to judge the judge), RAGAS for retrieval, benchmark contamination, eval-driven development, agent/trajectory evaluation with pass^k reliability, and the statistical rigor that separates a real win from noise. This is the discipline that makes the rest of the stack (07 - RAG, 08 - Agents & Agentic Systems) shippable. For running evals in production, see 16 - Observability & LLMOps Lifecycle.

A judge model scores a candidate output against a question and a reference — pointwise, pairwise, or rubric-based:

1. Why evals at all: a non-deterministic component on shifting ground

You’re shipping a component that is non-deterministic (same input → different output; even temperature=0 isn’t bit-exact, see 02 - Decoding, Sampling & Reasoning Controls) and built on a model you don’t control and that gets silently re-versioned. Both legs move:

The model changes — a provider point-release can shift behavior on your exact prompts overnight.
The prompt/RAG/agent code changes — every PR is a potential regression you can’t catch by reading a diff.

Unit tests assume determinism and exact equality. LLM output has neither. So you can’t ship — or refactor, or upgrade a model — with confidence unless you can measure quality automatically and repeatedly. Evals are that measurement. No evals = you are flying blind and “it looked fine in the demo” is your entire QA process.

The one-line case for evals

An eval is the regression test for a probabilistic system. Without it you cannot tell a real improvement from a lucky sample, or a silent model regression from a flaky run. Evals turn “feels better” into a number with a confidence interval.

2. Offline vs online evaluation

Two complementary regimes — you need both.

	Offline	Online
When	pre-deploy, in CI, on a fixed dataset	live, against real traffic
Against what	a golden set with known-good answers/criteria	production inputs (often no ground truth)
Measures	regression vs a baseline; gate releases	real user outcomes, drift, edge cases
Signals	judge scores, RAGAS, exact-match, pass^k	thumbs up/down, A/B conversion, escalation rate, guardrail trips
Cost / risk	cheap, safe, repeatable	real users feel the failures

How they feed each other

Online catches the long tail you never imagined; you harvest those failures back into the offline golden set so the next release can’t regress on them. Offline is your gate; online is your radar. A team with only offline evals ships things that surprise real users; a team with only online evals learns about every regression from users.

3. Golden / eval sets: the asset everything else depends on

The eval set is your definition of “good.” If it’s weak, every number downstream is theater.

Versioned. Treat it like code — in git, with a changelog. A score is meaningless without knowing which set produced it. Never silently edit cases.
Contamination-free. Cases must not have leaked into model training data (see §8) and must not be derivable from the system’s own prompt. Hold out a private set you never publish.
Head AND tail. Cover the common cases (the 80% of traffic) and the nasty tail — adversarial inputs, ambiguity, refusals, multilingual, the bug reports users actually file. Tail cases are where systems break and where naïve eval sets are silent.
Right-sized & labeled. Enough cases for a tight confidence interval (§10), each with a clear expected output or grading rubric. Start with ~50–100 hand-curated cases; grow from production failures, not from synthetic padding.
Stratified. Tag cases by category (retrieval-heavy, reasoning, tool-use, safety) so you can see where a regression landed, not just an aggregate that hides it.

Synthetic eval sets are a trap if unaudited

Generating eval cases with an LLM is fine for volume, but if the same model family writes and grades them you bake in its blind spots and self-preference (§5). Always human-audit a sample of synthetic cases before trusting the set.

4. Scoring methods: pick by what you can define

Method	Use when	Cost	Caveat
Exact / regex / schema match	structured output, classification, JSON, code that compiles	~free	brittle for free text; says nothing about quality
Reference-based metrics (BLEU/ROUGE/embedding similarity)	summarization, translation with a reference	cheap	n-gram overlap ≠ correctness; weak for open-ended text
LLM-as-judge (§5)	open-ended quality, helpfulness, tone, faithfulness	$$ per case	judge bias; needs calibration
Human eval (§9)	the gold standard / calibration anchor	slow, costly	doesn’t scale; rater disagreement

The trend: use cheap deterministic checks where the answer is checkable, LLM-as-judge for everything subjective, and humans to calibrate the judge — not to grade every run.

5. LLM-as-judge: modes and when to use each

Use a strong model to score outputs. Three modes, each fit for a different job:

Mode	What it does	Best for	Watch out
Pointwise (single-answer grading)	score one output on an absolute scale (e.g. 1–5, or pass/fail)	monitoring & regression tracking — you need a number per run over time	absolute scores drift between judge versions; calibrate
Pairwise (A vs B)	pick the better of two outputs	selection — prompt A vs B, model A vs B, choosing a winner	more reliable than absolute scores, but O(n²) comparisons; position bias (§6)
Reference-guided / rubric	grade against an explicit rubric and/or gold answer	anything with defined criteria; making scores explainable	rubric quality caps eval quality

Separate the rubric from the score (rubric-separated scoring)

Don’t ask “rate this 1–5.” Give the judge an explicit rubric with named criteria (accuracy, completeness, grounding, tone), have it score each criterion with a one-line justification, then aggregate. Decomposed, justified scoring is more stable, more auditable, and far less gameable than a single opaque number. Force the reasoning before the score, not after, or the score anchors the justification.

Prefer pairwise for decisions, pointwise for trends

When choosing between two candidates, pairwise beats comparing two absolute scores — relative judgments are more reliable than absolute ones. When watching for regressions over weeks, you need pointwise (you can’t pairwise-compare against last month’s run cheaply). Many teams run both.

6. Judge bias mitigation — and judging the judge

The judge is itself a non-deterministic model with systematic, measured biases. Ignoring them produces confident, wrong leaderboards.

Bias	What it is	Mitigation
Position bias	judge favors whichever answer is in a given slot (A vs B), worsens with 3–4 options	randomize order; run both orders and average (swap-and-average); discard cases where the verdict flips on swap
Verbosity / length bias	longer answers score higher even with no added information	control for length; instruct the judge to ignore length; penalize padding in the rubric
Self-preference bias	judge prefers outputs from its own model family (lower perplexity to it)	use a judge from a different family than the model under test; or an ensemble/cross-family panel and look for consensus
Sycophancy / formatting bias	favors confident tone, markdown, citations regardless of correctness	rubric scores substance separately from style

You must EVALUATE THE JUDGE itself

A judge is only trustworthy if it agrees with humans. Calibrate it against a human-labeled set and report agreement (e.g. Cohen’s κ, or % match on a held-out human-graded sample). If the judge–human agreement is low, your eval numbers are noise dressed as signal. Re-run this calibration whenever you change the judge model or its prompt — a judge model upgrade is a breaking change to your measurement instrument. Track judge agreement as a first-class metric, not a one-time setup step.

Don't grade with the same model you're shipping

If your product model is also your judge, self-preference bias inflates every score and you’ll never see your own model’s weaknesses. Pick a different (or independent) judge.

7. RAGAS: metrics for the RAG pipeline

RAGAS decomposes RAG quality so you can tell whether retrieval or generation broke — the single most useful diagnostic split in 07 - RAG. The four core metrics, split by stage:

Metric	Stage	Question it answers	Low score means
Faithfulness	generation	are the answer’s claims supported by the retrieved context?	model is hallucinating beyond its sources
Answer relevancy	generation	does the answer actually address the question (not padded/off-topic)?	answer drifts or is incomplete
Context precision	retrieval	are the retrieved chunks relevant, and ranked relevant-first?	retriever pulls noise; reranking weak
Context recall	retrieval	did retrieval pull all the info needed to answer?	needed chunks missing from the index/retrieval

Read them as a 2x2 diagnostic

Low faithfulness but high context precision → fix the generation prompt (it’s ignoring good context). Low context recall → fix retrieval (chunking, embeddings, top-k), no prompt change will help. RAGAS turns “RAG feels off” into a specific subsystem to fix. As of late 2025 RAGAS also ships extras — context entities recall, noise sensitivity, multimodal faithfulness — but the four above are the backbone.

8. Benchmark contamination

Public benchmarks (MMLU, GSM8K, MATH, HumanEval…) leak into pretraining corpora. When test items — or syntactic variants — sit in training data, the model memorizes answers and the score is inflated, telling you nothing about generalization. GSM8K and MATH already show score saturation from exactly this.

Public benchmark scores are marketing, not your eval

A model topping a leaderboard may have seen that leaderboard. For your product, contamination of a public benchmark is irrelevant — but contamination of your own eval set is fatal. Mitigations: keep a private, never-published hold-out; use canary strings to detect leakage; prefer dynamic/freshly-generated cases over static ones; and version cases by date so you can exclude anything predating a model’s training cutoff.

9. Human evaluation

The calibration anchor for everything automated. Use it to:

Validate the judge (§6) — humans grade a sample, you measure judge agreement against them.
Resolve the subjective tail — tone, harm, “would a user be satisfied” — where rubrics are fuzzy.
Build the initial golden set — early on there’s no other source of ground truth.

Human eval has its own noise

Raters disagree; “good” is subjective. Use clear rubrics, multiple raters, and report inter-rater agreement. A judge that disagrees with humans might be wrong — or the humans might be inconsistent. Measure both. Human eval is expensive and slow, so spend it on calibration, not on grading every run.

10. Eval-driven development

Treat evals like tests in TDD: the eval set defines “done,” and you don’t merge below the bar.

Gate releases on evals. A PR that drops faithfulness or pass-rate below baseline fails CI, same as a failing unit test. Prompt changes, RAG changes, and model upgrades all go through the gate.
Regression suites in CI. Every production failure becomes a permanent eval case. The suite only grows, so a fixed bug can never silently come back.
Model upgrades are eval events. “Should we move to the new model?” is answered by running both through the same eval set — not by vibes. This is the only safe way to handle the provider churn from §1.
Eval before optimize. You can’t improve what you can’t measure; build the eval before tuning the prompt, or you’ll optimize for a metric that doesn’t exist yet.

Eval-driven development = TDD for probabilistic systems

Write the eval, see the system fail it, change prompt/RAG/model, see the eval pass, lock it into CI as a regression guard. Same loop as code TDD — the only difference is the assertion is a score-above-threshold, not an exact equality.

11. Agent / trajectory evaluation

Agents (08 - Agents & Agentic Systems) need more than final-answer grading — how they got there matters, and they’re far more stochastic than a single LLM call. Evaluate at three levels:

Level	Question	Example metric
End-to-end (outcome)	did the task succeed?	task success rate, pass^k
Trajectory	was the path sound and efficient?	trajectory correctness, step count, no needless loops
Component	which tool / retriever / sub-agent broke?	tool-call accuracy, tool-arg validity

Tool-call accuracy — did it pick the right tool, with valid arguments, in the right order? A correct final answer reached by a broken path is luck, not reliability.
Trajectory correctness — compare the sequence of actions against a reference trajectory (exact, or judge-scored for “reasonable path”).

pass^k reliability: agents must be judged over MANY runs, not one

Agents are stochastic, so a single run is a coin flip dressed as a measurement. Two metrics:

pass@k — succeeds in at least one of k attempts (measures potential / best-case).

pass^k — succeeds in all k attempts (measures reliability / worst-case).

For anything users depend on, pass^k is the honest metric — “works if you’re lucky once” is not shippable. A 2025/2026 study ran 10 independent runs of agents on SWE-Bench-Verified (60k trajectories): single-run pass@1 estimates varied by 2.2–6.0 percentage points depending on which run you happened to observe, with best-vs-worst gaps up to 24.9 points — even at temperature 0. A single-run agent number is not a measurement.

12. Statistical rigor: the step everyone skips

Eval scores are estimates from a sample, and samples are noisy. Skipping the stats is how teams declare false “wins.”

Confidence intervals, always. “78%” means nothing; “78% ± 4% (95% CI)” tells you whether a change from 75% is real. Report the interval, not the point.
Sample size. A 50-case set can’t resolve a 2-point improvement — the CI is wider than the effect. Size the set to the effect you need to detect. Tail categories need their own adequate n, or a regression there hides inside the aggregate.
Significance, not vibes. Before claiming B beats A, check the difference exceeds the noise (overlapping CIs, or a paired test on the same cases). Many “improvements” vanish under a re-run.
Run-to-run variance. Run the eval multiple times; if the score swings more than your claimed gain, your gain is noise (this is exactly the §11 agent finding).

The false-win failure mode

Team tweaks a prompt, score goes 75% → 78% on a 50-case set, ships it as an improvement. It was sampling noise; the real change was zero (or negative). Without confidence intervals and adequate sample size you will ship noise as progress and chase phantom regressions. This is the single most common eval malpractice.

13. Pros / cons of the eval approaches

Approach	Pros	Cons
Exact/schema match	cheap, objective, deterministic	only for checkable outputs; ignores quality
LLM-as-judge	scales, handles open-ended quality	judge bias; cost; needs calibration
RAGAS	pinpoints retrieval vs generation faults	judge-based under the hood → same bias caveats
Human eval	gold standard, catches the subtle	slow, costly, rater noise
Public benchmarks	comparable across models	contamination; rarely match your task
Agent trajectory + pass^k	reflects real reliability	many runs = expensive; needs reference trajectories

14. Caveats & risk mitigation

The judge is a model, not an oracle. It has biases (§6) and gets re-versioned like any model. Calibrate against humans and re-validate on every judge change. An uncalibrated judge gives you confident, precise, wrong numbers.
Your eval set decays. As the product and traffic shift, an old golden set stops representing reality. Continuously harvest production failures into it; prune stale cases. A frozen eval set slowly lies.
Don’t optimize to the eval. If you tune the system to pass a fixed set, you overfit and it stops predicting real-world quality (Goodhart’s law). Keep a hold-out you never optimize against.
Contamination cuts both ways. Public-benchmark contamination inflates model claims; your-own-set contamination inflates your claims. Keep a private, dated hold-out (§8).
Single-run numbers lie for stochastic systems. Agents especially — always pass^k over multiple runs with CIs (§11, §12).
Cost. LLM-as-judge and multi-run agent evals burn tokens; budget them against 15 - Cost, Latency & Performance and cache where the eval set is stable.

15. In practice: an LLM-as-judge sketch

# Rubric-separated, pairwise judge with position-bias mitigation.
# Run BOTH orders; if the verdict flips, the case is unreliable -> drop or flag.
 
import json
 
JUDGE_PROMPT = """You are a strict evaluator. Compare two answers to the QUESTION.
Score EACH on these criteria (0-5) and justify in one line BEFORE the score:
  - accuracy: factually correct vs the REFERENCE
  - grounding: claims supported, no invented facts
  - completeness: addresses the full question
  - relevance: on-topic, no padding (ignore length when scoring)
 
Return STRICT JSON:
{"a":{"accuracy":n,"grounding":n,"completeness":n,"relevance":n,"why":"..."},
 "b":{...same...},
 "winner":"A"|"B"|"tie"}
QUESTION: {q}
REFERENCE: {ref}
ANSWER A: {a}
ANSWER B: {b}"""
 
def judge_once(client, judge_model, q, ref, ans_a, ans_b):
    msg = JUDGE_PROMPT.format(q=q, ref=ref, a=ans_a, b=ans_b)
    out = client.chat.completions.create(
        model=judge_model,          # different family than models under test (anti self-preference)
        messages=[{"role": "user", "content": msg}],
        temperature=0,              # judge should be as stable as possible
    )
    return json.loads(out.choices[0].message.content)
 
def judge_pairwise(client, judge_model, q, ref, cand_x, cand_y):
    # Order 1: X=A, Y=B   |   Order 2: swapped, to cancel position bias
    r1 = judge_once(client, judge_model, q, ref, cand_x, cand_y)
    r2 = judge_once(client, judge_model, q, ref, cand_y, cand_x)
    win1 = r1["winner"]                                  # in terms of A/B
    win2 = {"A": "B", "B": "A", "tie": "tie"}[r2["winner"]]  # remap swapped slots
    if win1 != win2:
        return {"verdict": "unreliable", "reason": "flipped on swap"}  # position-sensitive -> distrust
    verdict = {"A": "X", "B": "Y", "tie": "tie"}[win1]
    return {"verdict": verdict, "detail": [r1, r2]}
 
# Calibration loop (do this BEFORE trusting any judge number):
#   1. Take a human-graded sample of pairwise verdicts.
#   2. Run judge_pairwise on the same pairs.
#   3. Compute agreement (e.g. Cohen's kappa). Low agreement -> fix rubric/judge,
#      do NOT report eval scores yet. Re-run on every judge-model or prompt change.

What this sketch buys you

Position bias cancelled (both orders), self-preference reduced (cross-family judge), scores decomposed and justified (rubric-separated), and a hard rule to distrust cases where the verdict flips on swap. The missing piece — judge–human calibration — is sketched in the comment because it gates whether any of the numbers mean anything.

Sanket's Tutorials

Explorer

13 - Evaluation & Evals

1. Why evals at all: a non-deterministic component on shifting ground

2. Offline vs online evaluation

3. Golden / eval sets: the asset everything else depends on

4. Scoring methods: pick by what you can define

5. LLM-as-judge: modes and when to use each

6. Judge bias mitigation — and judging the judge

7. RAGAS: metrics for the RAG pipeline

8. Benchmark contamination

9. Human evaluation

10. Eval-driven development

11. Agent / trajectory evaluation

12. Statistical rigor: the step everyone skips

13. Pros / cons of the eval approaches

14. Caveats & risk mitigation

15. In practice: an LLM-as-judge sketch

Sources

Table of Contents

Backlinks

Graph View