Benchmarks and Evaluation

Every model launch arrives with a table of benchmark scores. Those numbers are useful, but they are easy to over-read. Knowing what each kind of test measures, and how scores get distorted, helps you choose models on evidence rather than headlines.

What a benchmark is

A benchmark is a fixed set of tasks with a scoring rule, run the same way on different models so their results can be compared. Some are multiple-choice exams; some check an exact answer; some run code against hidden tests; some put an AI agent into a simulated environment and check the outcome.

An analogy: a standardised exam for students. It tells you something real and comparable, but a high score does not guarantee someone will be good at your particular job, and students can be coached to the test.

What benchmarks measure

Benchmarks roughly fall into four families. These are well-known examples, described as published by their authors:

Knowledge

• MMLU (Massive Multitask Language Understanding, 2020): four-option multiple-choice questions across 57 subjects, from maths to law to history.

Reasoning and expert problem-solving

• GPQA Diamond: 198 hard multiple-choice questions in biology, physics and chemistry, written to be "Google-proof". Its authors reported that domain experts scored about 65% and skilled non-experts with web access about 34%.
• Humanity's Last Exam (2025): 2,500 questions across more than a hundred subjects, written by nearly 1,000 subject experts and organised by the Center for AI Safety and Scale AI. About a quarter are multiple-choice; the rest need an exact answer.

Coding

• SWE-bench Verified: 500 real GitHub issues from Python projects, screened by human developers to remove unclear or unfair tasks. The model gets the code and the issue text, writes a fix, and passes only if hidden tests confirm the bug is fixed and nothing else broke.

Agentic

• τ-bench (tau-bench): an AI agent handles simulated customer-service conversations in retail and airline settings, using tools and following a policy document while a second model plays the customer. It also measures consistency: "pass^k" is the chance the agent succeeds on all k repeated attempts. In the original 2024 paper, the strongest agent tested (built on GPT-4o) solved about 61% of retail tasks on a single attempt, but under 25% when it had to succeed on all eight attempts, a reminder that one good run is not reliability.

Contamination

Contamination happens when benchmark questions or answers end up in a model's training data, often because they are posted online. The model may then recall answers rather than solve problems, inflating its score. Defences include keeping test sets private, writing fresh questions regularly, and checking whether a model does much worse on newly written questions of the same type.

Saturation

A benchmark is saturated when top models score so close to the maximum that differences stop meaning much, and remaining errors may simply be mistakes in the test itself. The authors of Humanity's Last Exam noted that models already exceeded 90% accuracy on MMLU, which is why they built a harder test. Every successful benchmark eventually saturates, so the names in launch tables change over time.

Leaderboards and human-preference arenas

Leaderboards collect benchmark scores for many models in one place. Check who runs them, which settings were used (reasoning effort, number of attempts, tools allowed) and whether scores were measured independently or reported by the model's maker.

Human-preference arenas take a different approach. The best known began as Chatbot Arena in 2023, became LMArena, and was renamed Arena in January 2026. A visitor types a prompt, two anonymous models answer, the visitor votes for the better one, and only then are the names revealed. Many votes are combined into a ranking.

Arenas capture something exams miss: what people actually prefer. But they have known weaknesses:

• Votes reflect style as well as substance; longer, confident, well-formatted answers can win even when they are wrong.
• Most prompts are short, casual chats, not your work.
• A 2025 paper, "The Leaderboard Illusion", reported that some providers privately tested many model variants before release and published only the best, and that large closed-model providers received a disproportionate share of the voting data. In April 2025 a Meta model ranked highly with a version different from the one released publicly, after which the arena updated its policies.

Why your own tests matter

No public benchmark matches your tasks, your documents and your standards. A small private test set is the most reliable guide you can get, and it cannot have leaked into anyone's training data.

Worked example: you want a model to summarise customer complaints.

1. Collect 30 real (anonymised) complaints, including a few hard ones: long, angry, multilingual, ambiguous.
2. Write down what a good summary must include, such as product, problem and requested action, in a short checklist.
3. Run each candidate model on all 30, with the same prompt and settings.
4. Score each output against the checklist; for 30 items this takes about an hour by hand.
5. Run it three times. If a model gets 27/30 once but varies widely between runs, that matters as much as the average.

Thirty examples will not settle small differences, but they will reliably reveal a model that misses key details or fails on your hardest cases. Keep the set and rerun it whenever you consider switching models.

Going deeper

A benchmark score is an estimate

Treat each question as a trial that the model passes ($c_i = 1$) or fails ($c_i = 0$). The benchmark score is the sample mean $\hat{p} = \frac{1}{n}\sum_i c_i$, an estimate of the model's true accuracy $p$ on questions like these. Its standard error is

and an approximate 95% confidence interval is $\hat{p} \pm 1.96\, \mathrm{SE}$ (a normal approximation that is reasonable when $n\hat{p}$ and $n(1-\hat{p})$ are both at least about 10).

Worked example. GPQA Diamond has 198 questions. A model scoring 70% has $\mathrm{SE} = \sqrt{0.7 \times 0.3 / 198} \approx 0.0326$, so the 95% interval is about $\pm 6.4$ percentage points: roughly 64% to 76%. A launch table showing 70% versus 68% on GPQA Diamond is not evidence of a difference. MMLU's test set has 14,042 questions, so at 85% the interval is about $\pm 0.6$ points, much tighter, though by now its errors are dominated by saturation and label noise rather than sampling.

Compare models on the same questions

Two models answer the same items, so their errors are correlated. The right analysis is paired: for each question compute $d_i = c_i^A - c_i^B \in \{-1, 0, 1\}$ and use

where $s_d$ is the sample standard deviation of the $d_i$. Questions both models get right or both get wrong contribute zero, so only the disagreements carry information, and the paired interval is usually much narrower than comparing two separate intervals. Miller (2024) gives this and related recipes for language-model evals, including clustered standard errors when questions come in groups (several questions about one passage are not independent).

How many questions do you need? To detect a true difference $\delta$ between two independent accuracies near $p$ with 5% significance and 80% power,

For $p = 0.8$ and $\delta = 0.03$: $7.84 \times 0.32 / 0.0009 \approx 2{,}790$ questions per model. Pairing reduces this, but the lesson stands: a 30-item private set finds big failures, not 3-point differences.

pass@k and pass^k

For code, many papers sample $n$ solutions per problem, count the $c$ that pass, and report pass@k, the chance at least one of $k$ samples passes. Chen et al. (2021) introduced the unbiased estimator

averaged over problems. With $n = 10$ and $c = 3$: pass@1 $= 1 - 7/10 = 0.30$, and pass@5 $= 1 - \binom{7}{5}/\binom{10}{5} = 1 - 21/252 \approx 0.917$. The same model looks very different depending on $k$, so always check which is reported. τ-bench's pass^k asks the opposite question, whether all $k$ attempts succeed, estimated by $\binom{c}{k} / \binom{n}{k}$: with 5 successes out of 8 trials, pass^2 $= 10/28 \approx 0.36$, and pass^8 is 0.

How arenas turn votes into ratings

Arena-style leaderboards fit a Bradley–Terry model, in which each model $i$ has a strength $\theta_i$ and

Strengths are fitted by logistic regression on the votes and reported on an Elo-like scale where a gap of $\Delta R$ points means $P = 1 / (1 + 10^{-\Delta R / 400})$. A 100-point gap means the higher model wins about 64% of head-to-head votes; a 20-point gap, about 53%, close to a coin flip. Published arena tables include confidence intervals for this reason; overlapping intervals mean the ranks are not settled.

Checking an automatic grader

Many evaluations use a model as the judge. Before trusting one, grade a sample by hand and measure agreement beyond chance with Cohen's kappa:

where $p_o$ is the observed agreement rate and $p_e$ the agreement expected by chance from each grader's label frequencies. Example: on 100 outputs, a human marks 60% as pass, the judge marks 65% as pass, and they agree on 85. Then $p_e = 0.60 \times 0.65 + 0.40 \times 0.35 = 0.53$ and $\kappa = (0.85 - 0.53) / (1 - 0.53) \approx 0.68$: substantial but imperfect agreement. Raw agreement of 85% sounds better than it is, because two graders that both say "pass" most of the time agree often by chance. Look at the disagreements too; judges often favour longer answers or whichever answer is shown first.

Code: intervals, paired differences and pass@k

import numpy as np
from math import comb

def accuracy_ci(correct):
    """Accuracy with a normal-approximation 95% confidence interval."""
    c = np.asarray(correct, dtype=float)
    p, se = c.mean(), c.std(ddof=1) / np.sqrt(len(c))
    return p, p - 1.96 * se, p + 1.96 * se

def paired_diff_ci(a, b):
    """95% CI for accuracy(A) - accuracy(B) on the SAME questions."""
    d = np.asarray(a, float) - np.asarray(b, float)   # each entry -1, 0 or 1
    se = d.std(ddof=1) / np.sqrt(len(d))
    return d.mean(), d.mean() - 1.96 * se, d.mean() + 1.96 * se

def pass_at_k(n, c, k):
    """Unbiased pass@k from n samples per problem, c of them correct."""
    return 1.0 if n - c < k else 1 - comb(n - c, k) / comb(n, k)

rng = np.random.default_rng(1)
skill = rng.uniform(0, 1, 198)                 # per-question difficulty
a = rng.random(198) < 0.25 + 0.6 * skill       # model A right/wrong
b = rng.random(198) < 0.22 + 0.6 * skill       # model B, slightly weaker
print("A: %.3f [%.3f, %.3f]" % accuracy_ci(a))
print("B: %.3f [%.3f, %.3f]" % accuracy_ci(b))
print("A-B: %.3f [%.3f, %.3f]" % paired_diff_ci(a, b))
print("pass@1, pass@5 for n=10, c=3:", pass_at_k(10, 3, 1),
      round(pass_at_k(10, 3, 5), 4))

In this simulation model A really is about 3 points better by construction, yet with 198 questions the paired 95% interval for the difference still includes zero. That is the normal situation on small benchmarks, and the reason to report intervals rather than single numbers.

Common pitfalls

• Reporting point scores without intervals. On benchmarks of a few hundred items, differences of several points are often noise.
• Comparing numbers from different settings. Prompt format, few-shot examples, reasoning effort, sampling temperature, number of attempts and tool access all change scores.
• Mixing pass@1 and pass@k. They answer different questions; pass@k with a large k flatters a model that has no way to choose its best sample.
• Tuning on the test set. Iterating prompts against your evaluation set overfits it; keep a held-out split.
• Trusting an LLM judge blindly. Model graders have position and length biases; check agreement with human labels on a sample.

In practice

• Read benchmark names, not just numbers: ask whether the test resembles your task.
• Be sceptical of results on saturated tests, and look for independent measurements.
• Treat arena rankings as a signal about general chat preference, not about accuracy on your work.
• Build a small private test set from real examples and check consistency across repeated runs.