Evaluating AI Systems

AI features fail quietly. A prompt tweak that fixes one complaint can break ten cases nobody rechecked, and a model update can shift behaviour overnight. Evaluation, often shortened to evals, is how you replace "it seemed fine when I tried it" with evidence.

Start with a test set

A test set is a collection of realistic inputs that represent what your system will face. Good sources:

• Real user requests (with personal data removed).
• Known hard cases and past failures.
• Edge cases: empty input, very long input, other languages, ambiguous questions, requests that should be refused.

Start with 50 to 200 cases; a small set you actually run beats a large one you don't. Keep some cases aside that you never look at while tuning prompts, so you can detect overfitting to your own test set.

Golden answers and how to score

For each case, decide what "good" means. A golden answer (or reference answer) is the expected output, written or approved by someone who knows the domain. Scoring methods, from most to least objective:

Use the most objective method that captures what matters. Many systems combine them: code checks for format, a judge for quality, and periodic human review to keep the judge honest.

For RAG systems, evaluate retrieval separately (did the right passage appear in the top results?) from generation (was the answer faithful to the passages?).

LLM-as-judge and its biases

An LLM judge is a model prompted to grade another model's output against a rubric, or to pick the better of two answers. Research on this approach found strong judges can agree with human preferences about as often as humans agree with each other, but also documented systematic biases:

• Position bias: when comparing two answers, favouring whichever is shown first (or second).
• Verbosity bias: favouring longer answers even when they aren't better.
• Self-enhancement bias: favouring outputs written by the same model or model family.
• Limited ability to grade maths and reasoning it cannot itself do reliably.

Mitigations: swap the order and judge twice; use specific rubrics ("Does the answer state the refund deadline? yes/no") rather than "rate 1 to 10"; ask for a short justification before the score; use a different model family as judge where practical; and check the judge against a sample of human ratings before trusting it.

Regression-testing prompts

Treat prompts, model choices and retrieval settings like code. Every change runs the whole test set, and you compare scores before and after. This is regression testing: making sure fixes don't break what used to work. Keep prompts in version control and store each run's outputs so you can see exactly which cases changed.

Beware of noise. Worked example: prompt A scores 172 of 200 (86%) and prompt B scores 181 of 200 (90.5%). With 200 cases, the margin of error on each score is roughly plus or minus 4.5 percentage points at 95% confidence (using the standard approximation of 1.96 × √(0.88 × 0.12 ÷ 200) ≈ 0.045). So a 4.5-point gain is suggestive, not proof. Look at which specific cases flipped, run variable outputs more than once, and grow the test set where decisions hinge on small differences.

Measure cost and latency too

A more accurate answer that costs five times as much or takes 30 seconds may be worse overall. For every eval run, record:

• Tokens and cost per task, including all tool-loop steps and retries.
• Latency: time to first token and total time, reported as the median and the 95th percentile (the time within which 95% of requests finish), because averages hide slow outliers.
• Failure rate: timeouts, invalid outputs, tool errors.

Then compare options on all three axes: quality, cost and speed.

Red-teaming

Red-teaming means deliberately trying to make the system misbehave, the way an attacker or a careless user might. Typical targets:

• Prompt injection through documents, web pages or emails the system reads.
• Extracting the system prompt, other users' data or secrets.
• Getting tools to act outside their intended scope.
• Producing harmful, biased or off-brand content.

Turn every successful attack into a permanent test case so the fix stays fixed.

Monitoring in production

Test sets cannot anticipate everything users do. In production, log inputs, outputs, tool calls, costs and latency (respecting privacy rules), and watch for:

• Quality signals: thumbs up or down, edits users make to drafts, escalations to a human, repeated rephrasing.
• Drift: changes in the kinds of questions asked, or behaviour changes after a provider updates a model.
• Cost and latency spikes, often from runaway agent loops or bloated context.

Sample production conversations regularly for review, and feed interesting failures back into the test set. That cycle (collect, evaluate, fix, re-test) is how AI systems get steadily better instead of randomly different.

Going deeper

An eval score is an estimate

Suppose your system has some true success rate $p$ on the kind of traffic your test set represents. Running $n$ independent cases and observing $s$ successes gives the estimate $\hat{p} = s/n$, whose standard error is

The familiar 95% interval $\hat{p} \pm 1.96\,\text{SE}$ is what the earlier worked example used. It behaves poorly when $\hat{p}$ is near 0 or 1 or $n$ is small, so a safer default is the Wilson interval, which the code below computes. For 172 of 200 it gives about 80.5% to 90.1%; for 181 of 200, about 85.6% to 93.8%. The intervals overlap heavily, which matches the earlier conclusion that the gap is suggestive, not proven.

How many cases do you need? Rearranging the standard error gives the sample size for a desired margin $E$:

To measure a system near 85% accuracy to within ±3 points: $n \approx 3.8416 \times 0.85 \times 0.15 / 0.0009 \approx 545$ cases. For ±5 points, about 196. This is why 50 to 200 cases is a good start for catching big regressions but not enough to resolve small differences.

Compare on the same cases: paired analysis

When two prompts are run on the same test cases, most cases come out the same for both. Only the cases where they disagree carry information about which is better. Count $b$ = cases only A got right and $c$ = cases only B got right. Under the null hypothesis that A and B are equally good, each disagreement is equally likely to go either way, so $\min(b, c)$ follows a binomial distribution with $b + c$ trials and probability one half. That is McNemar's test, and it is usually much more sensitive than comparing two independent intervals.

Worked example. In the 172 versus 181 comparison, suppose 166 cases were right for both, 13 wrong for both, $b = 6$ and $c = 15$. The net gain is $c - b = 9$ cases, which is the 4.5 points. The exact two-sided McNemar p-value is about 0.08, so even the paired test does not reach the conventional 0.05 threshold, but it tells you precisely where to look: read the 21 flipped cases. If the 15 wins are real improvements and the 6 losses are noise or label errors, ship B; if the 6 losses are severe (for example, wrong refund amounts), the average hides a real regression.

import math, random

def wilson_ci(successes, n, z=1.96):
    p = successes / n
    denom = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    half = z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return centre - half, centre + half

def mcnemar_exact(b, c):
    """Two-sided exact p-value; b, c = discordant pair counts."""
    n, k = b + c, min(b, c)
    tail = sum(math.comb(n, i) for i in range(k + 1)) / 2**n
    return min(1.0, 2 * tail)

def paired_bootstrap(a, b, iters=10_000, seed=0):
    """a, b: per-case 0/1 scores on the SAME cases. 95% CI of mean(b - a)."""
    rng, n = random.Random(seed), len(a)
    diffs = []
    for _ in range(iters):
        idx = [rng.randrange(n) for _ in range(n)]
        diffs.append(sum(b[i] - a[i] for i in idx) / n)
    diffs.sort()
    return diffs[int(0.025 * iters)], diffs[int(0.975 * iters)]

print(wilson_ci(172, 200), wilson_ci(181, 200))
print(mcnemar_exact(6, 15))                      # about 0.078
a = [1]*166 + [1]*6 + [0]*15 + [0]*13            # prompt A per case
b = [1]*166 + [0]*6 + [1]*15 + [0]*13            # prompt B, same cases
print(paired_bootstrap(a, b))                    # roughly (0.0, 0.09)

The paired bootstrap resamples test cases with replacement and recomputes the score difference each time; the middle 95% of those differences is a confidence interval that works for any metric, including judge scores and costs, not just pass or fail. Here its lower end sits at about zero, consistent with the McNemar result.

Is the judge trustworthy? Measuring agreement

Before relying on an LLM judge, have humans label a sample of the same outputs and measure agreement. Raw agreement overstates quality when one label dominates, so use Cohen's kappa, which corrects for agreement expected by chance:

where $p_o$ is the observed agreement and $p_e$ is the agreement expected if the judge and the human labelled independently at their own base rates. Worked example on 100 outputs: both say "pass" on 40, both say "fail" on 45, the judge passes 10 that the human fails, and fails 5 that the human passes. Then $p_o = 0.85$. The judge passes 50% of outputs and the human 45%, so $p_e = 0.5 \times 0.45 + 0.5 \times 0.55 = 0.50$, and $\kappa = (0.85 - 0.50)/(1 - 0.50) = 0.70$. That is substantial agreement, but notice the direction of the errors: the judge is more lenient than the human (10 false passes versus 5 false fails). Tighten the rubric where false passes cluster, then re-measure.

Behavioural tests, not just averages

A single average can hide systematic failures. Borrowing from software testing, write small suites that each check one capability or invariance:

• Minimum functionality tests: simple cases that must always pass (a refund question must mention the refund window).
• Invariance tests: changes that should not change the answer (swapping a customer's name, adding a typo).
• Directional tests: changes that should move the answer in a known direction (adding "urgent" should not lower priority).

Report pass rates per suite. A model can improve its overall score while getting worse on one suite that matters, and per-suite reporting makes that visible.

Common pitfalls

• Testing on what you tuned on. If you edited prompts while looking at every test case, the score is optimistic. Keep a held-out set.
• Comparing runs with different settings. Changing temperature, model version or retrieval index between runs confounds the comparison; record the full configuration with each result.
• Single runs of stochastic systems. At non-zero temperature, run each case several times or accept wider error bars.
• Unvalidated judges. A judge that has never been compared with human labels may be measuring length or style, not correctness.
• Averages only. Look at failures by category and at the individual cases that flipped.
• Forgetting cost and latency. A quality gain that doubles p95 latency may be a net loss for users.

In practice

• Build a 50 to 200 case test set from real usage before changing anything.
• Score with code checks where possible, a validated LLM judge where necessary, and humans periodically.
• Run the full set on every prompt, model or retrieval change, and inspect the cases that flipped.
• Track cost and p95 latency alongside quality.
• Red-team before launch, monitor after, and keep turning failures into tests.