Model Evaluation and Metrics

A model is only as good as the evidence that it works. Evaluation decides whether a model ships, which of two models wins, what threshold it runs at, and whether a promised improvement is real or noise. Many failed ML projects did not have bad models; they had good-looking numbers produced by the wrong metric, a leaky split, or a test set used one time too many. This lesson covers the measurements professionals rely on and the judgment needed to pick among them.

Intuition: the metric is the contract

Think of a metric as the contract between the model and the business. A fraud model "scoring 95% accuracy" sounds excellent until you learn that 95% of transactions are legitimate, so a model that never flags anything also scores 95%. The contract was badly written. Good evaluation starts by asking what an error costs, who bears it, and how the predictions will be used (ranked? thresholded? summed into a forecast?), then chooses metrics that reward exactly that behaviour.

Classification metrics

The confusion matrix

For a binary classifier with a positive class (say, fraud), every prediction falls into one of four cells:

From these counts:

$\text{Accuracy} = \frac{TP+TN}{TP+TN+FP+FN}, \quad \text{Precision} = \frac{TP}{TP+FP}, \quad \text{Recall} = \frac{TP}{TP+FN},$

$\text{Specificity} = \frac{TN}{TN+FP}, \quad \text{FPR} = 1 - \text{Specificity}, \quad F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}.$

Precision answers "when the model raises an alarm, how often is it right?" Recall (also called sensitivity or true positive rate, TPR) answers "of all real positives, how many did we catch?" $F_1$ is their harmonic mean, which is dragged down by whichever is smaller. The general $F_\beta = (1+\beta^2)\frac{PR}{\beta^2 P + R}$ weights recall $\beta$ times as much as precision.

Worked example

1,000 transactions, 50 of them fraudulent. The model flags 100; 40 of those are fraud.

• $TP = 40$, $FP = 60$, $FN = 50 - 40 = 10$, $TN = 950 - 60 = 890$.
• Accuracy $= (40 + 890)/1000 = 0.93$.
• Precision $= 40/100 = 0.40$. Recall $= 40/50 = 0.80$.
• $F_1 = 2(0.40)(0.80)/(1.20) \approx 0.533$.
• Specificity $= 890/950 \approx 0.937$, so FPR $\approx 0.063$.

The trivial "never fraud" model has accuracy $950/1000 = 0.95$, higher than our model, with recall 0. Under imbalance, accuracy is nearly useless; report precision, recall and a cost.

For more than two classes, compute per-class metrics and average them: macro averaging weights classes equally (good when rare classes matter), weighted averaging weights by class frequency, and micro averaging pools all counts (for single-label multiclass problems, micro-F1 equals accuracy).

Thresholds, ROC and PR curves

Most classifiers output a score or probability $\hat p$; the label comes from a threshold $t$ (predict positive if $\hat p \ge t$). Every threshold gives a different confusion matrix, so a single precision or recall number describes one operating point. Curves describe all of them.

• The ROC curve plots TPR (recall) against FPR as $t$ sweeps from 1 to 0. ROC-AUC, the area under it, equals the probability that a randomly chosen positive is scored higher than a randomly chosen negative. Random scoring gives 0.5; perfect ranking gives 1.0.
• The precision-recall (PR) curve plots precision against recall. Its summary, average precision (AP, a step-wise estimate of PR-AUC), has a random baseline equal to the positive rate, not 0.5.

Why the difference matters: FPR divides by the number of negatives. When negatives vastly outnumber positives (say 1 fraud per 1,000), a model can produce many times more false alarms than true detections while FPR stays tiny, so ROC-AUC looks excellent. Precision exposes this because its denominator contains the false positives directly. Saito and Rehmsmeier (2015) showed empirically that PR plots are more informative than ROC plots on imbalanced data. Rules of thumb:

Calibration

A model is calibrated if, among all cases where it predicts probability $p$, the fraction that are positive is about $p$. Ranking metrics like AUC ignore calibration entirely: multiply every score by 0.5 and AUC is unchanged. But calibration matters whenever probabilities are used as probabilities: expected-cost decisions, risk scores shown to clinicians, combining models, or the threshold formula below.

Tools:

• Reliability diagram: bin predictions (e.g. 10 bins), plot mean predicted probability against observed positive rate; a calibrated model lies on the diagonal.
• Brier score: $\frac{1}{n}\sum_i (\hat p_i - y_i)^2$, a proper scoring rule (lower is better). Log loss is another.
• Expected calibration error (ECE): $\sum_{b} \frac{|B_b|}{n}\,\left|\overline{y}_{B_b} - \overline{\hat p}_{B_b}\right|$ over bins $B_b$. It depends on the binning, so treat it as a diagnostic.

Fixes: Platt scaling (fit a logistic regression on the scores) and isotonic regression (a monotone step function, needs more data), both available through CalibratedClassifierCV, and temperature scaling for neural networks (Guo et al., 2017, found that modern deep networks are often overconfident). Always fit calibrators on data not used to train the model.

Regression metrics

With residuals $e_i = y_i - \hat y_i$:

$\text{MAE} = \frac{1}{n}\sum_i |e_i|, \qquad \text{RMSE} = \sqrt{\frac{1}{n}\sum_i e_i^2}, \qquad \text{MAPE} = \frac{100\%}{n}\sum_i \left|\frac{e_i}{y_i}\right|.$

Worked example. True values $y = (100, 200, 300, 50)$, predictions $\hat y = (110, 180, 330, 40)$. Residuals: $-10, 20, -30, 10$.

• MAE $= (10+20+30+10)/4 = 17.5$.
• MSE $= (100+400+900+100)/4 = 375$, so RMSE $= \sqrt{375} \approx 19.36$.
• MAPE $= \frac{1}{4}(0.10 + 0.10 + 0.10 + 0.20) = 12.5\%$.

How to choose:

• MAE is in the target's units, robust to outliers, and is minimized by predicting the conditional median.
• RMSE penalizes large errors quadratically, is minimized by the conditional mean, and is the right choice when big misses are disproportionately costly. RMSE is always at least MAE; a large gap signals a few big errors.
• MAPE is scale-free and easy to explain, but it is undefined at $y=0$, explodes near zero, and is asymmetric: for positive targets and non-negative forecasts, an under-forecast can contribute at most 100%, while an over-forecast is unbounded, so optimizing MAPE biases forecasts low. Alternatives include weighted MAPE ($\sum|e_i| / \sum|y_i|$) and MASE (error scaled by a naive baseline's error), common in forecasting.
• $R^2$ $= 1 - \text{SSE}/\text{SST}$ is the fraction of variance explained relative to predicting the mean. It can be negative on test data.

Cross-validation

A single train/test split gives one noisy estimate. k-fold cross-validation splits the training data into $k$ folds, trains on $k-1$ and validates on the remaining one, rotating $k$ times, and reports the mean and spread of the $k$ scores. $k=5$ or $10$ is standard.

Pick the splitter to match how the model will meet data in production:

Using shuffled k-fold on time series or on grouped data leaks information and inflates scores, sometimes dramatically. Hyperparameter tuning adds another trap: if you pick the best configuration by its cross-validation score, that score is optimistically biased. Report the final number on an untouched test set, or use nested cross-validation (an inner loop for tuning, an outer loop for estimation).

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate

X, y = make_classification(n_samples=5000, weights=[0.95, 0.05],
                           n_informative=5, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
res = cross_validate(LogisticRegression(max_iter=1000), X, y, cv=cv,
                     scoring=["accuracy", "roc_auc", "average_precision", "f1"])
for m in ["accuracy", "roc_auc", "average_precision", "f1"]:
    s = res[f"test_{m}"]
    print(f"{m:18s} {s.mean():.3f} +/- {s.std():.3f}")

On this 5%-positive synthetic dataset you will typically see high accuracy and ROC-AUC alongside markedly lower average precision and F1, the imbalance story in numbers.

Bias, variance and learning curves

For squared error, the expected test error of a learning algorithm at a point $x$ decomposes as

$\mathbb{E}\big[(y - \hat f(x))^2\big] = \underbrace{\big(f(x) - \mathbb{E}[\hat f(x)]\big)^2}_{\text{bias}^2} + \underbrace{\mathbb{E}\big[(\hat f(x) - \mathbb{E}[\hat f(x)])^2\big]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{noise}},$

where $f$ is the true function, $y = f(x) + \varepsilon$ with noise variance $\sigma^2$, and expectations are over training sets (and noise). Bias is systematic error from a model too simple to capture $f$; variance is sensitivity to the particular training sample; noise is irreducible. Increasing model flexibility typically lowers bias and raises variance. (Very large neural networks can show "double descent," where test error falls again past the interpolation point, so treat the classic U-curve as a guide rather than a law.)

Learning curves plot training and validation scores against training-set size:

• High bias (underfitting): both curves converge to a poor score with a small gap. More data will not help; add capacity or better features.
• High variance (overfitting): training score is much better than validation, and the gap narrows as data grows. More data, regularization or a simpler model will help.

sklearn.model_selection.learning_curve and validation_curve (score against a hyperparameter) produce these directly.

Comparing models statistically

If model B beats model A by 0.4 points of accuracy, is that real? Some tools:

• Look at the spread. Compare fold-by-fold (paired) scores; if B wins on only 3 of 10 folds, the mean difference is not convincing.
• McNemar's test for two classifiers on the same test set. Let $b$ = cases A got right and B got wrong, $c$ = the reverse. Under the null of equal error rates, $\chi^2 = \frac{(|b-c|-1)^2}{b+c}$ approximately follows a chi-squared distribution with 1 degree of freedom. Example: $b = 30$, $c = 15$ gives $\chi^2 = 14^2/45 \approx 4.36$, above the 5% critical value 3.84 ($p \approx 0.037$).
• Paired tests over CV folds need care, because folds share training data and the scores are correlated, which makes a naive paired t-test overconfident. Dietterich (1998) analysed this problem and proposed the 5×2 cv paired t-test; corrected resampled t-tests are another option.
• Bootstrap confidence intervals on the test-set metric difference are simple and work for any metric.

Also budget for the multiple-comparisons problem: if you try 50 variants and pick the best on the same validation set, some of its edge is luck.

Choosing metrics that match business costs

Assign a cost to each cell of the confusion matrix and choose the threshold that minimizes total cost. Continuing the fraud example, suppose a missed fraud (FN) costs 500 dollars and a false alarm (FP) costs 10 dollars of analyst time. Three candidate thresholds:

The lenient threshold has the worst precision ($45/245 \approx 0.18$) yet the lowest cost. F1 would have picked the opposite: it is about 0.63 at the strict threshold (precision $30/45$, recall $0.60$), 0.53 at the medium one, and 0.31 at the lenient one, so maximizing F1 selects the most expensive option here. F1 encodes an implicit cost ratio that rarely matches the real one.

With calibrated probabilities there is a closed form. Predict positive when the expected cost of saying "negative" exceeds that of saying "positive": $p \cdot C_{FN} > (1-p)\cdot C_{FP}$, i.e.

$t^* = \frac{C_{FP}}{C_{FP} + C_{FN}} = \frac{10}{10 + 500} \approx 0.0196 .$

Real constraints also matter: if analysts can review only 120 alerts a day, the operating point is fixed by capacity, and the relevant metric becomes precision (or fraud value caught) in the top 120.

import numpy as np
from sklearn.metrics import confusion_matrix

def best_threshold(y_true, p, c_fp=10, c_fn=500):
    best = None
    for t in np.linspace(0.001, 0.999, 999):
        tn, fp, fn, tp = confusion_matrix(y_true, p >= t, labels=[0, 1]).ravel()
        cost = c_fp * fp + c_fn * fn
        if best is None or cost < best[1]:
            best = (t, cost)
    return best  # tune on validation data, report cost on test data

Common pitfalls

• Reporting accuracy on imbalanced data, or ROC-AUC alone when positives are rare.
• Tuning the threshold or hyperparameters on the test set, then reporting the test score as if it were unbiased.
• Leaky cross-validation: shuffling time series, splitting rows of the same customer across folds, or fitting scalers/encoders on all data before splitting.
• Ignoring variance: quoting a single split's score without a confidence interval or fold spread.
• Treating scores as probabilities without checking calibration, especially after resampling or class weighting, both of which distort predicted probabilities.
• Using MAPE on data with zeros or near-zero targets.
• Optimizing a proxy forever: offline metrics should be validated against the online outcome (A/B test, business KPI) at least once.

In practice

• Write down the cost of each error type and the operating constraint (review capacity, latency, regulatory requirements) before choosing metrics.
• Report a small, fixed panel: one ranking metric (ROC-AUC or AP), the metrics at the chosen threshold, a calibration check, and the business cost, each with fold spread or bootstrap intervals.
• Use stratified CV for classification, grouped CV whenever entities repeat, and time-based splits for anything temporal, including a final out-of-time test.
• Keep a locked test set touched once per major decision; use cross-validation or a validation set for everything else.
• Slice metrics by segment (region, device, customer type). Aggregate scores can hide a subgroup where the model fails, which matters for fairness as well as accuracy.
• For LLM and generative systems the same principles apply with different measurements; see the evaluation lessons in AI 300 and AI 330.