Monitoring, Drift and Incident Response for ML Systems

A deployed model starts going stale on the day it ships. Customers change, fraudsters adapt, a partner renames a field, a sensor is recalibrated, a pandemic rewrites shopping habits. Unlike ordinary software, a model can fail silently: it keeps returning well-formed predictions that are steadily getting worse. Monitoring is how you notice, and a retraining and incident process is how you respond. This lesson covers what to watch, how to test for change statistically, what to do when labels arrive late, and how to document models so others can operate them.

Intuition: a map of a city that keeps changing

A trained model is a map drawn from a survey of the city at one point in time. Data drift is the city growing new neighbourhoods: the inputs you see now come from places the survey barely covered. Concept drift is the city renaming its streets: the same location now leads somewhere different, so the relationship between input and correct answer has changed. Your map can be fine for one and useless for the other. You can spot new neighbourhoods by looking at where travellers are coming from (inputs alone). You only learn that streets were renamed when travellers report getting lost (labels).

Kinds of drift, formally

Write the joint distribution of inputs $X$ and target $Y$ as $P(X, Y) = P(Y \mid X)\,P(X) = P(X \mid Y)\,P(Y)$. Training data came from $P_{\text{train}}$; production data comes from $P_{\text{prod}}$.

Terminology varies between authors; some use "data drift" for any change in $P(X)$ and "concept drift" for any change in $P(Y \mid X)$, which is what we do here. Drift can also be sudden (a schema change, a policy change), gradual (slowly ageing customers), recurring (seasonality) or incremental.

Important: drift in $P(X)$ does not always hurt. If the model already handles the new region well, performance holds. And concept drift can destroy performance while every input histogram looks unchanged. That is why input-drift monitoring is a smoke detector, not a performance measurement.

What to monitor: four layers

1. Service health: request rate, error rate, latency percentiles, saturation. Standard SRE monitoring, and the first thing to break.
2. Data quality: schema (columns, types), null rates, out-of-range values, unseen category values, row counts per batch, freshness of upstream tables. Most "model incidents" are actually data-pipeline incidents.
3. Distribution drift: of each important input feature and of the model's output scores, compared with a reference window (training data or a known-good recent window).
4. Model performance: the actual metric (accuracy, AUC, calibration, precision at the operating threshold) computed when labels arrive, plus the business metric the model exists to move (approval rate, loss rate, revenue), broken down by important segments.

Detecting drift: PSI

The Population Stability Index, borrowed from credit scoring, compares a feature's binned distribution in production with a reference. Divide the reference data into $B$ bins (often deciles of the reference). Let $e_i$ be the fraction of reference rows in bin $i$ and $a_i$ the fraction of production rows in the same bin. Then

$\text{PSI} = \sum_{i=1}^{B} (a_i - e_i)\,\ln\frac{a_i}{e_i}.$

Each term is non-negative, because $(a_i - e_i)$ and $\ln(a_i/e_i)$ always have the same sign. PSI equals the symmetric (Jeffreys) form of KL divergence between the two binned distributions: $\text{KL}(a\|e) + \text{KL}(e\|a)$. An empty bin makes the logarithm infinite, so add a small $\varepsilon$ to every fraction.

Commonly quoted industry thresholds: PSI below 0.1 means little change, 0.1 to 0.25 a moderate shift worth investigating, above 0.25 a major shift. These are conventions from practice, not statistical tests; calibrate them to your own features by looking at PSI between historical windows where you know nothing went wrong.

Worked example

A feature, applicant income, was split into quartiles on training data, so each reference bin holds $e_i = 0.25$. This week's applicants fall into the bins as $a = (0.10, 0.20, 0.30, 0.40)$: the population has shifted to higher incomes.

$\text{PSI} = 0.1374 + 0.0112 + 0.0091 + 0.0705 = 0.228.$

That is in the "moderate, investigate" band, close to "major". Next steps: check whether the shift is real (a marketing campaign targeting higher earners) or an artefact (income now reported monthly instead of yearly would push nearly everything into one extreme bin). Then check whether model performance in those bins has changed.

Detecting drift: the Kolmogorov–Smirnov test

For a continuous feature, the two-sample Kolmogorov–Smirnov (KS) test compares the empirical cumulative distribution functions (ECDFs) $F_{\text{ref}}$ and $F_{\text{prod}}$. Its statistic is the largest vertical gap between them:

$D = \sup_x \left| F_{\text{ref}}(x) - F_{\text{prod}}(x) \right|.$

Under the null hypothesis that both samples come from the same distribution, large $D$ is unlikely. For large samples of sizes $n$ and $m$, the null is rejected at level $\alpha$ when $D > c(\alpha)\sqrt{(n+m)/(nm)}$, with $c(0.05) \approx 1.358$. With $n = m = 1{,}000$ the critical value is $1.358 \times \sqrt{2000/10^6} = 0.061$.

The trap: production monitoring has huge samples. With a million rows per day, a KS test will flag shifts far too small to matter, and with dozens of features you will get false alarms from multiple testing alone. Practitioners therefore (a) look at the effect size ($D$ itself, PSI, or Wasserstein distance) rather than only the p-value, (b) subsample to a fixed size, (c) correct for multiple comparisons or alert only on important features, and (d) require a shift to persist across several windows. For categorical features use a chi-squared test or PSI over categories, and alert separately on brand-new categories.

import numpy as np
from scipy.stats import ks_2samp

def psi(reference, current, bins=10, eps=1e-6):
    # Bin edges from reference quantiles; open-ended outer bins catch new extremes.
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))[1:-1]
    e = np.bincount(np.searchsorted(edges, reference), minlength=bins) / len(reference)
    a = np.bincount(np.searchsorted(edges, current), minlength=bins) / len(current)
    e, a = e + eps, a + eps
    return float(np.sum((a - e) * np.log(a / e)))

rng = np.random.default_rng(0)
ref = rng.lognormal(mean=10.8, sigma=0.5, size=50_000)    # training-time incomes
cur = rng.lognormal(mean=11.0, sigma=0.5, size=20_000)    # this week's incomes

print("PSI:", round(psi(ref, cur), 3))
res = ks_2samp(rng.choice(ref, 2_000, replace=False), rng.choice(cur, 2_000, replace=False))
print("KS D:", round(res.statistic, 3), "p-value:", res.pvalue)

The function uses quantile bins from the reference so each reference bin holds about 10% of rows, and the outer bins are open-ended so new extreme values are counted rather than dropped. A shift of 0.2 in the log-mean (about 22% higher incomes) produces a PSI of about 0.15 and, on 2,000-row subsamples, a KS statistic of about 0.15 with a p-value far below 0.001 (exact values vary with the random seed).

Multivariate drift. Per-feature tests miss changes in the relationships between features. Two practical approaches: train a domain classifier to distinguish reference rows from production rows (an AUC well above 0.5 means the distributions differ, and its feature importances show where), or compare distributions in a learned embedding space. Rabanser, Günnemann and Lipton's "Failing Loudly" study compares such methods empirically and is worth reading.

Monitor the outputs too. The distribution of predicted scores is one of the most useful drift signals: it compresses all features into what the model actually uses. A jump in the share of high-risk scores or a collapse in the predicted positive rate is often the first sign of trouble.

Performance monitoring when labels are late

Labels often arrive late or never: chargebacks take weeks, loan defaults take months or years, and recommendation "labels" are only for items shown. Strategies:

• Join predictions to labels when they arrive. Log every prediction with a request ID, timestamp, model version and the features used, then compute metrics on the labelled cohort as it matures. Report performance by prediction date, so you can see the cohort from three weeks ago once its labels are in.
• Use proxy labels: early signals that correlate with the true outcome (a manual-review verdict before the chargeback, a 30-day delinquency before the default, a click before a purchase). Validate the proxy's correlation with the real label periodically.
• Label a sample: send a small random sample of predictions to human reviewers every week. Random, not only uncertain cases, or your estimate is biased.
• Estimate performance without labels, with care. If the model is still well calibrated on the new data (plausible only under mild covariate shift), its expected accuracy on new data is the average of its confidence in its predicted class. This breaks down under concept drift, which is exactly when you most need it, so treat it as a hint.

Feedback loops complicate everything. A fraud model that blocks transactions never sees whether the blocked ones were really fraud; a recommender only gets clicks for items it chose to show. Over time the training data reflects the model's own decisions, which can lock in errors. Mitigations: keep a small randomised holdout or exploration slice where the model's decision is not applied or is randomised (within ethical and legal limits), log the propensity (probability of the action taken) so later training can correct for it, and be explicit about which outcomes are observed only for some decisions.

Alerting and retraining triggers

Good alerts are actionable, routed to an owner, and rare enough to be taken seriously.

• Page a human for service SLO breaches and data-quality failures that make predictions invalid (null rates spiking, schema change, a feature stuck at a constant).
• Open a ticket for moderate drift or slow metric decay.
• Use multiple windows (for example, alert only if PSI > 0.25 on two consecutive days) and severity tiers to reduce noise.
• Put the reference window, threshold and owner for each alert in configuration that is reviewed like code.

Retraining strategies, from simplest to most automated:

Whatever the trigger, retraining is not deployment. The new model passes the same evaluation gates and release process as any other (see the deployment lesson). And retraining cannot fix a data bug: if the drift is caused by a broken pipeline, retraining on the broken data teaches the model the bug. Diagnose first.

Incident response for ML systems

When a model misbehaves in production, borrow from site reliability engineering:

1. Detect and declare. Someone owns the incident and a channel is opened.
2. Mitigate first. Roll back to the previous model, switch to a rule-based fallback, raise the human-review share, or turn the feature off. Have these switches ready before you need them.
3. Diagnose. Is it the service, the data, or the model? Check upstream data freshness and schema, recent deploys (code, model, features, config), and drift reports. Most incidents are found in the data.
4. Fix and verify with the same evaluation gates.
5. Blameless post-mortem: timeline, root cause, why detection took as long as it did, and action items (a new data check, a new alert, a test).

Model documentation: model cards

A model card (Mitchell et al., 2019) is a short, structured document shipped with a model. It records: model details (version, owner, architecture, training date); intended use and out-of-scope uses; the factors (groups, conditions, instruments) across which performance may vary; metrics and decision thresholds; evaluation and training data; quantitative results disaggregated by those factors; ethical considerations; and caveats. For operations, add the monitoring plan: which features and metrics are watched, the reference window, thresholds, the retraining policy and the rollback plan. Datasheets for datasets play the same role for training data. Several governance frameworks and regulations expect documentation of this kind (see the governance lesson); keep it versioned next to the model in the registry, not in a slide deck.

Common pitfalls

• Monitoring only inputs. Input drift is neither necessary nor sufficient for performance loss. Monitor outputs and, when labels arrive, real metrics.
• Trusting p-values at scale. Large samples make trivial shifts "significant"; use effect sizes and persistence.
• Retraining on broken data instead of fixing the pipeline.
• Reference window mismatch: comparing a December week with a training set from July and calling seasonality "drift". Use a seasonally comparable reference too.
• No prediction logging, so labels can never be joined back and incidents cannot be reconstructed.
• Aggregate-only metrics that hide a collapse in one segment (a new region, a new device type).
• Ignoring feedback loops, so the model gradually trains on its own decisions.
• Alert fatigue: dozens of noisy drift alerts that everyone mutes.

In practice

• Start with data-quality checks and output-distribution monitoring; they catch most incidents cheaply.
• Log every prediction with model version, features and a request ID, and build the label join on day one.
• Compute PSI or a distance per important feature against both the training reference and a recent rolling window; alert on persistent, large shifts.
• Track the business metric and segment-level performance, not only global accuracy.
• Keep fallbacks (previous model, rules, human review) ready, and rehearse switching to them.
• Write a model card with a monitoring section before launch, and update it with each retrain.
• Open-source tools exist for drift reports and data validation (for example Evidently, NannyML, Great Expectations), as do monitoring features in cloud ML platforms; the statistics underneath are the ones in this lesson.