The ML Lifecycle, Reproducibility and Experiment Tracking

Most ML projects that fail do not fail because the model was wrong; they fail because nobody can say which data, code and settings produced the model in production, why last week's result cannot be reproduced, or how to ship an improvement without breaking something. MLOps is the set of practices that makes ML work repeatable, auditable and deployable. This lesson opens the MLOps & LLMOps course with the foundation everything else rests on: the lifecycle, reproducibility, experiment tracking, versioning and automation.

Intuition: a lab notebook and a flight recorder

A good scientist keeps a lab notebook: every experiment's materials, procedure, conditions and results, written so that someone else could repeat it. A good airline keeps a flight recorder: an automatic, tamper-evident log of what happened, so that when something goes wrong you can reconstruct why.

ML engineering needs both. Experiment tracking is the lab notebook, recording what you tried and what happened. Versioning and lineage are the flight recorder, letting you trace any prediction in production back to the exact model, the run that trained it, the commit, the data snapshot and the configuration. Neither is glamorous. Both separate hobby projects from professional systems.

The ML lifecycle

ML is a loop, not a line:

1. Problem framing: define the decision the model supports, the success metric, constraints (latency, cost, fairness, regulation) and a non-ML baseline.
2. Data: collect, label, validate and version the data (next lesson: Data engineering for AI).
3. Experimentation: features, model families, hyperparameters; track every run.
4. Evaluation: offline metrics on held-out data, slice analysis, robustness and fairness checks, comparison with the current production model.
5. Deployment: package, register, release gradually (shadow, canary, A/B).
6. Monitoring: data drift, prediction drift, performance against delayed labels, latency and cost.
7. Retraining and iteration: triggered by schedule, drift or new data, which loops back to step 2 or 3.

Sculley et al.'s widely cited paper "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015) observed that the model code is a small fraction of a real ML system; the surrounding configuration, data collection, verification, serving and monitoring infrastructure is far larger. It also named ML-specific debts such as entanglement ("changing anything changes everything") and undeclared consumers of model outputs. MLOps exists to manage that surrounding system.

Reproducibility

A run is reproducible if someone else, given your records, can obtain the same model, or a statistically equivalent one, and the same evaluation results.

Sources of non-reproducibility

Bit-exact reproducibility on GPUs is expensive and sometimes impossible across hardware; floating-point addition is not associative, and parallel reductions can sum in different orders. The practical standard is usually statistical reproducibility: re-running gives results within the known run-to-run variance. That requires knowing the variance.

Worked example 1: is the improvement real?

You compare a baseline and a new model, each trained with 5 seeds, on validation AUC:

• Baseline: 0.842, 0.838, 0.845, 0.840, 0.843
• New: 0.848, 0.851, 0.846, 0.850, 0.849

Means. Baseline: $(0.842+0.838+0.845+0.840+0.843)/5 = 4.208/5 = 0.8416$. New: $4.244/5 = 0.8488$. Difference $\Delta = 0.0072$.

Sample standard deviations $s = \sqrt{\sum_i (x_i - \bar{x})^2 / (n-1)}$.

• Baseline deviations: $0.0004, -0.0036, 0.0034, -0.0016, 0.0014$; squares sum to$29.2\times10^{-6}$;$s_b^2 = 7.3\times10^{-6}$,$s_b \approx 0.0027$.
• New deviations: $-0.0008, 0.0022, -0.0028, 0.0012, 0.0002$; squares sum to $14.8\times10^{-6}$; $s_n^2 = 3.7\times10^{-6}$, $s_n \approx 0.0019$.

Welch's t statistic:

With about 7 degrees of freedom, $t \approx 4.9$ corresponds to a two-sided p-value well below 0.01. The improvement is larger than seed noise. Had you compared a single seed of each (say 0.845 vs 0.846), you could not have told. Tracking multiple seeds per configuration, and logging them as grouped runs, is what makes this analysis possible. (This tests seed variance only; it does not account for validation-set sampling error or for multiple comparisons across many experiments.)

Experiment tracking

An experiment tracker records, for every training run:

• Parameters: hyperparameters and configuration.
• Metrics: scalar values over steps or epochs (loss, accuracy, AUC), plus final evaluation results.
• Artifacts: model files, plots, confusion matrices, sample predictions.
• Metadata: git commit, data version, environment, hardware, start/end time, user, tags.

and lets you search, compare and visualize runs.

Tools

• MLflow is an open-source platform with components for tracking, a model registry, model packaging (the "MLmodel" format with "flavors" for scikit-learn, PyTorch, Transformers and others), and evaluation, including features for LLM and GenAI tracing in recent versions. You can self-host it (a tracking server with a database backend and an artifact store such as S3), and several cloud platforms offer managed versions.
• Weights & Biases (W&B) is a commercial platform, offered as SaaS and in self-hosted/dedicated deployments, with experiment tracking and rich dashboards, hyperparameter sweeps, artifact versioning with lineage, a model registry, and LLM tracing and evaluation tools. It has a free tier for individuals; team use is paid.
• Others: TensorBoard (visualization, often used locally), and managed services within cloud ML platforms (for example Amazon SageMaker and Google Vertex AI) that include tracking and registries.

Choosing among them is mostly about hosting constraints (data residency, air-gapped environments), cost, collaboration features, and what integrates with your training and serving stack. The concepts transfer across tools.

MLflow in code

The example below trains a scikit-learn classifier, logs parameters, metrics, the model, and data/code identifiers, all in one tracked run.

import hashlib, subprocess
import mlflow, mlflow.sklearn
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_va, y_tr, y_va = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
data_hash = hashlib.sha256(X.tobytes() + y.tobytes()).hexdigest()[:12]
try:
    git_sha = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
except Exception:
    git_sha = "unknown"

mlflow.set_experiment("breast-cancer-logreg")
for C in [0.01, 0.1, 1.0]:
    with mlflow.start_run(run_name=f"logreg-C{C}"):
        params = {"C": C, "max_iter": 5000, "seed": 42}
        mlflow.log_params(params)
        mlflow.set_tags({"data_hash": data_hash, "git_sha": git_sha})
        model = LogisticRegression(C=C, max_iter=5000).fit(X_tr, y_tr)
        auc = roc_auc_score(y_va, model.predict_proba(X_va)[:, 1])
        mlflow.log_metric("val_auc", auc)
        mlflow.sklearn.log_model(model, name="model")
        print(f"C={C}: val_auc={auc:.4f}")

By default MLflow writes to a local store; run mlflow ui to browse and compare runs. In a team, set MLFLOW_TRACKING_URI to a shared tracking server. (The name= keyword is the MLflow 3 form; on MLflow 2.x write artifact_path="model" instead.) Many frameworks also support autologging (mlflow.autolog()), which captures parameters and metrics automatically; still log data and code identifiers explicitly, because autologging cannot know them.

Model registry

A model registry is the catalog of models intended for use: each registered model has numbered versions, each linked to the run (and therefore the code, data and metrics) that produced it. Registries support:

• Aliases or stages, such as champion and challenger, that deployment systems resolve to a specific version. (MLflow moved from fixed stages like "Staging"/"Production" to flexible aliases and tags.)
• Approval workflows and audit history: who promoted which version, when, and why.
• Model metadata: input/output signatures, evaluation reports, model cards, licenses.

In MLflow, info = mlflow.sklearn.log_model(model, name="model") returns a ModelInfo object, and mlflow.register_model(info.model_uri, "breast-cancer-clf") registers that model (older code passes runs:/<run_id>/model URIs, which MLflow 3 documents as deprecated); the registry requires a database-backed tracking store (for example mlflow server --backend-store-uri sqlite:///mlflow.db). Serving code should load models by alias (models:/breast-cancer-clf@champion) so promotion and rollback are registry operations, not code changes.

Versioning data, code and models together

• Code: git. Every training run should reference a commit SHA, and production training should run only from committed code.
• Data: files too large for git are versioned with tools such as DVC (Data Version Control). dvc add data/train.parquet computes a content hash, moves the file into a cache, and writes a small .dvc pointer file that you commit to git; dvc push uploads the content to remote storage (S3, GCS, Azure, SSH). Checking out an old commit and running dvc pull restores the matching data. DVC pipelines (dvc.yaml) also declare stages with their inputs and outputs so that only changed stages re-run. Alternatives include lakeFS (git-like branches over object storage) and table formats with time travel (Delta Lake, Apache Iceberg).
• Models: registry versions plus immutable artifacts with content hashes.

The goal is lineage: for model version 17 you can answer "which run, which commit, which data snapshot, which config, which evaluation?" in seconds. This is increasingly a regulatory expectation, not just good hygiene (see AI governance frameworks).

Worked example 2: tracing a regression

A production fraud model's precision drops. Lineage lets you compare version 17 (current) with version 16: same code SHA, same hyperparameters, but data hash differs. The DVC diff shows the training snapshot gained a month of data in which a labeling rule changed upstream. Without lineage, this investigation means days of guessing; with it, a few queries.

CI/CD for ML

Software CI/CD tests and ships code. ML adds two moving parts, data and models, and a third practice, continuous training (CT).

• Continuous integration (CI): on every change, run unit tests for feature code, data validation tests on sample data (schema, ranges, null rates), training smoke tests on a tiny dataset, and lint/type checks.
• Continuous training (CT): an automated pipeline retrains the model on fresh data on a schedule or trigger.
• Continuous delivery (CD): a candidate model passes gates before promotion:
  • beats the current champion on primary metrics by a margin larger than seed noise;
  • no regression beyond a tolerance on important slices (regions, device types, demographic groups where lawful and appropriate);
  • latency, memory and cost budgets;
  • fairness and safety checks;
  • then a shadow deployment or canary with live traffic before full rollout, with automated rollback.

Google's MLOps maturity levels

Google Cloud's architecture guide "MLOps: Continuous delivery and automation pipelines in machine learning" describes three levels:

Most teams start at level 0. The jump to level 1 (automated retraining with validation) pays off when data changes frequently; level 2 pays off when many people iterate on many models. Other vendors publish similar maturity models with more levels; the idea is the same.

Common pitfalls

• Logging metrics but not data versions, so results cannot be reproduced after the source table changes.
• Training from uncommitted notebooks.
• Comparing single runs and shipping noise; always know seed variance.
• Selecting on the test set through many tracked experiments; keep a final holdout untouched.
• Registry as a file dump: versions without metrics, signatures or owners.
• Hard-coding model paths in serving code instead of resolving registry aliases.
• Gates on aggregate metrics only, missing large regressions on important slices.
• Over-engineering early: a level-2 platform for one model that changes yearly.

In practice

• From day one: commit code, set seeds, log params, metrics, data hash and git SHA for every run, even in notebooks.
• Group runs by experiment and tag them with the question being tested; write a one-line conclusion on the winning run.
• Version datasets with DVC or your lakehouse's snapshots, and reference the version in every run.
• Register only candidates you would deploy, with signature, evaluation report and owner.
• Automate the path you repeat most first: usually retraining plus validation (level 1), then CI/CD for the pipeline itself.
• The same discipline applies to LLM applications: prompts, retrieval indexes and evaluation sets are versioned artifacts too (see LLMOps in production).