Deploying Models: From Artifact to Production Service

A trained model is only useful once something can call it. Deployment is the engineering that turns a weights file on a laptop into a service that answers fraud checks in 30 milliseconds, scores ten million customers overnight, or runs on a phone with no network. Most of the cost and risk in a production ML system sits here, not in the training notebook. Sculley et al. made the same point in their well-known "hidden technical debt" paper: the model code is a small box in the middle of a large system.

Intuition: a restaurant, not a recipe

Think of a trained model as a recipe. Deployment is opening the restaurant. You have to decide whether you cook to order (real-time), prep everything the night before (batch), or run a conveyor belt that cooks as ingredients arrive (streaming). You need a kitchen that can be copied exactly in another city (containers), a manager who adds cooks at the dinner rush (autoscaling), and a way to try a new dish on a few tables before it goes on every menu (canary and A/B tests). If the new dish makes people sick, you need to take it off the menu in minutes (rollback).

Three inference patterns

Choose the cheapest pattern the product allows. If a prediction is only read the next morning, batch scoring is simpler, cheaper and easier to debug than a live service. A common hybrid precomputes expensive parts in batch (embeddings, candidate lists) and does a cheap final step online (re-ranking with fresh context). The deciding question is how fresh the inputs must be: if the prediction depends on something that happened two seconds ago, you need online or streaming inference.

A frequent bug in real-time systems is training–serving skew: features computed one way in the offline training pipeline and another way in the online service. A feature store, or at least one shared feature-computation library used by both paths, is the standard defence (see the data-engineering lesson).

Packaging the model: serialization formats

Before you can serve a model you must save it in a form another process can load safely and run quickly.

• Pickle-based checkpoints (classic torch.save, joblib): convenient but unsafe. Unpickling can execute arbitrary code, so never load untrusted pickles. Since PyTorch 2.6, torch.load defaults to weights_only=True for this reason.
• safetensors: a simple format that stores only tensors (a JSON header plus raw bytes). It cannot execute code and supports zero-copy, memory-mapped loading. It has become the default for sharing weights on the Hugging Face Hub. It stores weights only; you still need the model code to rebuild the architecture.
• TorchScript (torch.jit.script / torch.jit.trace): PyTorch's older way to capture a model as a graph that can run without Python. PyTorch now describes TorchScript as nearing end of life and steers new work towards torch.export; treat it as legacy for new projects.
• torch.export: captures a full, ahead-of-time graph (ExportedProgram) of a PyTorch model, with explicit dynamic shapes. It is the entry point for PyTorch's newer deployment paths (for example AOTInductor compilation and ExecuTorch for mobile and edge devices).
• ONNX (Open Neural Network Exchange): a framework-neutral graph format with a versioned operator set ("opset"). Export once, then run in ONNX Runtime, TensorRT, OpenVINO or other engines on many kinds of hardware.
• GGUF: the single-file format used by llama.cpp and related tools for quantized LLMs (covered in the local-runtimes lesson).

Rule of thumb: store training checkpoints as safetensors plus versioned code, and produce a separate inference artifact (ONNX, an exported program, or an engine-specific compiled file) in CI for each target runtime. Record the exact artifact hash in the model registry.

import torch, torch.nn as nn
from safetensors.torch import save_file, load_file

model = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 1)).eval()
save_file(model.state_dict(), "fraud_v3.safetensors")      # weights only, no code execution on load
model.load_state_dict(load_file("fraud_v3.safetensors"))

x = torch.randn(4, 20)
torch.onnx.export(model, (x,), "fraud_v3.onnx",
                  input_names=["x"], output_names=["logit"],
                  dynamic_axes={"x": {0: "batch"}, "logit": {0: "batch"}})  # batch dim stays dynamic

ep = torch.export.export(model, (x,))                      # ahead-of-time graph capture
torch.export.save(ep, "fraud_v3.pt2")

import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("fraud_v3.onnx", providers=["CPUExecutionProvider"])
onnx_out = sess.run(None, {"x": x.numpy()})[0]
assert np.allclose(onnx_out, model(x).detach().numpy(), atol=1e-5)   # parity check

The last line is the most important: always check numerical parity between the training framework and the exported artifact on real inputs before shipping. Exporters change between PyTorch releases (since PyTorch 2.9, torch.onnx.export defaults to a torch.export-based path, which needs the onnxscript package and prefers a dynamic_shapes argument over the older dynamic_axes), so pin versions in your build and let the parity test catch surprises.

Serving over REST and gRPC

A model server wraps the artifact in a network API.

• REST/JSON over HTTP is universal, easy to debug with curl, and fine for most workloads.
• gRPC uses HTTP/2 and Protocol Buffers: binary payloads, a strict schema, streaming, and lower per-call overhead. It suits high-volume service-to-service calls and large tensor payloads.

You can write a small server yourself or use a dedicated inference server (NVIDIA Triton Inference Server, TorchServe, KServe on Kubernetes, Ray Serve, or vLLM and similar engines for LLMs; see Inference serving at scale). Dedicated servers add dynamic batching: they briefly hold incoming requests and run them through the GPU together, trading a few milliseconds of latency for much higher throughput.

A minimal FastAPI service:

from contextlib import asynccontextmanager
import numpy as np, onnxruntime as ort
from fastapi import FastAPI
from pydantic import BaseModel

MODEL_VERSION = "fraud_v3"
state = {}

@asynccontextmanager
async def lifespan(app: FastAPI):
    state["sess"] = ort.InferenceSession(f"{MODEL_VERSION}.onnx",
                                         providers=["CPUExecutionProvider"])
    yield                                   # load once at startup, not per request
    state.clear()

app = FastAPI(lifespan=lifespan)

class PredictRequest(BaseModel):
    features: list[list[float]]             # batch of 20-dim feature vectors

@app.post("/v1/predict")
def predict(req: PredictRequest):           # sync def: FastAPI runs it in a thread pool
    x = np.asarray(req.features, dtype=np.float32)
    logits = state["sess"].run(None, {"x": x})[0]
    probs = 1 / (1 + np.exp(-logits))
    return {"model_version": MODEL_VERSION, "scores": probs.ravel().tolist()}

@app.get("/healthz")
def healthz():
    return {"ready": "sess" in state}

Run it with uvicorn app:app --workers 2. Points to notice: the model loads once at startup; the response includes the model version (essential for debugging and for joining predictions to labels later); a health endpoint lets the orchestrator know when the replica is ready. In production you would also validate input ranges, enforce a request size limit, log a sample of inputs and outputs, and export latency metrics.

Containers and Kubernetes basics

A container image packages the code, the Python environment, system libraries (and CUDA user-space libraries for GPU work) and often the model artifact, so the same thing runs on a laptop, in CI and in production. Good images are small (slim base image, multi-stage builds), pinned (exact dependency versions, base image digest), and run as a non-root user.

Kubernetes runs containers across a cluster. The objects you meet first:

• Pod: one or more containers scheduled together; the unit that runs.
• Deployment: declares "run N replicas of this pod template" and performs rolling updates.
• Service: a stable network name and load balancer in front of a set of pods.
• HorizontalPodAutoscaler (HPA): changes the replica count based on a metric (CPU, or custom metrics such as requests per second or queue depth).
• Readiness and liveness probes: readiness says "send me traffic" (only after the model has loaded); liveness says "restart me if I hang".

GPU scheduling. GPUs are exposed to Kubernetes through vendor device plugins, and pods request them as extended resources (for example nvidia.com/gpu: 1 under resources.limits). By default a GPU is allocated whole and is not shared between containers. Ways to share are vendor specific: NVIDIA offers Multi-Instance GPU (MIG) partitioning on supported data-center GPUs, plus time-slicing. Use node labels and taints so that only GPU workloads land on expensive GPU nodes. Model loading is slow (gigabytes from storage into GPU memory), so set generous readiness delays, and scale on queue depth or in-flight requests rather than CPU utilisation, which says little about a GPU-bound server.

Worked example: sizing a real-time service

A fraud service must handle a peak of $\lambda = 200$ requests per second with mean service time $W = 0.15$ s per request. Little's law says the average number of requests in flight is

$L = \lambda W = 200 \times 0.15 = 30.$

Load tests show one replica handles 8 concurrent requests before its p99 latency exceeds the 250 ms budget. Running replicas at 100% of that is fragile, so target 70% utilisation:

$\text{replicas} = \left\lceil \frac{L}{8 \times 0.7} \right\rceil = \lceil 5.36 \rceil = 6.$

Add at least one more so that losing a node or doing a rolling update does not breach the SLO: plan for 7, and let the autoscaler go higher if traffic exceeds the forecast.

Releasing safely: shadow, canary, A/B, rollback

A new model that scored better offline can still fail online: the features differ slightly, the latency is worse, or users react differently. Release strategies limit the blast radius.

• Shadow (dark) deployment: the new model receives a copy of live traffic, but its outputs are only logged, never returned to users. It tests latency, errors and prediction distributions on real inputs at zero user risk. It cannot measure user-facing outcomes, because users never see its outputs.
• Canary release: send a small fraction of real traffic (say 1%, then 5%, 25%, 100%) to the new version, comparing error rate, latency and key business metrics against the old version at each step. Automate the gates: advance only if every metric is within its threshold.
• A/B test: randomly assign users (not requests, so each user has a consistent experience) to the old or new model for long enough to measure a business outcome with statistical power. A canary asks "is it safe?"; an A/B test asks "is it better?"
• Blue/green: run the old (blue) and new (green) stacks side by side and switch the router all at once; switching back is instant.
• Rollback: always keep the previous model version deployed or instantly deployable, and make "roll back" a one-command or automatic action. Roll back first, investigate second.

Worked example: how long must an A/B test run?

The current model's conversion rate is $p = 0.10$. You want to detect an absolute improvement of $\delta = 0.01$ (to 0.11) with significance level $\alpha = 0.05$ (two-sided) and power $1-\beta = 0.8$. A standard approximation for users needed per arm is

$n \approx \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2\, p(1-p)}{\delta^2},$

where $z_q$ is the $q$ quantile of the standard normal distribution: $z_{0.975} = 1.960$ and $z_{0.8} = 0.842$.

1. $z$ sum: $1.960 + 0.842 = 2.802$; squared: $7.85$.
2. Variance term: $p(1-p) = 0.10 \times 0.90 = 0.09$.
3. Numerator: $2 \times 7.85 \times 0.09 = 1.413$.
4. Divide by $\delta^2 = 0.0001$: $n \approx 14{,}130$ users per arm.

With 4,000 eligible users a day split 50/50, that is about 7 days. Run full weeks anyway, because behaviour differs by weekday. Decide the duration before starting and do not stop early when the dashboard looks good: repeatedly peeking and stopping at the first significant result inflates the false-positive rate. (This formula is a simplification that uses $p$ for both arms. Use a proper power calculator, or sequential testing methods designed for peeking, in real work.)

Edge and on-device deployment

Running the model on the user's device (phone, browser, car, camera, laptop) removes network latency, works offline, keeps data local, and moves compute cost off your servers. The price is tight memory, power and thermal budgets, a wide spread of hardware, and slow update cycles. Typical runtimes include ONNX Runtime (including mobile and web builds), TensorFlow Lite (now branded LiteRT by Google), Core ML on Apple devices, ExecuTorch for PyTorch models, and llama.cpp for LLMs.

Compression makes models fit:

• Quantization: store weights (and often activations) in 8-bit or 4-bit integers instead of 32-bit floats. Post-training quantization needs only a small calibration set; quantization-aware training recovers more accuracy. See the quantization lesson.
• Pruning: remove weights or whole channels with little effect. Unstructured sparsity rarely speeds up ordinary hardware; structured pruning (whole channels or heads) and hardware-supported patterns such as 2:4 sparsity do.
• Knowledge distillation: train a small "student" model to match a large "teacher" model's output distribution (Hinton, Vinyals and Dean, 2015).
• Architecture choice: start from a model designed for mobile (depthwise-separable convolutions, small transformers).

Worked size example: a ResNet-50 has about 25.6 million parameters. At FP32 (4 bytes each) that is about $25.6\text{M} \times 4 = 102$ MB; at INT8 about 26 MB, a 4× reduction before any pruning. Always re-measure accuracy on the target device and runtime, not just in Python on a workstation: operator support and numerical behaviour differ.

Common pitfalls

• Training–serving skew: different feature code, different preprocessing (image resize, tokenizer version) or different default values online versus offline. Test with recorded production inputs.
• Loading the model per request or lazily on the first request, which causes multi-second cold starts and timeouts. Load at startup and gate readiness on it.
• No model version in logs or responses, making it impossible to tell which model produced a bad prediction.
• Scaling GPU services on CPU utilisation. Use queue depth, in-flight requests or GPU utilisation.
• Skipping the parity test after export or quantization, then finding a silent accuracy drop in production.
• Unpinned dependencies and latest image tags, so a rebuild changes behaviour without anyone changing code.
• Calling an A/B test early or randomising by request instead of by user.
• No tested rollback path. A rollback that has never been rehearsed often fails when needed.
• Loading untrusted pickles. Prefer safetensors and weights_only loading.

In practice

• Pick the simplest inference pattern the product allows; batch is underrated.
• Treat models as versioned, immutable artifacts in a registry, with the training data snapshot, code commit, metrics and export hash attached.
• Build inference artifacts in CI with an automated parity test and a latency benchmark on the target hardware.
• Ship every model through shadow, then a staged canary with automated metric gates, then (if the change targets a business metric) an A/B test.
• Define SLOs (for example, p99 below 250 ms and 99.9% availability) and alert on them; the monitoring lesson covers model-quality signals.
• Keep N−1 deployed or instantly deployable, and practise rollbacks.
• For edge targets, budget memory, latency and energy up front, and test on the slowest device you support.