Privacy-Preserving AI: Attacks, Differential Privacy, Federated Learning and the Law

A model is a function fitted to data, and some of that data can be read back out of it. Language models have reproduced phone numbers from their training sets, image classifiers have been inverted into recognizable faces, and simple statistical tests can reveal whether a particular patient's record was used to train a hospital's model. This lesson covers how those leaks happen, the mathematical tools that bound them, the engineering techniques that keep raw data out of reach, and the legal principles that apply when the data is about people.

Intuition: a model is a lossy compression of its data

Picture a student who studies for an exam by memorizing past papers. Ask them a question they saw before and they answer instantly and confidently; ask a new one and they hesitate. Watching that difference in confidence tells you which questions were in their study set. Ask them to "write out a practice paper" and some of what they produce will be the past papers, word for word.

Models behave the same way. Training minimizes loss on the training set, so training examples tend to get lower loss and higher confidence than unseen ones, and rare or repeated examples can be memorized verbatim. Every privacy attack below exploits this gap between how the model treats data it has seen and data it has not. Every defense either shrinks the gap (regularization, differential privacy), keeps the raw data away from the party that could exploit it (federated learning, secure computation), or avoids collecting the data in the first place (minimization).

The threat model

Specify an attack by access (black-box: query outputs only, perhaps only labels; white-box: weights and gradients), knowledge (does the attacker know the data distribution or have similar data?) and goal.

Membership inference

Goal: decide whether a given record $x$ was in the training set. Shokri et al. (2017) trained shadow models on data similar to the target's, observed how shadow models' output vectors differ on their own members versus non-members, and trained an attack classifier on those vectors. A simpler and strong baseline is the loss-threshold attack: predict "member" if the target model's loss on $(x, y)$ is below a threshold $\tau$.

Membership matters because membership itself can be sensitive (being in a cancer-treatment dataset reveals a diagnosis), and because membership inference is the standard yardstick for measuring leakage. Report attacks by their true-positive rate at a low false-positive rate (for example TPR at 0.1% FPR), not by average accuracy: an attack that confidently identifies a few people is a real privacy failure even if its overall accuracy is near 50% (Carlini et al., 2022, make this argument in detail).

Model inversion and attribute inference

Goal: reconstruct features of training data, or infer a sensitive attribute of a known person. Fredrikson et al. (2015) used confidence scores from a face-recognition model to reconstruct recognizable images of people in the training set by optimizing an input to maximize the confidence for a target name, and showed attribute inference against a lifestyle-survey model. Returning only labels or rounded confidences reduced their attack's effectiveness, which is an early example of output minimization as a defense.

Training-data extraction

Goal: recover verbatim training examples from a generative model. Carlini et al. (2021) generated large numbers of samples from GPT-2, ranked them by signals of memorization (for example, a low perplexity under GPT-2 relative to a smaller reference model or to zlib compression), and confirmed hundreds of verbatim training sequences, including names, phone numbers and email addresses. Extraction grows with model size, the number of times a string is duplicated in the training data, and the length of the prompt context. Deduplicating training data is one of the cheapest and most effective mitigations.

Other leak paths you should list in any threat model: gradient sharing (gradients can be inverted into inputs, see federated learning below), embeddings (text embeddings can be partly inverted back to text), retrieval-augmented systems (the model faithfully quotes sensitive documents a user should not see; this is an access-control failure, not a training one) and logs of prompts and outputs.

Differential privacy

The definition

Call two datasets $D$ and $D'$ neighbors if they differ in one individual's record (either by adding/removing one record, or replacing one; state which, since constants change). A randomized mechanism $M$ is $(\varepsilon, \delta)$-differentially private if for all neighboring $D, D'$ and every set of outputs $S$:

Here $\varepsilon \ge 0$ is the privacy loss budget (smaller is more private) and $\delta \in [0, 1)$ is a small slack, the probability mass on which the $e^{\varepsilon}$ bound may fail; it should be much smaller than $1/n$ for a dataset of $n$ people. With $\delta = 0$ we call it pure $\varepsilon$-DP. The definition is a property of the algorithm, not of a dataset: it holds whatever the attacker knows, including every other record.

What epsilon means for an attacker

Treat membership inference as a hypothesis test between $D$ (target present) and $D'$ (absent). Any attack that outputs "present" is an event $S$, so DP implies

With $\varepsilon = 1$, $\delta = 0$ and an attacker who tolerates a 1% false-positive rate, the true-positive rate is at most $e^{1} \times 0.01 \approx 0.0272$: barely better than guessing at that operating point. With $\varepsilon = 8$, the bound is $e^{8} \times 0.01 \approx 29.8$, which is no constraint at all at 1% FPR. That is why large $\varepsilon$ values still often defeat practical attacks (the bound is worst-case, and real attacks are weaker) but offer weak guarantees.

Two key properties

• Post-processing: any computation on the output of a DP mechanism, without touching the data again, remains DP with the same parameters. You can publish the model and let anyone use it.
• Composition: running mechanisms with budgets $\varepsilon_1, \dots, \varepsilon_k$ on the same data is at most $(\sum_i \varepsilon_i, \sum_i \delta_i)$-DP. This basic bound is loose; advanced composition and Rényi-DP accountants give much tighter totals for many repeated noisy steps, which is what makes DP training feasible.

The Laplace and Gaussian mechanisms

For a numeric query $f$, the $\ell_1$-sensitivity is $\Delta_1 f = \max_{D \sim D'} \lVert f(D) - f(D') \rVert_1$. The Laplace mechanism releases $f(D) + \text{Lap}(\Delta_1 f / \varepsilon)$ and is $\varepsilon$-DP. The Gaussian mechanism releases $f(D) + \mathcal{N}(0, \sigma^2)$ with $\ell_2$-sensitivity $\Delta_2 f$; the classical analysis gives $(\varepsilon, \delta)$-DP for $\varepsilon < 1$ when

(Tighter calibrations exist and are used by modern libraries; this one is the textbook version.)

Worked example 1: a private count

A clinic asks "how many patients in this dataset have diabetes?" Adding or removing one person changes the count by at most 1, so $\Delta_1 f = 1$. With $\varepsilon = 0.5$, add Laplace noise with scale $b = 1/0.5 = 2$. The Laplace distribution has standard deviation $\sqrt{2}\, b \approx 2.83$, and $\Pr[|\text{noise}| > t] = e^{-t/b}$, so the released count is within $\pm 6$ of the truth with probability $1 - e^{-3} \approx 0.95$. For a true count of 1,240 that error is negligible; for a true count of 7 in a rural clinic it swamps the answer. DP protects individuals by making small groups invisible, which is both the point and the cost.

Worked example 2: a private mean

Release the mean salary of $n = 10{,}000$ employees. First clip each salary to $[0, 200{,}000]$ so sensitivity is bounded. Under replace-one neighbors, the mean changes by at most $200{,}000 / 10{,}000 = 20$, so $\Delta_2 f = 20$. With $\varepsilon = 0.9$ (the classical bound needs $\varepsilon < 1$) and $\delta = 10^{-5}$:

so $\sigma \approx 20 \times 4.845 / 0.9 \approx 107.7$. The released mean is typically within about $\pm 215$ (two standard deviations) of the true mean, on salaries averaging tens of thousands. Notice that clipping introduced bias (anyone above 200,000 is counted as 200,000); choosing the clipping bound is a real modeling decision.

DP-SGD: training models with differential privacy

Abadi et al. (2016) made deep learning differentially private with three changes to stochastic gradient descent. At each step $t$:

1. Sample a batch by including each example independently with probability $q = L/N$ (Poisson sampling; $L$ is the expected batch size, $N$ the dataset size).
2. Clip each per-example gradient $g_i = \nabla_\theta \ell(\theta_t; x_i)$ to norm at most $C$:

1. Add noise and average:

Clipping bounds each person's influence on the sum (sensitivity $C$); noise with standard deviation $\sigma C$ hides it; and a privacy accountant (the moments accountant in the original paper, now usually Rényi DP or numerical accountants) converts the noise multiplier $\sigma$, sampling rate $q$, number of steps $T$ and target $\delta$ into a total $\varepsilon$. Sampling helps a lot: each example is only touched in a fraction $q$ of steps ("privacy amplification by subsampling").

import torch
import torch.nn as nn

def dp_sgd_step(model, loss_fn, xb, yb, lr=0.1, clip=1.0, noise_mult=1.1):
    # One DP-SGD step with explicit per-example clipping (clear, not fast).
    params = [p for p in model.parameters() if p.requires_grad]
    summed = [torch.zeros_like(p) for p in params]
    for x, y in zip(xb, yb):                      # per-example gradients
        model.zero_grad()
        loss_fn(model(x.unsqueeze(0)), y.unsqueeze(0)).backward()
        grads = [p.grad.detach().clone() for p in params]
        norm = torch.sqrt(sum((g ** 2).sum() for g in grads))
        scale = min(1.0, clip / (norm.item() + 1e-12))
        for s, g in zip(summed, grads):
            s += g * scale                        # clipped contribution
    with torch.no_grad():
        for p, s in zip(params, summed):
            noise = torch.randn_like(s) * noise_mult * clip
            p -= lr * (s + noise) / len(xb)       # noisy average
    model.zero_grad()

torch.manual_seed(0)
model = nn.Sequential(nn.Linear(20, 32), nn.ReLU(), nn.Linear(32, 2))
X, Y = torch.randn(64, 20), torch.randint(0, 2, (64,))
for _ in range(5):
    dp_sgd_step(model, nn.CrossEntropyLoss(), X, Y)

This loop is written for clarity. Production code computes per-example gradients in one vectorized pass and uses a maintained library (Opacus for PyTorch, TensorFlow Privacy, or JAX-based equivalents) that also performs Poisson sampling and privacy accounting. Never hand-roll the accountant for a real release. Two practical notes: batch normalization mixes examples within a batch and breaks the per-example analysis (use group or layer normalization), and hyperparameter tuning on private data also consumes privacy budget.

DP training costs accuracy, more so for small datasets and for under-represented groups, whose examples tend to have large gradients that get clipped. Common strategies are pretraining on public data and fine-tuning privately, larger batches, and fewer, larger-noise steps. There is no universal "safe" $\varepsilon$; reported deployments range from below 1 to well above 10, and you should state $\varepsilon$, $\delta$, the neighbor definition and the unit of privacy (one record, or one user with many records) whenever you claim DP.

Differential privacy has been deployed at scale outside ML too: the U.S. Census Bureau used it to protect 2020 Census redistricting data, and Google's RAPPOR (Erlingsson et al., 2014) used local DP, where each user randomizes their own data before sending it, so the collector never sees raw values.

Federated learning

In federated learning (McMahan et al., 2017), data stays on clients (phones, hospitals) and only model updates travel. Federated Averaging (FedAvg): the server sends weights $w_t$ to a sample of clients; each client $k$ runs a few epochs of local SGD on its $n_k$ examples to get $w_{t+1}^{k}$; the server averages

Federated learning is data minimization, not a privacy guarantee. Updates can leak training data: gradient-inversion attacks have reconstructed images and text from shared gradients, especially with small local batches. The standard stack therefore adds secure aggregation (a cryptographic protocol so the server learns only the sum of many clients' updates, not any single update; Bonawitz et al., 2017) and differential privacy on the aggregate (often user-level, clipping each client's whole update). Practical challenges include non-identically-distributed client data, stragglers and dropouts, and poisoning by malicious clients.

Secure computation: confidential computing, HE and MPC

None of these limit what the output reveals: a model trained inside an enclave can still memorize and leak data. Combine "who can see the data during computation" defenses with "what the result reveals" defenses such as DP.

Anonymization pitfalls

Removing names is pseudonymization, not anonymization. Sweeney (2000) estimated that about 87% of the U.S. population could be uniquely identified by 5-digit ZIP code, gender and date of birth. Narayanan and Shmatikov (2008) re-identified users in the "anonymized" Netflix Prize dataset by matching a handful of movie ratings and dates against public IMDb reviews: high-dimensional, sparse data makes almost everyone unique. Classical fixes such as $k$-anonymity (every record indistinguishable from at least $k-1$ others on quasi-identifiers) and $\ell$-diversity break down in high dimensions and against background knowledge. Synthetic data is not automatically private either: a generator can memorize and reproduce records unless it is trained with DP and evaluated with membership-inference tests. Under the GDPR (Recital 26), pseudonymized data is still personal data; only data where identification is not reasonably likely, taking into account all means reasonably likely to be used, falls outside the regulation.

Data minimization and the GDPR

The GDPR's Article 5 principles map directly onto ML practice:

• Lawfulness, fairness and transparency: have a lawful basis (Art. 6) for training and for inference; tell people. In December 2024 the European Data Protection Board's Opinion 28/2024 addressed when AI models trained on personal data can be considered anonymous and how legitimate interest may serve as a basis, with a case-by-case three-step test.
• Purpose limitation: data collected for support tickets is not automatically available to train a marketing model.
• Data minimization: use only what is adequate, relevant and necessary. Drop columns, aggregate, truncate free text, filter PII before training, and prefer features over raw records.
• Accuracy: models that infer facts about people can create inaccurate personal data.
• Storage limitation: retention limits for raw data, features, prompts and logs.
• Integrity and confidentiality: security, including against the model-level attacks above.
• Accountability: be able to demonstrate all of this (see AI Governance Frameworks).

Also relevant: special-category data (Art. 9: health, ethnicity, biometrics for identification and more) needs an additional condition; data protection by design and by default (Art. 25); DPIAs (Art. 35) for high-risk processing, which large-scale profiling with AI often is; automated decision-making with legal or similarly significant effects (Art. 22) requires safeguards including human intervention; and data-subject rights such as erasure (Art. 17), which is hard to honor for a trained model. Retraining without the data is the clean answer; "machine unlearning" methods that approximate it are an active research area without strong general guarantees, which is another argument for minimizing and filtering data before training.

Common pitfalls

• Claiming "the model is private because we removed names" or "because we used federated learning". Neither is a guarantee.
• Quoting $\varepsilon$ without $\delta$, the neighbor definition, the unit of privacy, or whether hyperparameter tuning was counted.
• Evaluating membership inference by accuracy or AUC alone instead of TPR at low FPR.
• Forgetting non-training leaks: logs, retrieval corpora without access control, embeddings, and evaluation sets.
• Training on duplicated data; duplication sharply raises memorization.
• Using batch normalization or non-per-example operations inside DP-SGD, silently voiding the guarantee.
• Treating synthetic data as anonymous without testing it.

In practice

Start with minimization and filtering: decide which fields you truly need, scrub PII from text corpora, deduplicate, and set retention limits; this removes most risk cheaply. Restrict outputs (labels or rounded scores rather than full probability vectors) where the product allows. Before release, run a membership-inference audit (loss-threshold and a stronger calibrated attack) and, for generative models, canary and extraction tests: insert unique secret strings into training data and check whether the model reproduces them. For sensitive tabular or clinical data, consider DP-SGD with a maintained library, starting from a publicly pretrained model. Use federated learning with secure aggregation when data legally or practically cannot leave its owners, and confidential computing when the threat is the infrastructure operator. Document all of it in the DPIA, and route deletion requests to a process that can actually honor them, which in practice often means scheduled retraining.