Generative Models: The Big Picture

Generative AI is the part of machine learning that produces new data: images, audio, video, text, molecules, code. Every modern system you have heard of, from chat assistants to text-to-image tools, is built on one of a small number of model families. If you understand what each family learns, how it is trained, and where it breaks, you can read any new paper or product announcement and place it on the map within minutes.

The core idea: learning a distribution

A discriminative model learns $p(y \mid x)$: given an input $x$, how likely is each label $y$? A generative model learns $p(x)$ itself (or $p(x \mid c)$ for some condition $c$ such as a text prompt): how probable is this particular image, waveform or sentence among all the data the world could produce?

A useful picture: imagine every possible 64×64 color image as a point in a space with $64 \times 64 \times 3 = 12{,}288$ dimensions. Almost all of those points are static. Real photos occupy a thin, curved region (a "manifold") inside that space. A generative model is a map of that region: it tells you where the dense areas are and, crucially, lets you sample new points that land inside it.

Learning $p(x)$ gives you up to three abilities, and the model families differ in which they provide cheaply:

1. Sampling: draw new $x \sim p_\theta(x)$, where $p_\theta$ is the model distribution with parameters $\theta$.
2. Density evaluation: compute $p_\theta(x)$ for a given $x$ (useful for compression, anomaly detection, and comparing models by likelihood).
3. Representation: obtain a compact latent code $z$ that describes $x$.

Training almost always means making $p_\theta$ close to the data distribution $p_{\text{data}}$. The most common target is maximum likelihood, which is equivalent to minimizing the KL divergence $D_{\mathrm{KL}}(p_{\text{data}} \,\|\, p_\theta)$:

The families below are different answers to one hard question: how do you build a $p_\theta$ that is flexible enough to describe photographs, yet tractable enough to train and sample?

Family 1: Autoregressive models

Any joint distribution can be factored exactly with the chain rule of probability. Order the dimensions of $x$ as $x_1, \dots, x_D$:

An autoregressive model is a neural network that outputs each conditional. For text, $x_i$ are tokens and the network is a causal transformer: this is exactly how large language models work. For images, PixelRNN and PixelCNN (van den Oord et al., 2016) predicted pixels one at a time in raster order; WaveNet (van den Oord et al., 2016) did the same for raw audio samples. Today, images and audio are usually first compressed into discrete tokens by a learned codec (covered in later lessons), and a transformer predicts those tokens.

Worked example. A toy model generates three binary values. It predicts $p(x_1{=}1) = 0.6$, then $p(x_2{=}0 \mid x_1{=}1) = 0.3$, then $p(x_3{=}1 \mid x_1{=}1, x_2{=}0) = 0.8$. The probability of the sequence $(1, 0, 1)$ is

so its negative log-likelihood is $-\ln 0.144 = 1.938$ nats, or $1.938 / \ln 2 = 2.796$ bits. Training minimizes the average of exactly this quantity over the dataset.

Strengths: exact likelihood, very stable training (it is just cross-entropy), and they scale predictably with data and compute. Weaknesses: sampling is sequential, so generating $D$ elements needs $D$ network evaluations (mitigated by KV caching and parallel decoding tricks); the fixed ordering is arbitrary for images; and errors can compound because each step conditions on the model's own earlier outputs.

Family 2: Variational autoencoders (VAEs)

Kingma and Welling introduced the VAE in "Auto-Encoding Variational Bayes" (2013). The idea is a latent-variable model: first draw a low-dimensional code $z$ from a simple prior $p(z) = \mathcal{N}(0, I)$, then generate $x$ from a decoder $p_\theta(x \mid z)$. The marginal likelihood is

which is intractable for a neural decoder because you cannot integrate over all $z$. The VAE adds an encoder (inference network) $q_\phi(z \mid x)$ with parameters $\phi$ that guesses which codes could have produced $x$. Using Jensen's inequality you get the evidence lower bound (ELBO):

The gap between the two sides is exactly $D_{\mathrm{KL}}(q_\phi(z \mid x) \,\|\, p_\theta(z \mid x))$, so the bound is tight when the encoder matches the true posterior. Maximizing the ELBO trains both networks at once: the first term asks the decoder to reconstruct $x$ from codes the encoder proposes; the second keeps the encoder's codes close to the prior so that sampling $z \sim \mathcal{N}(0, I)$ at generation time lands somewhere the decoder understands.

The reparameterization trick

The encoder outputs a mean $\mu_\phi(x)$ and standard deviation $\sigma_\phi(x)$ for a diagonal Gaussian. We need gradients of an expectation over $z$ with respect to $\phi$, but sampling is not differentiable. The trick is to move the randomness outside the network:

where $\odot$ is element-wise multiplication. Now $z$ is a deterministic, differentiable function of $\phi$ given $\epsilon$, so ordinary backpropagation works and gradient estimates have low variance.

For a diagonal Gaussian encoder and standard normal prior, the KL term has a closed form per latent dimension $j$:

Worked example. A 2-dimensional latent with $\mu = (0.5, -1.0)$ and $\sigma = (1.0, 0.5)$:

• Dimension 1: $\tfrac{1}{2}(0.25 + 1 - 1 - \ln 1) = 0.125$
• Dimension 2: $\tfrac{1}{2}(1 + 0.25 - 1 - \ln 0.25) = \tfrac{1}{2}(0.25 + 1.386) = 0.818$
• Total KL $= 0.943$ nats.

If the decoder's reconstruction log-likelihood for this $x$ (estimated with one sample of $z$) is $-12.30$, the ELBO is $-12.30 - 0.943 = -13.24$, and the training loss is its negative, $13.24$.

Strengths: fast single-pass sampling, a meaningful latent space, a likelihood bound, stable training. Weaknesses: with simple Gaussian decoders, samples tend to be blurry because the reconstruction term behaves like mean-squared error, which averages over plausible outputs. Posterior collapse (the encoder ignores $x$ and the KL term goes to zero) is common when the decoder is powerful. The modern role of VAEs is mostly as compressors: the latent space of Stable Diffusion is a VAE (trained with extra perceptual and adversarial losses), and VQ-VAE (van den Oord et al., 2017) replaced the Gaussian latent with a discrete codebook, which is the ancestor of today's image and audio tokenizers.

Family 3: Generative adversarial networks (GANs)

Goodfellow et al. (2014) proposed training two networks against each other. A generator $G$ maps noise $z \sim p(z)$ to a sample $G(z)$. A discriminator $D$ outputs the probability that its input is real. They play a minimax game:

For a fixed generator with sample distribution $p_g$, the optimal discriminator is

For example, at a point where $p_{\text{data}}(x) = 0.3$ and $p_g(x) = 0.1$, $D^*(x) = 0.3 / 0.4 = 0.75$. Substituting $D^*$ back shows the generator is minimizing $2\, \mathrm{JSD}(p_{\text{data}} \,\|\, p_g) - \log 4$, where JSD is the Jensen–Shannon divergence; the global optimum is $p_g = p_{\text{data}}$, where $D^* = 1/2$ everywhere and $V = -\log 4$. In practice, the generator is trained with the "non-saturating" loss $-\log D(G(z))$ instead, because $\log(1 - D(G(z)))$ has vanishing gradients early in training when $D$ easily rejects fakes.

GANs produced the sharpest images of the late 2010s (the StyleGAN line from NVIDIA is the best-known example) because the discriminator acts as a learned perceptual loss rather than pixel-wise error.

Failure modes

• Mode collapse: the generator finds a few outputs that fool the current discriminator and produces only those, covering a small part of the data distribution. The adversarial objective does not directly penalize missing modes.
• Training instability: it is a two-player game, not a single minimization. Gradients can oscillate, the discriminator can overpower the generator (vanishing generator gradients), and results are sensitive to architecture, learning rates and batch size. Remedies include Wasserstein GAN losses with gradient penalties, spectral normalization, two-timescale learning rates and careful architecture choices, but none make GANs as plug-and-play as likelihood training.
• No likelihood: you cannot evaluate $p_g(x)$, which makes model comparison depend on sample-based metrics such as FID.

GANs remain important as components: adversarial losses sharpen VAE decoders and neural audio codecs, and GAN vocoders such as HiFi-GAN convert spectrograms to waveforms in many speech systems.

Family 4: Normalizing flows

A flow builds $p_\theta(x)$ from an invertible, differentiable function $f_\theta$ that maps data $x$ to a latent $z = f_\theta(x)$ with a simple density $p_Z$. The change-of-variables formula gives an exact likelihood:

The Jacobian determinant accounts for how $f_\theta$ stretches or compresses volume. Worked 1D example: let $x = 2z + 1$ with $z \sim \mathcal{N}(0, 1)$, so $f(x) = (x - 1)/2$ and $|df/dx| = 1/2$. At $x = 3$, $z = 1$, $p_Z(1) = 0.2420$, so $p(x{=}3) = 0.2420 \times 0.5 = 0.1210$.

Computing a $D \times D$ determinant costs $O(D^3)$, so flows use layers with triangular Jacobians. Coupling layers (RealNVP, Dinh et al. 2016; Glow, Kingma and Dhariwal 2018) split $x$ into halves and transform one half with a scale and shift computed from the other; the log-determinant is just the sum of the log-scales. Rezende and Mohamed (2015) popularized the term "normalizing flow" for variational inference.

Strengths: exact likelihood, exact inversion, one-pass sampling. Weaknesses: the invertibility constraint forces the latent to have the same dimension as the data and limits architecture, so pure flows have lagged in sample quality. The ideas live on, however, in continuous-time flows and flow matching, which train a neural ODE velocity field and now power several leading image and video generators (see the diffusion lesson).

Family 5: Diffusion models

Diffusion models (Sohl-Dickstein et al., 2015; made practical by Ho, Jain and Abbeel's DDPM, 2020) define a fixed forward process that gradually adds Gaussian noise to data until nothing is left but noise, then train a network to reverse it one small step at a time. Each reverse step is an easy denoising problem, so training is a simple regression loss, stable and scalable. Sampling starts from pure noise and runs the learned denoiser for tens of steps. The training objective is a reweighted ELBO, so diffusion models also give likelihood bounds.

Diffusion (and its close relative flow matching) is the dominant approach for images, video and much of audio as of November 2026. Its main weakness is sampling cost: many network evaluations per sample, which a whole research area of fast samplers and distillation works to reduce. The next lesson covers it in depth.

Family 6: Energy-based models (briefly)

An energy-based model (EBM) assigns a scalar energy $E_\theta(x)$, low for realistic data, and defines

The partition function $Z_\theta$ is intractable, which makes both likelihood training and sampling hard; sampling typically uses Markov chain Monte Carlo such as Langevin dynamics, which repeatedly steps along $-\nabla_x E_\theta(x)$ plus noise. EBMs are conceptually important because the gradient of the log-density, $\nabla_x \log p(x) = -\nabla_x E(x)$, does not depend on $Z_\theta$. That quantity is the score, and learning the score directly is exactly what score-based diffusion models do. In that sense diffusion is the practical descendant of energy-based thinking.

Comparison

The table summarizes typical behavior; individual models can deviate.

A well-known framing is the "generative trilemma": high quality, fast sampling and mode coverage are hard to get simultaneously. GANs are fast and sharp but drop modes; VAEs and flows are fast with coverage but lower quality; diffusion has quality and coverage but is slow. Much of the last few years of research is about escaping that triangle, for example distilling diffusion models into few-step generators.

Code: a small VAE in PyTorch

This VAE learns the two-moons dataset (2D points), small enough to train on a CPU in seconds. It shows the encoder, reparameterization, closed-form KL and sampling.

import torch, torch.nn as nn
from sklearn.datasets import make_moons

X = torch.tensor(make_moons(4000, noise=0.05)[0], dtype=torch.float32)

class VAE(nn.Module):
    def __init__(self, d=2, h=64, k=2):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(d, h), nn.ReLU(), nn.Linear(h, 2 * k))
        self.dec = nn.Sequential(nn.Linear(k, h), nn.ReLU(), nn.Linear(h, d))
    def forward(self, x):
        mu, logvar = self.enc(x).chunk(2, dim=-1)
        eps = torch.randn_like(mu)
        z = mu + torch.exp(0.5 * logvar) * eps          # reparameterization
        return self.dec(z), mu, logvar

model = VAE()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for step in range(3000):
    x = X[torch.randint(0, len(X), (256,))]
    xhat, mu, logvar = model(x)
    recon = ((xhat - x) ** 2).sum(-1).mean()           # Gaussian decoder, fixed variance
    kl = 0.5 * (mu**2 + logvar.exp() - 1 - logvar).sum(-1).mean()
    loss = recon + 0.1 * kl                             # beta-weighted ELBO (beta = 0.1)
    opt.zero_grad(); loss.backward(); opt.step()
    if step % 1000 == 0:
        print(step, round(recon.item(), 4), round(kl.item(), 4))

with torch.no_grad():
    samples = model.dec(torch.randn(500, 2))            # generate: z ~ N(0, I)
print(samples[:5])

The 0.1 weight on the KL term makes this a β-VAE with β < 1: it trades a looser prior match for sharper reconstructions. With β = 1 on such a tiny model, samples typically smear into a blurry cloud that also fills the gap between the moons, a direct view of the VAE blurriness problem. Plot samples for both settings to compare (details vary by seed).

Common pitfalls

• Comparing likelihoods across families carelessly. Bits-per-dimension from a flow and an ELBO from a VAE are not directly comparable (one is exact, one is a bound), and likelihood correlates poorly with perceptual quality: a model can have excellent likelihood and ugly samples, or vice versa.
• Assuming sharp samples mean good coverage. A GAN can produce beautiful images while ignoring half the classes. Always check diversity (recall-style metrics, per-class counts, nearest-neighbor checks against training data).
• Forgetting the KL weight. Reported "VAE" results often use β ≠ 1 or KL annealing. Copying the architecture without the schedule leads to posterior collapse or blurry output.
• Using pixel MSE as the only loss for anything that must look realistic. Pixel losses average; perceptual and adversarial losses are why modern latent autoencoders look crisp.
• Treating memorization as generalization. Large generative models can reproduce near-copies of training examples. Deduplicate training data and test for near-duplicates of outputs.

In practice

Production systems as of November 2026 combine families rather than pick one. A typical image or video generator is a VAE-style autoencoder (trained with perceptual and adversarial losses) that compresses pixels into latents, plus a diffusion or flow-matching transformer that generates those latents. Text and code generation is autoregressive. Speech and music systems often use a codec (an autoencoder with vector quantization and GAN losses) whose tokens are generated by an autoregressive transformer, sometimes with a diffusion or flow stage on top. When you evaluate a new system, ask four questions: what is the representation (pixels, continuous latents, discrete tokens)? Which family generates it? How many network evaluations does one sample cost? How was quality measured, and does that metric capture diversity?