Image and Video Generation Systems

Text-to-image and text-to-video systems are now ordinary tools in design, marketing, film pre-visualization, games and product photography. Building with them professionally means understanding the pipeline well enough to control it, measure it, budget it and use it lawfully. This lesson assumes you know how diffusion and flow-matching models work and focuses on the full systems built around them.

Intuition: a translator, an artist and a printer

Picture a studio with three specialists. A translator reads your brief and turns it into precise internal notes (the text encoder). An artist works on a small, abstract sketch pad, starting from random scribbles and refining them over many passes while re-reading the notes (the diffusion or flow model in latent space). A printer turns the finished sketch into a full-resolution picture (the autoencoder's decoder). Each specialist can be swapped or tuned separately, and most control techniques are ways of slipping extra instructions to the artist.

The text-to-image pipeline

1. Text encoding. The prompt is tokenized and embedded by a pretrained text encoder. Stable Diffusion 1.x used the text tower of CLIP ViT-L/14; Google's Imagen (Saharia et al., 2022) showed that scaling a large frozen language-model encoder (T5-XXL) improved prompt fidelity more than scaling the image model; Stable Diffusion 3 (Esser et al., 2024) combines two CLIP text encoders with T5-XXL. Newer systems increasingly use decoder-only language models as text encoders. Output: a sequence of embedding vectors (for cross-attention) and often a pooled vector (for global conditioning).
2. Latent generation. Start from Gaussian noise of latent shape, for example $128 \times 128 \times 16$ for a 1024-pixel image with an 8× downsampling autoencoder and 16 channels. A U-Net or diffusion transformer denoises it over $N$ steps, conditioned on the text embeddings, timestep and sometimes size or crop metadata, typically with classifier-free guidance.
3. Decoding. The autoencoder's decoder maps the final latent to pixels. Optional stages: a super-resolution or refiner model, safety classifiers on the prompt and output, and provenance marking (watermarks or C2PA metadata, covered in the deepfakes lesson).

The modular design means three different models with three different failure modes: misread prompts (text encoder), poor composition (denoiser), smeared faces or text (decoder).

Control beyond the prompt

Text is a weak control signal for layout, pose and identity. Professionals rely on these techniques:

Image-to-image (SDEdit)

Instead of starting from pure noise, take an input image, encode it to a latent, noise it to an intermediate timestep, and denoise from there with the new prompt (Meng et al., 2022, "SDEdit"). The strength parameter $\in [0, 1]$ sets how far into the noise schedule you go: with 50 steps and strength 0.6, about $0.6 \times 50 = 30$ denoising steps run, starting from a moderately noised version of your image. Low strength keeps structure and color; high strength keeps only rough composition.

Inpainting and outpainting

Regenerate only a masked region. Dedicated inpainting models take the masked image latent and the mask as extra input channels; a training-free alternative re-imposes the known region (suitably noised) at every step. Outpainting is inpainting beyond the original borders.

Structural conditioning (ControlNet-style)

ControlNet (Zhang, Rao and Agrawala, 2023) copies the encoder blocks of a frozen pretrained U-Net into a trainable branch that reads a conditioning image (edges, depth, human pose skeletons, segmentation maps). The branch's outputs are added back into the frozen network through zero-initialized convolutions, so at the start of training the model behaves exactly like the original and the control is learned without damaging it. Lighter alternatives include T2I-Adapter. For identity or style from a reference image, IP-Adapter (Ye et al., 2023) adds image-prompt embeddings through decoupled cross-attention. Diffusion-transformer models use analogous adapters.

Personalization and style: LoRA and friends

LoRA (Hu et al., 2021) freezes a weight matrix $W \in \mathbb{R}^{d \times k}$ and learns a low-rank update:

where $r$ is the rank and $\alpha$ a scaling constant. For a $1280 \times 1280$ attention projection with $r = 16$, LoRA trains $16 \times (1280 + 1280) = 40{,}960$ parameters instead of $1{,}638{,}400$, 2.5% of the matrix. Style and subject LoRAs are typically a few to a few hundred megabytes and can be mixed at inference with per-LoRA weights. DreamBooth (Ruiz et al., 2023) fine-tunes on 3 to 5 images of a subject bound to a rare token, with a prior-preservation loss to avoid forgetting the class; textual inversion (Gal et al., 2022) learns only a new token embedding. In practice, DreamBooth-style training with LoRA is the common combination.

Evaluation

There is no single number for "good image." Use complementary measures.

FID

The Fréchet Inception Distance (Heusel et al., 2017) embeds real and generated images with an Inception-v3 network, fits a Gaussian to each set of 2048-dimensional features, and computes the Fréchet distance between the Gaussians:

where $\mu_r, \Sigma_r$ and $\mu_g, \Sigma_g$ are the feature means and covariances of real and generated images. Lower is better; it rewards both fidelity and diversity.

Worked example (1D features). In one dimension FID reduces to $(\mu_r - \mu_g)^2 + (\sigma_r - \sigma_g)^2$. Real features: $\mu_r = 0$, $\sigma_r = 1.0$. Model A: $\mu = 0.5$, $\sigma = 1.2$, so $\mathrm{FID} = 0.25 + 0.04 = 0.29$. Model B: $\mu = 0.1$, $\sigma = 0.4$, so $\mathrm{FID} = 0.01 + 0.36 = 0.37$. Model B's samples sit near the real mean but have collapsed diversity, and FID penalizes that.

FID's caveats are serious: it is biased with small sample counts (use the same number, commonly 10k to 50k, for all compared models), sensitive to image resizing and compression implementation details, based on ImageNet features that ignore text and fine detail, and blind to prompt alignment. Variants such as features from self-supervised encoders (DINOv2) address some of this.

CLIP score

CLIPScore (Hessel et al., 2021) measures prompt-image alignment as the cosine similarity of CLIP embeddings of the image $v$ and caption $c$: $\mathrm{CLIPScore} = w \cdot \max(\cos(v, c), 0)$ with $w = 2.5$ in the original paper (many libraries instead report $100 \times \cos$). If $\cos = 0.31$, the score is $2.5 \times 0.31 = 0.775$. It is cheap but inherits CLIP's weaknesses: poor at counting, spatial relations and attribute binding ("a red cube on a blue sphere").

Human preference and compositional benchmarks

Because automatic metrics miss so much, serious evaluation uses pairwise human preference (show two images for one prompt, ask which is better on alignment and quality, aggregate into win rates or Elo-style ratings). Learned preference models trained on such data, such as ImageReward, PickScore and HPS, approximate it cheaply. Object-focused benchmarks like GenEval use detectors to check counts, colors and positions. Always also check text rendering, hands, and demographic diversity across a fixed prompt set.

Video generation

Video adds the time axis, and with it the hardest requirement: temporal consistency. Objects must keep identity, shape and texture across frames; motion must obey plausible physics; camera movement must be coherent.

Early approaches inflated image U-Nets with temporal layers (Ho et al., 2022, "Video Diffusion Models"; Make-A-Video; Stable Video Diffusion). The dominant design described publicly since 2024 treats video as spacetime latents: a video autoencoder compresses space and time together (for example 8× spatially and 4× temporally), the latent is cut into spacetime patches, and a diffusion transformer with full or factorized attention over all patches denoises them jointly. OpenAI's 2024 Sora report and Meta's Movie Gen paper (2024) describe this pattern, and open models follow it.

Worked token-count example. A 5-second clip at 24 fps and 1024×1024 resolution has 120 frames. With 4× temporal compression that is about 30 latent frames (many encoders keep the first frame separately, giving 31). Each latent frame is $128 \times 128$; with $2 \times 2$ patches it becomes $64 \times 64 = 4{,}096$ tokens. Total: $30 \times 4{,}096 = 122{,}880$ tokens per denoising pass. Full self-attention scales with the square of that, about $1.5 \times 10^{10}$ token pairs per layer, versus $1.7 \times 10^7$ for a single 4,096-token image: roughly 900 times more attention work, before multiplying by steps and guidance. This is why video generation is expensive, why clips are short, and why systems generate at lower resolution and upsample.

Other video techniques: image-to-video (animate a given first frame), keyframe interpolation, camera-control conditioning, joint audio-video generation, and extending clips by conditioning on previous latents (which accumulates drift over long durations).

Notable systems (as of November 2026)

The commercial landscape changes monthly, so treat this as a snapshot and check vendor documentation. Google's text-to-image and video models, Imagen and Veo, embed Google's SynthID watermark in their outputs, according to reporting on SynthID. Stability AI's Stable Diffusion line (latent diffusion; SD3 uses rectified-flow transformers) is an open-weight family with published technical papers, and Meta's Movie Gen paper documents a large video model in detail even though its weights were not openly released. Evaluate any system on your own prompts; public leaderboards measure someone else's priorities.

Compute costs

Generation cost is driven by: (forward passes) = steps × (2 if CFG else 1); (cost per pass) which grows with token count and model size; plus text-encoder and decoder overhead (the video decoder can be a large share for long clips). Practical levers: fewer steps with a better solver, distilled few-step models, guidance distillation, lower resolution plus upscaling, reduced precision (fp16/bf16, sometimes 8-bit), caching repeated computations across steps, and batching. For budgeting, measure seconds per image on your hardware at your settings; vendor prices for hosted APIs are usually per image or per second of video and change frequently.

Code: text-to-image and image-to-image with diffusers

The Hugging Face diffusers library wraps the full pipeline. This uses the openly available SDXL base model and requires a GPU for reasonable speed.

import torch
from diffusers import AutoPipelineForText2Image, AutoPipelineForImage2Image
from diffusers import DPMSolverMultistepScheduler

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
model_id = "stabilityai/stable-diffusion-xl-base-1.0"

pipe = AutoPipelineForText2Image.from_pretrained(model_id, torch_dtype=dtype).to(device)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

gen = torch.Generator(device=device).manual_seed(42)      # reproducible
image = pipe(
    prompt="studio photo of a ceramic teapot shaped like a snail, soft light",
    negative_prompt="blurry, low quality, text, watermark",
    num_inference_steps=25,
    guidance_scale=6.0,
    generator=gen,
).images[0]
image.save("teapot.png")

# Image-to-image: reuse the loaded components, keep composition, change style
i2i = AutoPipelineForImage2Image.from_pipe(pipe)
edited = i2i(prompt="watercolor illustration of a snail teapot",
             image=image, strength=0.55, guidance_scale=6.0,
             generator=torch.Generator(device=device).manual_seed(42)).images[0]
edited.save("teapot_watercolor.png")

Key points: swapping the scheduler changes the sampler without retraining; fixing the generator seed makes runs reproducible for A/B tests; from_pipe shares weights so the second pipeline costs no extra memory; and strength=0.55 runs roughly 55% of the 25 steps on a noised copy of the first image. Loading a style LoRA is one line, pipe.load_lora_weights(path_or_repo), assuming you have the rights to that LoRA.

Copyright, consent and responsible use

These are legal and ethical questions that vary by jurisdiction and are actively changing; this section is general information, not legal advice.

• Training data. Most large image models were trained on web-scraped image-text pairs. Whether and when that is lawful is being contested in courts and legislatures in several countries, and some jurisdictions offer rights holders text-and-data-mining opt-outs. Prefer models with documented, licensed or opt-out-respecting training data for commercial work, and read the license (some restrict commercial use or require attribution).
• Output ownership. Copyright offices, including the US Copyright Office, have taken the position that copyright protects human authorship; how much human control over prompts, selection and editing is enough remains fact-specific.
• Memorization. Models can reproduce near-copies of training images, logos and characters, especially heavily duplicated ones. Avoid prompts that name living artists or trademarked characters for commercial outputs, and screen outputs.
• Likeness and consent. Generating identifiable real people, especially in sexual, defamatory or deceptive contexts, raises right-of-publicity, privacy and criminal issues. Several jurisdictions now require labelling of synthetic media (see the deepfakes lesson). Obtain written consent for any person whose likeness you train or generate.

Common pitfalls

• Prompting where control is needed. Layout, pose and brand-exact colors are better solved with ControlNet-style conditioning, inpainting or reference adapters than with ever-longer prompts.
• Over-training personalization. DreamBooth or LoRA runs that are too long or too high-rank make every output look like the training photos and destroy prompt following. Hold out prompts and check.
• Mismatched components. LoRAs, ControlNets and VAEs are tied to a base model family; loading an SD 1.5 LoRA into SDXL fails or produces garbage.
• Trusting one metric. Optimizing FID or CLIP score alone produces images that score well and look worse. Combine automatic metrics with human review on a fixed prompt set.
• Ignoring licenses and consent. Model licenses, LoRA sources and people's likeness rights are real constraints; check them before deployment, not after.

In practice

Production image pipelines usually look like: prompt templating and safety filtering, a licensed base model with a few tuned LoRAs or adapters, structural control from sketches or 3D renders, inpainting for fixes, upscaling, provenance marking, and human review for anything public. Teams track a fixed evaluation prompt suite with seeds, measure latency per image on target hardware, and re-run the suite whenever the model, sampler or LoRA set changes. For video, expect to generate many short candidates and select, to fix the first frame with an image model for consistency, and to budget compute far above image work.