Multimodal Foundation Models

Most frontier AI assistants now accept images, screenshots, PDFs, audio and video alongside text, and some generate images and speech as well. This is what makes it possible to ask questions about a chart, extract fields from a scanned invoice, debug a UI from a screenshot or summarize a recorded meeting. This lesson explains how multimodal foundation models are built, from contrastive pretraining to vision-language adapters to natively multimodal and any-to-any designs, and how to judge their limits.

Intuition: a shared language for different senses

Imagine a library where every book, photograph and recording has a catalog card written in one shared code. Books about dogs and photos of dogs get similar cards. Once everything is cataloged in the same code, you can search photos with words, describe photos in words, or reason about both together. Multimodal models build that shared code: an embedding space, or a shared token stream, in which an image of a dog and the word "dog" end up near each other.

There are three broad designs, in roughly historical order:

1. Dual encoders (CLIP-style): separate image and text encoders trained to agree. Good for retrieval and zero-shot classification; cannot generate text.
2. Adapter-style vision-language models (VLMs): a pretrained image encoder connected to a pretrained LLM through a small projector. The LLM reads image features as if they were tokens.
3. Native multimodal models: one model trained from the start (or early) on interleaved sequences of text, image, audio and video tokens, sometimes generating several modalities.

CLIP: contrastive language-image pretraining

CLIP (Radford et al., 2021) trained an image encoder and a text encoder on about 400 million image-caption pairs collected from the web. The task: in a batch of $N$ pairs, identify which caption goes with which image.

Let $f(\cdot)$ and $g(\cdot)$ be the image and text encoders, each followed by L2 normalization, producing unit vectors $u_i = f(\text{image}_i)$ and $v_j = g(\text{text}_j)$. The similarity matrix is $s_{ij} = u_i^\top v_j / \tau$, where $\tau$ is a learned temperature. The loss is a symmetric cross-entropy (an InfoNCE loss) over rows and columns:

The first term asks each image to pick its caption among all captions in the batch; the second asks each caption to pick its image. The other $N-1$ items in the batch act as negatives, so larger batches give harder, more informative training (CLIP used a batch size of 32,768).

Worked example. A batch of $N = 2$ with cosine similarities $u_1^\top v_1 = 0.9$, $u_1^\top v_2 = 0.2$, $u_2^\top v_1 = 0.1$, $u_2^\top v_2 = 0.8$ and $\tau = 0.5$. Logits: $s = \begin{pmatrix} 1.8 & 0.4 \\ 0.2 & 1.6 \end{pmatrix}$.

• Image 1 (row 1): $p = e^{1.8}/(e^{1.8} + e^{0.4}) = 1/(1 + e^{-1.4}) = 0.802$; loss $-\ln 0.802 = 0.220$.
• Image 2 (row 2): $1/(1 + e^{-1.4}) = 0.802$; loss $0.220$.
• Text 1 (column 1): $e^{1.8}/(e^{1.8} + e^{0.2}) = 1/(1 + e^{-1.6}) = 0.832$; loss $0.184$.
• Text 2 (column 2): $e^{1.6}/(e^{0.4} + e^{1.6}) = 1/(1 + e^{-1.2}) = 0.769$; loss $0.263$.
• Total: $\tfrac{1}{2}\left(\tfrac{0.220 + 0.220}{2} + \tfrac{0.184 + 0.263}{2}\right) = \tfrac{1}{2}(0.220 + 0.224) = 0.222$.

Lowering $\tau$ sharpens the softmax and would make this already-correct batch's loss smaller.

Zero-shot classification follows directly: embed prompts such as "a photo of a {class}" for every class, embed the image, and pick the class whose text embedding has the highest cosine similarity. CLIP-style encoders (and successors such as SigLIP, which replaces the softmax with a pairwise sigmoid loss that does not need the whole batch for normalization) are the standard vision towers inside VLMs, and they are used for retrieval, filtering training data and evaluation (CLIP score).

Adapter-style vision-language models

The adapter recipe connects strong unimodal parts:

1. Image encoder, usually a Vision Transformer (ViT) from CLIP or SigLIP. A ViT splits an image into patches (for example $14 \times 14$ pixels) and outputs one feature vector per patch. At $336 \times 336$ resolution with 14-pixel patches you get $24 \times 24 = 576$ patch features.
2. Projector that maps image features into the LLM's embedding space. LLaVA (Liu et al., 2023) used a single linear layer; LLaVA-1.5 used a two-layer MLP. Flamingo (Alayrac et al., 2022) instead used a Perceiver Resampler to compress features into a fixed small number of tokens and inserted gated cross-attention layers into a frozen LLM. BLIP-2 (Li et al., 2023) used a Q-Former, a small transformer with learned queries.
3. LLM that receives the projected image tokens interleaved with text tokens and generates the answer.

Training typically has two stages. Alignment pretraining: freeze encoder and LLM, train only the projector on image-caption pairs so image tokens land in a region the LLM understands. Visual instruction tuning: train the projector and the LLM (fully or with LoRA) on instruction-response data about images; LLaVA generated much of this data by prompting a text-only LLM with captions and bounding boxes. The approach is cheap because it reuses expensive pretrained parts, which is why it became the standard open-model recipe.

Resolution and token budget. High resolution matters for documents and small text, but tokens grow quadratically with side length. A common solution ("any-resolution" tiling) splits a large image into tiles at the encoder's native resolution plus a downscaled global view. For a $1008 \times 672$ screenshot with $336$-pixel tiles: $3 \times 2 = 6$ tiles plus 1 global view, $7 \times 576 = 4{,}032$ image tokens before any compression. Many models then merge neighboring patches (for example $2 \times 2$ pooling, giving $1{,}008$ tokens) to save context.

Native multimodal and any-to-any models

Adapter models bolt vision onto a language model after the fact. Native multimodal models train on mixed-modality data from early in pretraining so all modalities share one representation. Several frontier developers describe their flagship models this way, though most do not publish architectural details.

A published example of the early-fusion approach is Chameleon (Meta, 2024): images are converted into discrete tokens by a learned image tokenizer and mixed with text tokens in one sequence, and a single transformer is trained with next-token prediction over everything. Because images are just tokens, the same model can both read and generate interleaved text and images.

Image tokenization choices:

• Continuous patch embeddings (ViT features): best for understanding, cannot be generated by next-token prediction directly.
• Discrete tokens from a VQ-VAE or VQGAN-style tokenizer: an image becomes, for example, a $32 \times 32$ grid of codebook indices. These can be generated autoregressively, but quantization loses fine detail such as small text.
• Hybrid designs: understanding through continuous features, generation through a diffusion or flow head attached to the LLM, or a decoder that maps LLM outputs into a diffusion model's conditioning.

Any-to-any models accept and produce several modalities (text, image, audio, sometimes video), typically by giving each output modality its own tokenizer or decoder. Speech-native assistants that hear and speak in one model are the most widely deployed instance as of November 2026. Design trade-offs: shared models transfer knowledge across modalities and handle interleaved content naturally, but training is harder to balance, one modality can degrade another, and serving costs rise.

Document understanding

Documents combine text, layout, tables, charts and handwriting. Two approaches:

• OCR pipeline: run OCR (and layout detection), then feed recognized text with coordinates to a language model. Precise for clean scans, auditable, and cheap; fragile on complex layouts, charts and handwriting.
• OCR-free VLMs: feed page images directly (Donut and Pix2Struct were early research examples; general VLMs now do this). They capture layout and visual cues but can hallucinate plausible text, misread digits and lose accuracy on dense small print unless resolution is high.

Production systems often combine them: OCR provides exact strings, the VLM interprets layout and semantics, and validation rules (totals add up, dates parse, IDs match checksums) catch errors. For extraction, ask for structured output with a schema and ask the model to cite where each value appears.

Video understanding

A video is a stream of frames plus audio. Models sample frames (for example 1 frame per second), encode each, often pool or merge tokens across neighboring frames, and add timestamps. Token arithmetic dominates: one hour at 1 fps with 256 tokens per frame is $3{,}600 \times 256 = 921{,}600$ tokens, which is why long-video models compress aggressively, sample adaptively (more frames when scenes change) or use retrieval over clip embeddings. Brief events between sampled frames are simply invisible to the model, a frequent cause of wrong answers about fast actions. Audio transcripts often carry more information than frames for talks and meetings.

Code: zero-shot classification with CLIP

import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

model_id = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(model_id).eval()
processor = CLIPProcessor.from_pretrained(model_id)

labels = ["cat", "dog", "invoice", "bar chart"]
prompts = [f"a photo of a {c}" if c in ("cat", "dog") else f"an image of a {c}"
           for c in labels]
image = Image.open("example.jpg").convert("RGB")   # any local image

inputs = processor(text=prompts, images=image, return_tensors="pt", padding=True)
with torch.no_grad():
    out = model(**inputs)
probs = out.logits_per_image.softmax(dim=-1)[0]    # logits are scaled cosine similarities
for label, p in sorted(zip(labels, probs.tolist()), key=lambda x: -x[1]):
    print(f"{label:10s} {p:.3f}")

logits_per_image already includes the learned temperature (CLIP stores $\log(1/\tau)$ as logit_scale). Prompt wording measurably changes zero-shot accuracy; averaging embeddings over several templates ("a photo of a {}", "a picture of a {}") is a standard improvement.

The contrastive loss itself is a few lines of PyTorch:

import torch.nn.functional as F

def clip_loss(img_emb, txt_emb, temperature=0.07):
    img = F.normalize(img_emb, dim=-1)
    txt = F.normalize(txt_emb, dim=-1)
    logits = img @ txt.T / temperature           # (N, N)
    targets = torch.arange(len(img), device=img.device)
    return 0.5 * (F.cross_entropy(logits, targets) +
                  F.cross_entropy(logits.T, targets))

Applying the same two cross-entropies to the worked example's logit matrix (cosine similarities divided by 0.5) reproduces the 0.222 computed above.

Limitations

• Hallucination: describing objects, text or details that are not in the image, often driven by language priors ("a kitchen usually has a refrigerator"). Benchmarks such as POPE probe this with yes/no questions about object presence.
• Counting, spatial relations and fine detail: patch features and token compression lose small objects and exact positions; models often miscount beyond a handful of objects.
• Text in images: reading small, rotated or low-contrast text depends heavily on input resolution.
• Charts and diagrams: models may read values approximately or infer them from titles rather than the plotted data.
• Prompt injection through images: text embedded in an image or document can carry instructions the model may follow. Treat image content from untrusted sources as untrusted input in agent systems.
• Bias and privacy: models can infer sensitive attributes or identify people; many providers restrict face identification.

Evaluation

Benchmarks probe different skills, so look at several. Representative examples (verify current versions and known contamination issues before relying on scores):

Public scores saturate and leak into training data, and multiple-choice formats can be partly answered from text alone (always check a text-only baseline). For your application, build a private evaluation set from real inputs with ground truth: your invoices, your screenshots, your camera conditions. Report accuracy per field or per question type, not only overall.

Common pitfalls

• Downscaling inputs so small text becomes unreadable, then blaming the model. Check the effective resolution the API or model actually uses.
• Trusting extracted numbers without validation. Add arithmetic checks, cross-field consistency checks and human review for high-stakes fields.
• Assuming video means every frame. Know the sampling rate; ask about events that last longer than the sampling interval or use higher frame rates for short clips.
• Using CLIP similarity as ground truth for fine-grained correctness. It is a coarse, bag-of-concepts signal.
• Ignoring image-borne prompt injection in agents that browse or read documents.

In practice

For classification, search and deduplication, CLIP-style embeddings are cheap and strong. For question answering, extraction and reasoning over images, use a VLM, choosing between an API model and an open adapter-style model based on privacy, cost and accuracy on your own evaluation set. Combine VLMs with OCR and rule-based validation for documents, constrain outputs with schemas, log inputs and outputs for audit, and re-run your evaluation whenever the model version changes. Fine-tuning an open VLM with LoRA on a few thousand domain examples is a common, affordable way to close gaps on specialized forms or imagery.