Beyond Text: Images, Audio, Video and Multimodal AI

Why this matters

Much of today's AI does far more than chat in text. It describes photos, transcribes meetings, speaks aloud, and generates images and video. The same few ideas sit underneath all of it, and knowing them helps you understand what these tools can do and where they go wrong.

Embeddings: meaning as coordinates

Start with the idea that ties everything together. An embedding is a list of numbers that represents the meaning of something, whether a word, a sentence, an image or a sound clip, as a point in space. Think of it as "meaning coordinates".

A toy example with only three dimensions:

"Puppy" and "dog" sit close together; "invoice" is far away. Real embeddings use hundreds or thousands of dimensions, and no single number has a neat human meaning, but the principle is the same: things with similar meaning land near each other. Embeddings power search by meaning (finding "car repair" when you typed "fixing my vehicle"), recommendations, and the links between text and images described below.

Image understanding

To a computer, a photo is a grid of numbers: each pixel has values for red, green and blue. A vision model learns, through the same training loop as any neural network, to turn that grid into useful patterns: edges in early layers, then textures, shapes, and finally objects and scenes.

Many modern vision models cut an image into small square patches and treat each patch much like a token, so the same transformer design used for text can process pictures.

A key step was learning to connect pictures and words. OpenAI's CLIP, released in 2021, was trained on about 400 million images paired with their captions. It learned to place a picture and a matching caption close together in a shared embedding space. That lets software ask "which of these descriptions fits this photo?" without being trained on a fixed list of categories.

Image generation: diffusion in plain words

Many of the best-known image generators use diffusion or close variants of it. The idea, simplified:

1. During training, take real images and add random noise step by step until each becomes pure static.
2. Train a network to undo one small step of that: given a noisy image, predict what it looked like slightly less noisy.
3. To create a new image, start from pure random noise and apply the denoiser many times, guided by your text prompt (usually via a text embedding), until a clean picture emerges.

An analogy: a sculptor who has seen millions of statues, handed a rough block and told "a horse", chipping away a little at a time until a horse appears. Because each run starts from different random noise, the same prompt gives a different image each time.

Stable Diffusion, publicly released in August 2022, was an early widely used diffusion model whose weights anyone could download and run. Many newer generators use variations of the same approach.

Speech: listening and talking

Speech-to-text (automatic speech recognition) turns audio into words. Audio is converted into a picture-like representation of which sound frequencies occur over time, and a model maps that to text. OpenAI's Whisper, released as open source in September 2022, was trained on about 680,000 hours of audio paired with transcripts, which is part of why it copes well with accents and background noise.

Text-to-speech goes the other way: a model predicts the sound waves for a given text, now often with natural rhythm and emotion. Modern systems can imitate a specific voice from a short sample, which is useful for accessibility and also a real risk for fraud and impersonation.

Video

Video adds time. Video generators typically extend diffusion: they compress video into a smaller internal representation, cut it into small blocks that span both space and time, and denoise those blocks together so that motion stays consistent from frame to frame. OpenAI described this approach for its Sora model in February 2024, saying it could generate up to a minute of high-definition video.

Video understanding models do the reverse, sampling frames and describing or answering questions about them. Keeping objects, people and physics consistent over many seconds remains hard, and generated video can still show objects appearing, vanishing or moving impossibly.

Multimodal models

A modality is a type of data: text, image, audio or video. A multimodal model handles more than one. Early products chained separate models together, for example speech-to-text, then a language model, then text-to-speech. Each hand-off added delay and lost information such as tone of voice.

Newer models are trained to handle several modalities in one network. When OpenAI introduced GPT-4o in May 2024, it said a single model was trained end-to-end across text, images and audio, and reported average spoken response times of about 320 milliseconds, compared with several seconds for its earlier chained voice mode.

Limits to keep in mind

• Vision models can misread small text, count objects wrongly, or confidently describe things that aren't there.
• Speech recognition is less accurate for under-represented accents, languages and noisy recordings.
• Generated images and voices can look and sound real. Treat unexpected media, especially urgent requests "from" someone you know, with caution.

Going deeper

Measuring closeness: cosine similarity

"Close together" in an embedding space is usually measured by the angle between two vectors rather than the straight-line distance. Cosine similarity is

$\cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}$

where $\mathbf{a} \cdot \mathbf{b}$ is the dot product (multiply matching entries and add) and $\lVert \mathbf{a} \rVert$ is the length of $\mathbf{a}$ (the square root of the sum of its squared entries). The result runs from $-1$ (opposite) through $0$ (unrelated) to $1$ (same direction).

Worked example with the toy table above:

• puppy $\cdot$ dog $= 0.9 \times 0.8 + 0.8 \times 0.9 + 0.1 \times 0.1 = 1.45$. Both lengths are $\sqrt{0.81 + 0.64 + 0.01} = \sqrt{1.46} = 1.208$. Cosine $= 1.45 / 1.46 = 0.993$.
• puppy $\cdot$ invoice $= 0.09 + 0 + 0.09 = 0.18$. The invoice length is $\sqrt{0.82} = 0.906$. Cosine $= 0.18 / (1.208 \times 0.906) = 0.165$.

import numpy as np
emb = {
    "puppy":   np.array([0.9, 0.8, 0.1]),
    "dog":     np.array([0.8, 0.9, 0.1]),
    "invoice": np.array([0.1, 0.0, 0.9]),
}
def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
for a, b in [("puppy", "dog"), ("puppy", "invoice"), ("dog", "invoice")]:
    print(f"{a:>6} vs {b:<8} {cosine(emb[a], emb[b]):.3f}")

The code prints 0.993, 0.165 and 0.155. Many systems normalise every embedding to length 1 when it is stored; the cosine is then just the dot product, which is fast to compute across millions of items.

How CLIP builds a shared space

CLIP uses two encoders: one turns an image into a vector, the other turns a caption into a vector, both normalised to length 1. Training takes a batch of $N$ matching image–caption pairs and builds an $N \times N$ table of cosine similarities: row $i$ is image $i$ compared with every caption. The correct pairs sit on the diagonal. The loss is a softmax cross-entropy that pushes each diagonal entry up and every other entry in its row and column down. This style of training, which learns by pulling matching pairs together and pushing mismatched pairs apart, is called contrastive learning.

The payoff is zero-shot classification. To sort photos into "dog", "cat" or "invoice" without any training for those labels, embed a caption for each ("a photo of a dog", and so on), embed the photo, and pick the caption with the highest cosine similarity. Wording matters: the CLIP authors found that a caption template such as "a photo of a {label}" usually works better than the bare label, because captions in the training data are rarely a single word. Zero-shot accuracy therefore depends partly on writing labels the way real captions are written, and it should always be checked on a labelled sample before use.

Images as tokens

The Vision Transformer, introduced in a paper titled "An Image is Worth 16x16 Words", cuts an image into square patches and treats each patch as a token. A $224 \times 224$ pixel image cut into $16 \times 16$ patches gives $(224 / 16)^2 = 14^2 = 196$ patches. Each patch holds $16 \times 16 \times 3 = 768$ numbers (three colour channels), which a learned linear layer maps to an embedding.

This explains why image size matters for cost. Standard attention compares every token with every other, so the work grows with the square of the token count: $196^2 = 38{,}416$ pairs. Doubling the side to $448$ pixels quadruples the patches to $784$ and multiplies the pairs by 16, to $614{,}656$. Assistants that accept images typically resize or tile them for this reason, and charge more tokens for larger images.

Diffusion, a little more formally

In the influential 2020 formulation called denoising diffusion probabilistic models (DDPM), the noising process has a convenient closed form. After $t$ steps, a clean image $x_0$ becomes

$x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I)$

where $\epsilon$ is fresh random noise and $\bar{\alpha}_t$ is a number that falls from about 1 (almost clean) to about 0 (pure noise) as $t$ goes from 0 to the last step $T$ (the DDPM paper used $T = 1000$). A network $\epsilon_\theta$ is trained to look at $x_t$, the step number $t$ and, for text-to-image, the prompt, and predict the noise:

$\mathcal{L} = \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2$

Worked example for one pixel. Let $x_0 = 0.8$ and $\bar{\alpha}_t = 0.64$, so $\sqrt{\bar{\alpha}_t} = 0.8$ and $\sqrt{1 - \bar{\alpha}_t} = 0.6$. Draw $\epsilon = -0.5$. Then $x_t = 0.8 \times 0.8 + 0.6 \times (-0.5) = 0.64 - 0.30 = 0.34$. If the network predicts the noise perfectly, the clean value can be recovered by rearranging: $\hat{x}_0 = (0.34 - 0.6 \times (-0.5)) / 0.8 = 0.64 / 0.8 = 0.8$. Generation runs this in reverse from pure noise, removing a little predicted noise at each step; modern samplers need far fewer steps than the training schedule.

Audio as a picture

Speech models rarely read raw sound waves. The original Whisper models, for example, resample audio to 16,000 samples per second, convert it into an 80-channel log-Mel spectrogram, computed on 25-millisecond windows every 10 milliseconds, and process audio in 30-second chunks (a later version, large-v3, uses 128 Mel channels). A 30-second chunk is $16{,}000 \times 30 = 480{,}000$ samples, but only $30 / 0.01 = 3{,}000$ spectrogram frames of 80 numbers each: a compact, image-like grid that a transformer can attend over.

Common pitfalls

• Mixing embedding models. Vectors from two different embedding models live in unrelated spaces. Comparing them, or searching an index built with one model using queries from another, gives meaningless scores.
• Forgetting to normalise. A raw dot product rewards long vectors as well as similar directions. Use cosine similarity or normalise first.
• Trusting numbers read from images. Vision models can misread digits, units and small print. Check any figure that matters against the source.
• Feeding huge images blindly. More pixels means more tokens and cost, and the model may downscale anyway. Crop to the region that matters.
• Assuming detectors can reliably spot generated media. Detection is an arms race; provenance records and verification through a second channel are more dependable.
• Ignoring licences. Check the licence and terms of the generator and of any model weights before using outputs commercially.

In practice

• Try giving an assistant a photo of a chart, a receipt or an error screen; it is often faster than typing a description, but check numbers it reads out.
• Use transcription for meetings and lectures, then skim for errors in names and technical terms.
• When generating images, iterate: vary the prompt and generate several options, since each run starts from new random noise.
• Agree a verification phrase with family or colleagues for urgent voice or video requests involving money.