Multimodal prompting: images, documents, audio and video

Most of the world's information is not plain text: invoices, scanned contracts, dashboards, slide decks, whiteboard photos, product images, meeting recordings, screen recordings of bugs. Modern multimodal models can take these directly as input, which removes whole pipelines of manual transcription. But visual and audio inputs behave differently from text: they cost tokens in ways that depend on resolution and duration, models misread them in characteristic ways, and they often carry private information nobody meant to share. This lesson covers how to prompt with them well.

Intuition: the model sees a mosaic, not a photograph

A vision-language model does not look at your image the way you do. It cuts the image into a grid of small square tiles called patches, turns each patch into a vector, and reads those vectors much as it reads word tokens. Think of looking at a scene through a mosaic of small frosted-glass squares: large shapes, colours, layout and big text come through clearly, but a 6-pixel-high footnote may fall across two squares and blur away, and counting 47 tiny screws means tracking which square each one landed in. Everything about cost and accuracy follows from this picture: more pixels mean more patches (more tokens, more cost), and details smaller than a patch are fragile.

How images become tokens

The standard design comes from the Vision Transformer (ViT; Dosovitskiy et al., 2020):

1. Resize the image to a size the model supports.
2. Split it into non-overlapping $P \times P$ patches. An image of width $W$ and height $H$ gives about $\lceil W/P \rceil \times \lceil H/P \rceil$ patches.
3. Embed each patch: flatten its $P \times P \times 3$ pixel values and multiply by a learned matrix to get a $d$-dimensional vector, then add position information so the model knows where the patch was.
4. Encode and connect: a vision encoder (often pretrained, for example with CLIP-style image-text contrastive learning; Radford et al., 2021) processes the patch vectors; a projection layer maps them into the language model's embedding space, where they sit in the sequence alongside text tokens. LLaVA (Liu et al., 2023) is a well-documented open example of this "encoder plus projector plus LLM" recipe. Many models also merge or pool neighbouring patches to reduce the token count.

The original ViT used $224 \times 224$ images with $16 \times 16$ patches: $(224/16)^2 = 14^2 = 196$ patch tokens. Production models use larger images and their own patch sizes and merging schemes, and do not always publish them.

A documented provider formula (as of November 2026)

Anthropic's vision documentation states that Claude processes images in 28 by 28-pixel patches, each counted as one visual token, so an image costs

$\text{tokens} = \left\lceil \frac{W}{28} \right\rceil \times \left\lceil \frac{H}{28} \right\rceil$

where $W$ and $H$ are the width and height in pixels after any resizing. Images beyond a model's limits are downscaled first, preserving aspect ratio: on its newer models (described as the high-resolution tier) the documented limits are a 2,576-pixel long edge and 4,784 visual tokens; on other models, 1,568 pixels and 1,568 tokens.

Other providers use different schemes. OpenAI documents patch-based image tokenization with per-model resizing rules and a detail setting that changes how images are preprocessed (and so what they cost); Google's Gemini documentation describes its own per-image and tiling rules. The general rule holds everywhere: token cost grows with the pixel area actually processed. Always check the current documentation for the specific model before budgeting.

Worked example: what a screenshot costs

Using the formula above with the high-resolution tier:

• A $1000 \times 1000$ image: $\lceil 1000/28 \rceil = \lceil 35.71 \rceil = 36$, so $36 \times 36 = 1{,}296$ tokens.
• A full-HD screenshot, $1920 \times 1080$: $\lceil 68.57 \rceil = 69$ and $\lceil 38.57 \rceil = 39$, so $69 \times 39 = 2{,}691$ tokens. (On the standard tier the same image is downscaled to about $1456 \times 819$, costing $52 \times 30 = 1{,}560$ tokens.)
• The same screenshot resized to $1280 \times 720$ first: $\lceil 45.71 \rceil = 46$, $\lceil 25.71 \rceil = 26$, so $46 \times 26 = 1{,}196$ tokens: a 56% saving.
• A 4K image, $3840 \times 2160$: it exceeds the 2,576-pixel edge limit, so it is downscaled to about $2576 \times 1449$ and costs the 4,784-token cap. The extra detail beyond the cap is simply thrown away, but you still pay for the upload bandwidth and latency.

At scale: a pipeline processing 10,000 full-HD screenshots per day at 2,691 tokens each sends 26.9 million image tokens per day. At a hypothetical 3 US dollars per million input tokens that is about 81 dollars per day; resizing to 1280 by 720 cuts it to 11.96 million tokens, about 36 dollars per day. If the task only needs the top toolbar, cropping to that region cuts cost far more and usually improves accuracy because the model is not distracted by the rest of the screen.

import math

def visual_tokens(w, h, patch=28):
    return math.ceil(w / patch) * math.ceil(h / patch)

def fit_to_limits(w, h, max_edge, max_tokens, patch=28):
    # approximate downscale preserving aspect ratio (the exact provider rule may differ slightly)
    s = min(1.0, max_edge / max(w, h))
    while visual_tokens(round(w * s), round(h * s), patch) > max_tokens:
        s *= 0.99
    nw, nh = round(w * s), round(h * s)
    return nw, nh, visual_tokens(nw, nh, patch)

for size in [(1000, 1000), (1920, 1080), (3840, 2160)]:
    print(size, "high-res:", fit_to_limits(*size, 2576, 4784),
          "standard:", fit_to_limits(*size, 1568, 1568))

This prints 1,296 tokens for the square image on both tiers, 2,691 versus about 1,560 for the screenshot and 4,784 versus about 1,560 for the 4K image, matching the documented figures to within a few pixels. In a real pipeline, use Pillow (Image.thumbnail or Image.crop) to resize or crop before upload, so you control exactly what the model sees.

Documents and PDFs: OCR versus native vision

Optical character recognition (OCR) converts an image of text into machine-readable characters. Traditional OCR engines are fast, cheap, deterministic and give character positions, but they struggle with complex layouts, handwriting and tables, and they output text without understanding. Native vision sends the page image straight to a multimodal model, which reads layout, tables, stamps, checkboxes and handwriting in context and can answer questions about them, but costs more, is slower, and can hallucinate plausible text it did not actually see.

Some APIs accept PDFs directly and process both the extracted text and an image of each page, so a page costs text tokens plus image tokens. A 100-page report can therefore cost well over 100,000 tokens; send only the pages you need.

Extracting charts and tables

• Ask for a specific structure: "Return the table as JSON with columns quarter, revenue_musd. Use null for unreadable cells; do not estimate."
• Ask for the reading first, interpretation second: first transcribe axis labels, units and legend, then the values, then the trend.
• Reading values off a chart is estimation from pixels. Expect errors of a few percent of the axis range; ask the model to state uncertainty, and get the underlying data whenever it exists.
• Validate: do row totals match the stated total? Are the years in sequence? Do units match the axis label?

Writing multimodal prompts

Practices that hold across providers:

1. Put the image before the question (most providers' guidance and the same ordering logic as long documents), and label multiple images: "Image 1 is the invoice; Image 2 is the purchase order. List every line where quantities differ."
2. Say what to look at: "Focus on the error dialog in the lower right." Better still, crop to it.
3. Request evidence: "For each extracted field, quote the exact text you read." Quotes can be checked against OCR output; invented values often have no matching text.
4. Allow "unreadable": give the model an explicit way to say it cannot read a value, which reduces fabricated output.
5. Use structured output with a schema for extraction, and validate types and ranges in code.
6. Send adequate resolution, no more: small text needs pixels; large images get downscaled anyway. Check what size the model actually processes.

A compact extraction prompt:

<image: receipt.jpg>
<instructions>
Extract the receipt into JSON matching the schema below.
- Copy text exactly as printed; use null if a field is missing or unreadable.
- amounts are numbers without currency symbols.
- Verify: the sum of line totals should equal subtotal; set "check_passed".
</instructions>
<schema>{"merchant": str, "date": "YYYY-MM-DD", "lines": [{"desc": str, "total": float}],
"subtotal": float, "tax": float, "total": float, "check_passed": bool}</schema>

Audio and video

Audio is usually converted into a spectrogram-like representation and encoded into a sequence of tokens whose count scales with duration (for example, Whisper processes audio as 30-second windows of log-Mel spectrogram; Radford et al., 2022). Natively audio-capable models can capture tone, emphasis, speaker changes and non-speech sounds that a transcript drops, but a speech-to-text step followed by a text model is often cheaper, easier to audit and good enough. Prompt with specifics: "Identify speakers as Speaker A/B, give timestamps for each decision, and note where the audio is unclear."

Video is typically handled as frames sampled at some rate (often around one per second, though providers differ) plus the audio track, so cost scales with duration times frames per second times tokens per frame. A one-hour video can consume a very large share of a context window. Events shorter than the sampling interval can be missed entirely, so for fast events (a UI glitch, a sports play) extract the relevant clip, sample more densely, or send selected key frames yourself. Ask for timestamps so answers can be checked.

Known limitations

Vision-language models are strong at describing scenes, reading clear text, understanding layouts and answering questions about charts. Documented weak points include:

• Counting many small or overlapping objects; Anthropic's documentation, for example, notes counts may be approximate.
• Precise spatial reasoning and localisation: exact positions, coordinates, distances, which line crosses which. Rahmanzadehgervi et al. ("Vision language models are blind", 2024) showed that leading models failed surprisingly often on simple visual tasks such as judging whether two circles overlap or counting line intersections.
• Small, low-resolution, rotated or low-contrast text, especially after downscaling.
• Fine visual comparison between two similar images.
• Hallucinated detail: filling in plausible but absent text or objects, especially when the prompt suggests something should be there.
• Specialist imagery: medical scans and similar inputs need purpose-built, validated systems, not a general assistant.
• Detecting AI-generated images is unreliable; do not use a general model as a deepfake detector.

Mitigations: crop and upscale the region of interest, split dense pages into sections, ask for grid-based or step-by-step counting, cross-check with OCR or classic computer vision (object detectors count far better), and keep a human in the loop for high-stakes outputs.

Privacy of images and recordings

Images and audio leak more than people realise. Screenshots capture notification banners, open tabs, email addresses, API keys in a terminal and customer records. Photos carry faces of bystanders, licence plates, home interiors and, in their EXIF metadata, often the GPS location and device. Recordings capture voices of people who did not consent, which may be biometric data under some privacy laws. Before sending media to any model, especially a cloud API:

• Crop to the needed region and redact sensitive areas; strip EXIF metadata.
• Check the provider's data-retention and training policies and your organisation's rules (see privacy-and-your-data).
• Prefer local models (see why-run-ai-locally) for highly sensitive images.
• Remember that major providers prohibit or restrict identifying real people from their faces.

Common pitfalls

• Sending full-resolution images everywhere: paying for pixels that are downscaled away, while the detail you needed is lost anyway. Crop instead.
• Trusting extracted numbers without validation: totals that do not add up, dates in the wrong format, digits misread (8 and 3, 1 and 7).
• Unlabelled multi-image prompts: the model confuses which image is which.
• Assuming the model watched the whole video: sparse frame sampling misses short events.
• Using a chart image when the data exists: read the spreadsheet, not a picture of it.
• Leaking secrets in screenshots of terminals, dashboards and inboxes.

In practice

Production multimodal pipelines preprocess deliberately: they deskew, crop, resize to a target resolution, strip metadata and split long documents into page batches before calling a model. They combine tools: a text layer or OCR for exact characters, a vision-language model for layout and understanding, and classic detectors for counting and measurement. Prompts label each input, request structured output with quoted evidence and an explicit "unreadable" option, and every extraction is validated in code, with low-confidence cases routed to people. Teams track cost per document and accuracy per field on a labelled evaluation set, and re-run that set whenever they change resolution, model or prompt.