Tokens per Second: Estimating Local AI Speed

"How fast will it run?" is the first question after "will it fit?". The answer is not one number. A local model has two very different speeds, and knowing which one matters for your task saves you from buying the wrong machine.

Two phases: prefill and decode

When you send a prompt, the runtime works in two phases:

• Prefill (prompt processing): the model reads your entire prompt. All prompt tokens can be processed in parallel, so this phase is limited mainly by compute (TFLOPS).
• Decode (generation): the model writes its answer one token at a time. A token is a chunk of text, roughly three-quarters of an English word. For each new token, the chip must read essentially all of the model's active weights from memory, so this phase is limited mainly by memory bandwidth.

Time to first token (TTFT) is how long you wait before the answer starts appearing. It is mostly prefill time. Tokens per second (tok/s) usually refers to decode speed, how quickly the answer streams once it starts.

An analogy: prefill is a chef reading a whole recipe card at a glance; decode is plating each dish one at a time, walking to the pantry for every plate. A fast reader with a slow walk starts quickly but finishes slowly, and the reverse is also possible.

Estimating decode speed

The site's AI Index uses this estimate:

tok/s ≈ 1 ÷ (bytes read per token ÷ (0.55 × bandwidth) + 0.0015 s)

• Bytes read per token ≈ active parameters × bits per weight ÷ 8.
• 0.55 reflects that real software reaches only about 55% of the theoretical bandwidth.
• 0.0015 s (1.5 ms) is fixed per-token overhead.

Active parameters matter because of mixture-of-experts (MoE) models. An MoE model is split into many "expert" sub-networks and uses only a few for each token. All of its weights must fit in memory, but only the active ones are read per token, so an MoE model generates faster than a dense model of the same total size.

Worked example: two machines

Machine A is a discrete GPU with 32 GB of VRAM and 1,792 GB/s (the specification of an RTX 5090 as of November 2026). Machine B is a unified-memory box with 128 GB and 273 GB/s (the specification of NVIDIA's DGX Spark). Effective bandwidth is 0.55 × 1,792 ≈ 986 GB/s for A and 0.55 × 273 ≈ 150 GB/s for B. All models below are at about 4.8 bits per weight, so each billion active parameters is about 0.6 GB read per token.

Two lessons jump out. The fast GPU wins on speed for anything it can hold. The big, slower box wins on what it can hold at all, and an MoE model makes that capacity useful: a model with 100B total parameters can generate faster on Machine B than a 32B dense model, because it reads only 6B parameters per token.

Estimating prefill and time to first token

Prefill costs about 2 × active parameters floating-point operations per token. For an 8B model that is 16 billion operations per prompt token.

Illustrative numbers (sustained compute figures here are assumptions, not specifications of the machines above):

• A 1,000-token prompt to an 8B model is 16 trillion operations. On a chip sustaining 100 TFLOPS, that is about 0.16 seconds.
• An 8,000-token prompt to a 70B dense model is 140 billion × 8,000 ≈ 1,120 trillion operations. At a sustained 50 TFLOPS, that is about 22 seconds before the first word appears.

This is why pasting a long document into a large model on a modest machine can mean a long pause, even when the answer then streams at a reasonable pace. Runtimes that cache a shared prompt prefix (such as a long system prompt) can skip repeated prefill.

Why speed falls as the conversation grows

During decode the model also reads the KV cache, its stored memory of every earlier token in the conversation. The KV cache grows with context length, so each new token costs more bytes.

As a simplification, a typical 8B model using grouped-query attention stores about 128 KB of KV cache per token at 16-bit precision. At 32,000 tokens of context that is about 4.2 GB, which is read alongside the 4.8 GB of weights. On Machine A, bytes per token rise from 4.8 GB to about 9.0 GB, and speed drops from about 157 to about 94 tok/s. Prefill also gets slower per token as context grows, because attention compares each new token with all earlier ones.

Single-user versus batched throughput

Everything above describes single-user speed: one conversation, one token at a time. A server handling many requests can use batching, processing several conversations' next tokens in the same pass over the weights. Since the weights are read once for the whole batch, total throughput across all users can be many times higher than single-user speed, while each individual user sees roughly the same or slightly lower speed. When someone quotes a very high tok/s figure, check whether it is per user or summed across a batch.

Why measured benchmarks beat estimates

The formula is a planning tool. Real speed depends on the runtime, the quantization format, driver versions, and how well software is tuned for a specific chip. As one example, Ollama reported in March 2026 that its preview MLX engine for Apple silicon raised decode on one test model from 58 to 112 tok/s compared with its previous engine, on the same hardware (the new run also used a different 4-bit format). When a trustworthy measured result exists for your exact model, runtime, and machine, prefer it over any estimate.

Going deeper

The decode estimate, formally

Let $N_a$ be active parameters, $b$ bits per weight, $W$ peak memory bandwidth in bytes/s, $\eta = 0.55$ the achieved fraction, and $t_0 = 1.5$ ms the fixed per-token cost (kernel launches, sampling, scheduling). Bytes read per token are $B = N_a\, b/8$, and

Two limits are worth seeing. For large models, $B/(\eta W) \gg t_0$ and speed is simply proportional to bandwidth over model size. For tiny models, the memory term shrinks and $r_{\text{dec}}$ approaches $1/t_0 \approx 667$ tokens/s, which is why very small models do not speed up without limit on fast GPUs.

The Index uses weights only for $B$. At long contexts you should add the KV cache read, $B = N_a b/8 + \text{KV bytes}$, as the earlier section did.

Offloading: a weighted harmonic speed

When a model is too large for VRAM, llama.cpp-style runtimes keep the KV cache and as many weights as fit on the GPU and read the rest from system RAM each token. If a fraction $f$ of the weights lives in RAM with bandwidth $W_r$ and the rest in VRAM with bandwidth $W_g$, the memory time per token is

This is the formula the Index uses for its "GPU+RAM" label. How the Index sets $f$: the GPU holds $0.85 \times \text{VRAM} - \text{KV} - 1.5$ GB of weights; the remainder goes to RAM, which must fit in $0.85 \times (\text{RAM} - 8\text{ GB})$; and if the machine's RAM bandwidth is unknown it assumes 80 GB/s. It does not estimate prefill in this mode, because prompt processing then depends on PCIe transfers and the CPU.

Worked example: a 70B model on a 32 GB GPU

Machine: 32 GB VRAM at 1,792 GB/s, 64 GB of dual-channel DDR5 at 96 GB/s. Model: 70B dense, 4.8 bits, with an architecture of 80 layers, 8 KV heads, head dimension 128.

1. Weights: $70 \times 4.8/8 \times 1.05 = 44.1$ GB.
2. KV cache at 8,192 tokens: $2 \times 80 \times 8 \times 128 \times 2$ bytes $= 327{,}680$ bytes per token, × 8,192 ≈ 2.68 GB.
3. Need: $44.1 + 2.68 + 1.5 \approx 48.3$ GB, more than 32 GB, so it does not fit in VRAM.
4. Weights on the GPU: $0.85 \times 32 - 2.68 - 1.5 \approx 23.0$ GB. In RAM: $44.1 - 23.0 = 21.1$ GB, within $0.85 \times (64-8) = 47.6$ GB. So $f \approx 0.478$.
5. Bytes per token: $70 \times 0.6 = 42$ GB.
6. $t_{\text{mem}} = \frac{42}{0.55}\left(\frac{0.478}{96} + \frac{0.522}{1792}\right) \approx 76.4 \times 0.00527 \approx 0.403$ s.
7. $r \approx 1/(0.403 + 0.0015) \approx 2.5$ tokens/s.

For comparison, the same model entirely in RAM would give about 1.3 tokens/s, and entirely in VRAM (if it fit) about 22.7. The RAM half accounts for about 94% of the memory time: in a harmonic mean, the slowest component dominates. Offload makes a model run; it rarely makes it pleasant, except for mixture-of-experts models where the per-token bytes are small.

A Python estimator that reproduces the Index

ETA_DECODE, ETA_PREFILL = 0.55, 0.30
T0, OVERHEAD_GB, OS_RESERVE_GB = 0.0015, 1.5, 8

def kv_gb(layers, kv_heads, head_dim, ctx, bytes_el=2):
    return 2 * layers * kv_heads * head_dim * bytes_el * ctx / 1e9

def estimate(total_b, active_b, bits, kv, usable_gb, bw, ram_gb=0, ram_bw=80, dense_tflops=None):
    weights = total_b * bits / 8 * 1.05          # GB on disk / in memory
    need = weights + kv + OVERHEAD_GB
    per_tok = active_b * bits / 8                # GB read per generated token
    if need <= 0.85 * usable_gb:
        label, f = "fits", 0.0
    elif need <= usable_gb:
        label, f = "tight", 0.0
    else:                                        # try GPU + system-RAM offload
        on_gpu = 0.85 * usable_gb - kv - OVERHEAD_GB
        in_ram = weights - on_gpu
        if on_gpu <= 0 or in_ram > 0.85 * max(0, ram_gb - OS_RESERVE_GB):
            return {"label": "no", "need_gb": round(need, 1)}
        label, f = "offload", in_ram / weights
    mem_s = per_tok / ETA_DECODE * (f / ram_bw + (1 - f) / bw)
    out = {"label": label, "need_gb": round(need, 1), "ram_share": round(f, 3),
           "decode_tps": round(1 / (mem_s + T0), 1)}
    if dense_tflops and f == 0:                  # no prefill estimate when offloading
        out["prefill_tps"] = round(ETA_PREFILL * dense_tflops * 1e12 / (2 * active_b * 1e9))
    return out

print(estimate(70, 70, 4.8, kv_gb(80, 8, 128, 8192), 32, 1792, ram_gb=64, ram_bw=96))
print(estimate(8, 8, 4.8, kv_gb(32, 8, 128, 8192), 32, 1792, dense_tflops=100))

The first call reproduces the worked example (offload, need 48.3 GB, RAM share 0.478, 2.5 tok/s). The second gives fits, 157 tok/s decode, and about 1,875 tok/s prefill for an 8B model on a 32 GB, 1,792 GB/s GPU with an assumed 100 dense TFLOPS. Notes: usable_gb is VRAM on a discrete GPU, or, as in the Index, unified memory minus the operating-system reserve (8 GB on macOS and Windows, 4 GB on Linux or machines with 16 GB or less), which assumes a Mac's default GPU cap of roughly 75% has been raised; OS_RESERVE_GB in the code is the separate 8 GB the Index keeps back from system RAM when offloading; "tight" means the model loads but leaves under 15% headroom; and weights use the 1.05 file-size factor while bytes per token do not, matching the Index.

Why the constants are what they are

The 0.55, 0.30, and 1.5 ms values are calibration constants chosen so estimates land near measured results across many machines; they are not physical laws. A well-tuned runtime on a well-supported chip can beat them, and an immature backend can fall well short. That is the formal version of "measured benchmarks beat estimates".

Common pitfalls

• Using total instead of active parameters for MoE decode. Memory need uses total parameters; bytes per token use active ones.
• Ignoring the KV cache at long context. At 32K tokens an 8B model's cache can be nearly as large as its weights, and speed drops accordingly.
• Estimating offload speed with GPU bandwidth. The RAM share dominates; use the weighted harmonic formula.
• Comparing batched server throughput with single-user speed. Always ask whether a tok/s figure is per user or summed.
• Expecting tiny models to scale forever. The fixed per-token overhead caps speed (about 667 tok/s at 1.5 ms), whatever the bandwidth.

In practice

• For chat and coding, check decode speed: about 10 tok/s feels slow, 20–30 tok/s comfortable, faster is a bonus.
• For long documents and agents, check prefill too: a weak-compute machine may take many seconds before answering.
• Prefer MoE models on large-but-slower memory machines.
• Use the AI Hardware & Model Index for estimated speeds on specific combinations, and favor measured numbers when available.