AI Hardware & Model Index

How the numbers are made

• Usable AI memory = the vendor's stated GPU-allocatable memory or VRAM; otherwise total memory minus an OS reserve (8 GB on macOS/Windows, 4 GB on Linux or small systems).
• Memory need = model file size (or parameters × bits ÷ 8 × 1.05) + KV cache for the chosen context + 1.5 GB runtime overhead. KV cache uses the model's layers, KV heads and head size when published.
• Fit: Fits when need ≤ 85% of usable memory, Tight up to 100%, otherwise No.
• Est. decode tok/s = 1 ÷ (time to read the active weights at 0.55 × bandwidth + 1.5 ms fixed per-token overhead), where bytes per token = active parameters × bits ÷ 8. The overhead term is calibrated against the measured benchmarks below and keeps small-MoE estimates from running far too high. Single user; real results vary with runtime and settings.
• Speed estimates run optimistic for MoE models with very few active parameters (roughly under 6B active): at that size runtime overhead, expert routing and memory latency dominate, so real decode speeds are often well below the bandwidth ceiling shown. Where a model's active parameter count is not published, no speed is shown.
• Apple silicon: macOS caps GPU-usable memory at roughly 75% by default; fit figures assume the limit has been raised (see each Mac's page).
• Est. prefill tok/s = 0.3 × dense compute ÷ (2 × active parameters), shown only where the vendor publishes compute. Vendor FP4/FP8 "AI TOPS" figures quoted with 2:4 sparsity are marked "sparse" and halved, because inference does not use sparsity.
• GPU+RAM offload (discrete-GPU machines only): when a model is too big for VRAM but fits in VRAM plus system RAM (minus an 8 GB OS reserve), it is marked GPU+RAM. The KV cache and as much of the model as fits stay on the GPU; the rest is read from system RAM every token, so speed = 0.55 ÷ (bytes per token × (share in RAM ÷ RAM bandwidth + share in VRAM ÷ GPU bandwidth)). Prompt processing is much slower in this mode and is not estimated.
• Measured benchmarks: where a published single-user benchmark exists for a model on a machine (llama.cpp, MLX, Ollama, vLLM), it replaces the estimate up to 32K context and is shown in green as MEASURED with a link to the source.
• Largest tier = biggest dense model that Fits at ~4-bit and 8K context.
• Specs come only from vendor or official pages linked on each item; unpublished fields stay blank. Prices are dated snapshots; anything older than 14 days is marked "recheck". This index is refreshed weekly by an automated research run, with uncertain changes held for human review.

Glossary

Token

The unit a model reads and writes; roughly ¾ of an English word.

Tokens/sec (decode)

How fast a model writes its answer for one user. Mostly limited by memory bandwidth.

Prefill / prompt processing

How fast a model reads your prompt before answering. Mostly limited by compute (TFLOPS).

TTFT

Time to first token: the wait between sending a prompt and the first word of the reply.

Memory bandwidth (GB/s)

How many gigabytes per second the chip can read from memory. The single biggest driver of local chat speed.

Unified memory

CPU and GPU share one memory pool, so large models fit without a big discrete GPU.

VRAM

Dedicated memory on a discrete GPU. Fast, but the model must fit inside it to run at full speed.

Usable AI memory

The part of memory a model runtime can actually use, after the OS and any vendor GPU-allocation limit.

TFLOPS / PFLOPS

Trillions / quadrillions of math operations per second at a stated precision (FP4, FP8, FP16).

TOPS

Trillions of integer operations per second, usually INT8; the usual NPU rating.

NPU

Neural processing unit: a low-power AI accelerator. Only helps when the runtime supports it.

FP4 / FP8 / FP16 / BF16

Number formats. Fewer bits means smaller, faster models with some quality loss.

Quantization (Q4, Q8)

Storing weights with fewer bits. Q4 cuts a model to roughly a quarter of its 16-bit size.

GGUF / MLX / safetensors

File formats for model weights: GGUF for llama.cpp and Ollama, MLX for Apple silicon, safetensors for most others.

Parameters (B)

The model's learned weights, in billions. More parameters usually means more capable and more memory.

MoE (mixture of experts)

A model that uses only some of its parameters per token. All experts must fit in memory; only the active ones set the speed.

Active parameters

Parameters used for each token in an MoE model. Drives speed, not memory.

Context window

How many tokens the model can consider at once (prompt + answer).

KV cache

Memory that stores the context while generating. Grows with context length and number of users.

Offload

Running part of a model from system RAM when it does not fit in GPU memory. Works, but much slower.

Inference runtime

Software that runs the model: Ollama, LM Studio, llama.cpp, MLX, vLLM and others.

Tensor / pipeline parallelism

Ways to split one model across several GPUs or machines.

Open weights

The model file is downloadable to run yourself, under the stated license.