AI Hardware Explained

Spec sheets for AI computers are full of big numbers, and most of them do not tell you what you actually want to know: will this model fit, and how fast will it answer? Three resources decide almost everything. Learn to read those and you can judge any machine.

The three resources

Think of a model as a huge reference book and the processor as a reader who must flip through the entire book to write each word of an answer.

1. Memory capacity is the size of the desk. The whole book must fit on it, or the reader cannot work at all.
2. Memory bandwidth is how fast the reader can flip pages. Since every word requires going through the book, this sets the writing speed.
3. Compute is how fast the reader thinks. It matters most when reading your question, because a long prompt can be processed many words at a time.

Memory capacity: what fits

The weights of a model take roughly parameters × bits per weight ÷ 8 bytes, plus about 5%. A parameter is one learned number; bits per weight depends on quantization, which means storing each number with fewer bits to save space.

On top of the weights you need room for the KV cache (the model's working memory of the conversation so far, which grows with context length) and about 1.5 GB of runtime overhead.

Worked examples, using the same method as the site's AI Index:

• An 8-billion-parameter (8B) model at about 4.8 bits per weight: 8 × 4.8 ÷ 8 ≈ 4.8 GB, about 5 GB with overhead in the file.
• A 70B model at 4-bit quantization: about 40–42 GB.
• The same 70B model at 16-bit: about 2 bytes per parameter, roughly 140 GB.

The Index treats a model as fitting when the total need is at most 85% of usable memory, leaving headroom for the operating system and longer conversations.

Memory bandwidth: generation speed

Bandwidth is how many gigabytes per second the processor can read from memory. When generating text one token at a time, the chip must read essentially all of the model's active weights for every token. So a model that is 5 GB on a chip with about 1,000 GB/s of bandwidth can, in principle, generate far faster than the same model on a chip with 250 GB/s. The next lesson turns this into a formula.

Compute: prompt processing

Compute is measured in TFLOPS, trillions of floating-point operations per second. Reading your prompt (called prefill) costs about 2 × parameters operations per token, but the tokens can be processed in parallel, so this stage is limited by compute rather than bandwidth. A chip with weak compute will pause noticeably before answering a long prompt, even if it then generates text at a decent pace.

Unified memory versus discrete VRAM

A discrete GPU is a separate graphics card with its own memory, called VRAM. VRAM is very fast but limited in size. As of November 2026, NVIDIA's GeForce RTX 5090 has 32 GB of VRAM at 1,792 GB/s.

A unified-memory machine shares one pool of memory between the CPU and the built-in GPU. The pool can be much larger, but it is usually slower:

On a discrete GPU, usable AI memory is the VRAM. On unified-memory machines, two separate limits apply, and it helps to keep them apart:

• The operating-system reserve. The OS and your other apps need some memory. The site's AI Index models usable memory as total memory minus a reserve: 8 GB on macOS and Windows, 4 GB on Linux or on any machine with 16 GB or less.
• The macOS GPU cap. By default, macOS limits how much unified memory the GPU may use, at roughly 75% of RAM on large configurations (a smaller share on small ones). The cap can be raised, for example with sudo sysctl iogpu.wired_limit_mb=<megabytes>, which lasts until the next reboot. The Index assumes this cap has been raised; on a Mac left at default settings, the real limit is the lower of the two.

On AMD's Ryzen AI Max systems, a 128 GB machine can assign up to 96 GB to the GPU.

Offloading to system RAM

If a model is too big for VRAM, many runtimes can keep part of it in ordinary system RAM. It will run, but slowly, because speed drops toward the bandwidth of system RAM. Dual-channel DDR5-6000, for example, peaks at about 96 GB/s (6,000 million transfers × 8 bytes × 2 channels), far below a high-end GPU. Offloading is a useful way to try a model, not a substitute for enough fast memory.

NPUs: present, but mostly unused for LLMs

An NPU (neural processing unit) is a low-power AI accelerator built into many recent laptop chips. It is efficient for small, always-on tasks like background blur or speech features. As of November 2026, the most popular local LLM tools, such as Ollama and LM Studio, do not run models on the NPU; they use the GPU and CPU. NPU support exists in narrower, vendor-specific tools (for example AMD's FastFlowLM and Lemonade, Intel's OpenVINO GenAI) and as experimental llama.cpp work for Qualcomm's Hexagon NPU. Apple's Neural Engine is not used by MLX. NPUs also typically share the same memory and bandwidth as the rest of the chip, so they would not raise the bandwidth ceiling on generation anyway.

Sparse versus dense TOPS

Vendors often advertise TOPS (trillions of operations per second) or "AI TOPS" measured with 2:4 sparsity, a trick where half of the numbers in a matrix are zero and skipped. Ordinary LLM inference does not use that pattern, so real dense throughput is about half the headline figure. For example, NVIDIA lists the DGX Spark at up to 1 petaFLOP of FP4 compute "with sparsity", so dense FP4 is roughly half that.

Power, heat, and storage

• Power and thermals. The RTX 5090 is rated at 575 W of board power; the DGX Spark ships with a 240 W power supply. High power means more heat, louder fans, and higher electricity costs, and sustained loads can cause thermal throttling in small cases or laptops.
• Storage speed. Models are loaded from disk into memory. A 40 GB model loads in seconds from a fast NVMe SSD and much longer from a hard drive or network share. Storage speed does not affect generation once loaded, but it matters if you switch models often. Model files are large, so plan for plenty of disk space too.

Going deeper

Where bandwidth numbers come from

Peak memory bandwidth is a product of three numbers:

Dual-channel DDR5-6000 has a 64-bit (8-byte) bus per channel: $6000\times10^6 \times 8 \times 2 = 96$ GB/s, as stated above. A graphics card with a 512-bit GDDR7 bus at 28 Gb/s per pin gives $28 \times 512 / 8 = 1{,}792$ GB/s, the RTX 5090 figure. These are theoretical peaks; real kernels reach a fraction, which is why the Index multiplies by 0.55 for decode.

The roofline model

The roofline model (Williams, Waterman and Patterson, 2009) answers a precise question: for a given piece of work, is the chip limited by compute or by memory? It uses three quantities:

• $\pi$: peak compute in FLOP/s,
• $\beta$: peak memory bandwidth in bytes/s,
• $I$: arithmetic intensity, the FLOPs performed per byte moved from memory.

The attainable throughput is

Plotted against $I$, this is a sloped line (memory-bound) that flattens into a horizontal roof (compute-bound). The corner, the ridge point, is at $I^{*} = \pi / \beta$. Work with $I < I^{*}$ is memory-bound; work with $I > I^{*}$ is compute-bound.

Arithmetic intensity of decode and prefill

Most of an LLM's work is multiplying a weight matrix of shape $d_{\text{out}} \times d_{\text{in}}$ by activation vectors. Each weight is used in one multiply and one add per token, so a single token costs $2\,d_{\text{in}} d_{\text{out}}$ FLOPs, while loading the weights at $b$ bits each moves $d_{\text{in}} d_{\text{out}}\, b/8$ bytes. If $n$ tokens share one pass over the weights,

(This ignores reading activations and the KV cache, which only lowers $I$ further for decode.)

• Decode, one user: $n = 1$. At 4.8 bits, $I \approx 3.3$ FLOP/byte. At 16-bit, $I = 1$.
• Decode, batch of $B$ users: $n = B$, so $I \approx 3.3B$ at 4.8 bits. This is why batching raises server throughput.
• Prefill of a 1,000-token prompt: $n = 1000$, $I \approx 3{,}333$ FLOP/byte.

Worked example

Take an illustrative chip with $\pi = 200$ dense TFLOPS and $\beta = 1{,}000$ GB/s. The ridge point is $I^{*} = 200\times10^{12} / 10^{12} = 200$ FLOP/byte.

1. Single-user decode, $I = 3.3$: attainable $= 1{,}000 \text{ GB/s} \times 3.3 = 3.3$ TFLOPS, under 2% of peak. The chip's compute sits mostly idle; bandwidth decides speed.
2. Batch of 32: $I \approx 107$, attainable $\approx 107$ TFLOPS, about 53% of peak. Same bandwidth, about 32 times the useful work per byte.
3. Batch needed to reach the ridge: $16B/4.8 = 200 \Rightarrow B = 60$.
4. Prefill of 1,000 tokens: $I \approx 3{,}333 > 200$, so compute-bound at the full 200 TFLOPS.

def intensity(tokens_sharing_weights, bits):
    """FLOPs per byte of weights: 2 FLOPs per weight per token."""
    return 2 * tokens_sharing_weights / (bits / 8)

def attainable_tflops(peak_tflops, bw_gbs, flops_per_byte):
    return min(peak_tflops, bw_gbs * flops_per_byte / 1000)  # GB/s x FLOP/B = GFLOP/s

peak, bw = 200, 1000                      # illustrative chip: ridge = 200 FLOP/byte
for name, n in [("decode, batch 1", 1), ("decode, batch 32", 32), ("prefill, 1,000 tokens", 1000)]:
    I = intensity(n, 4.8)
    att = attainable_tflops(peak, bw, I)
    print(f"{name:22s} I={I:7.1f} FLOP/B  attainable={att:6.1f} TFLOPS  ({att/peak:.0%} of peak)")

This is the formal reason behind the three-resource picture: single-user decode lives far left of the ridge (bandwidth), and prefill lives far right (compute).

From headline TOPS to a usable number

Vendors quote peak throughput for a specific precision and often with 2:4 sparsity. To compare chips on the dense, 16-bit-class compute that matters for prefill, the Index applies simple conversions: halve a sparse figure, then divide FP8 figures by 2 and FP4 figures by 4 to express them in FP16-class terms (each halving of precision roughly doubles peak throughput on hardware that supports it). For a figure of 1 petaFLOP FP4 with sparsity: dense FP4 ≈ 500 TFLOPS, FP16-class ≈ 125 TFLOPS. The Index then assumes prefill reaches about 30% of that dense peak:

with $F_{\text{dense}}$ in FLOP/s and $N_a$ the active parameters. For a 70B dense model on 125 TFLOPS: $0.3 \times 125\times10^{12} / (1.4\times10^{11}) \approx 268$ tokens/s, so an 8,000-token prompt takes about 30 seconds. These conversions are estimates; a measured prompt-processing benchmark always wins.

Common pitfalls

• Comparing sparse TOPS with dense TFLOPS. Halve sparse figures, and make sure both numbers are for the same precision (INT8, FP8, FP4, FP16 are not interchangeable).
• Adding NPU TOPS to GPU TOPS. The popular LLM runtimes do not use both at once, and both share the same memory bandwidth.
• Treating peak bandwidth as achieved bandwidth. Real decode reaches a fraction of the spec; the Index uses 55%.
• Mixing GB and GiB, or Gb and GB. One GiB is about 1.074 GB, and link speeds in Gb/s are eight times smaller in GB/s.
• Ignoring sustained behavior. Laptops and small cases can throttle under long loads, so short benchmarks overstate long jobs.

In practice

• When comparing machines, find three numbers first: usable AI memory, memory bandwidth, and dense compute. Ignore sparse TOPS headlines.
• Check fit with the rule of thumb: weights + KV cache + 1.5 GB must be at most 85% of usable memory.
• Treat system-RAM offload and NPUs as bonuses, not reasons to buy.
• The AI Hardware & Model Index lists these numbers for many machines and shows which models fit; its methodology tab explains the formulas.