Choosing a Machine for Local AI

Buying hardware for local AI is easy to get wrong. A fast machine with too little memory cannot run the models you want, and a huge-memory machine with low bandwidth may run them too slowly to enjoy. This lesson turns the earlier lessons into a buying process. It deliberately avoids quoting prices, which change often; check the AI Hardware & Model Index for current prices and fit.

Start with the goal, not the hardware

Answer three questions first:

1. What is the largest model you want to run? This sets the memory you need.
2. How fast must it answer? This sets the bandwidth you need.
3. What else will the machine do? A gaming PC, a portable laptop, and a silent office box are different purchases.

An analogy: buying for local AI is like buying a truck. First decide the heaviest load you will carry (memory), then how fast you need to drive with it (bandwidth). Horsepower (compute) matters too, but a fast truck that cannot hold the load is useless.

Priority order: memory first, then bandwidth

• Memory capacity first. If the model does not fit, nothing else matters. Recall the rule from earlier: weights + KV cache + about 1.5 GB should be at most 85% of usable memory.
• Bandwidth second. It sets decode speed. Divide 0.55 × bandwidth by the bytes read per token to estimate tokens per second.
• Compute third. It sets prompt-processing speed, which matters for long documents and agents.
• Then everything else: noise, power, size, operating system, and software support.

Remember the "usable" part. On a discrete GPU, usable memory is the VRAM. On a unified-memory machine, keep two numbers apart:

• The AI Index's figure is total memory minus an operating-system reserve: 8 GB on macOS and Windows, 4 GB on Linux or on machines with 16 GB or less. For a Mac, this assumes the GPU memory cap has been raised. A 64 GB Mac counts as 56 GB usable, and 85% of that is about 47.6 GB.
• A Mac at default settings lets the GPU use only part of unified memory: roughly 75% on large configurations (a smaller share on small ones) until you raise the cap, for example with sudo sysctl iogpu.wired_limit_mb=<megabytes>, which lasts until reboot. At the default, the same 64 GB Mac has about 48 GB usable, and 85% of that is about 41 GB.

The main classes of machine

As of November 2026, these are the common options.

Laptops and desktops with 16–32 GB. Most people already own one. They comfortably run small models (roughly 3B–8B parameters at 4-bit), which handle drafting, summarizing, and simple coding help well. Apple silicon laptops use unified memory, so a 32 GB MacBook has about 24 GB usable by the Index's measure (32 − 8 GB), and somewhat less at the default GPU cap; Windows laptops with a discrete GPU are limited by that GPU's VRAM, often 8–16 GB.

Unified-memory machines with large pools. These trade bandwidth for capacity:

• Apple Mac Studio and higher-memory MacBook Pro configurations; for example, the M5 Ultra Mac Studio announced in August 2026 offers up to 512 GB at up to about 1.2 TB/s.
• AMD Ryzen AI Max+ 395 systems with up to 128 GB at 256 GB/s, of which up to 96 GB can be assigned to the GPU.
• NVIDIA GB10 systems such as the DGX Spark, with 128 GB at 273 GB/s, running Linux with NVIDIA's software stack.

These are the most practical way to run 70B dense models and large mixture-of-experts (MoE) models on one quiet box. Generation is slower than on a big GPU, and prompt processing can be noticeably slower on some of them.

Single big-VRAM GPUs. A desktop with one high-end card is the fastest option for models that fit. The GeForce RTX 5090 has 32 GB at 1,792 GB/s and 575 W board power. The workstation RTX PRO 6000 Blackwell has 96 GB at about 1,792 GB/s. These suit people who want maximum speed on small and mid-sized models, or who also game, render, or train.

Workstations and multi-GPU builds. Two or more GPUs, or a professional card in a workstation, extend capacity while keeping high bandwidth. They cost more, draw more power, and need more setup. The next lesson covers what splitting a model across GPUs does and does not do.

Decision table by target model size

Sizes assume about 4.8 bits per weight with a moderate context (a few thousand tokens). Longer contexts need more.

Two notes. First, a 64 GB Mac at the default GPU cap does not fit a 70B model at this quantization under the 85% rule. With the cap raised (the Index's assumption, 56 GB usable), it fits only at short context, roughly 6,000 tokens or fewer; a smaller quantization buys more room. Second, MoE models suit unified-memory machines especially well, because they need lots of capacity but read only their active parameters per token.

Matching goals to machines

• "I want private chat and writing help." Use the laptop you have with a 4B–8B model.
• "I want a capable coding or research assistant at home." Aim for enough memory for 27B–32B dense or a mid-sized MoE model: a 32 GB GPU for speed, or a 64–128 GB unified-memory machine for flexibility.
• "I want to run the largest open models." You need 128 GB or more of usable memory, or a cluster, and should accept slower generation.
• "I want to serve a small team." Favor high-bandwidth GPUs and a server runtime with batching.

Common mistakes

• Buying for compute (TOPS) instead of memory.
• Forgetting the operating-system reserve on unified memory.
• Ignoring context: a 100,000-token context can add many gigabytes of KV cache.
• Treating NPUs or system-RAM offload as a way to run big models fast.

Going deeper

The decision table is a summary of three inequalities. Writing them out lets you answer questions the table cannot, such as "what is the biggest model this machine can hold?" or "how much context do I have left?".

Inequality 1: fit

With usable memory $U$ (GB), total parameters $N$ (billions), bits per weight $b$, KV cache $K$ (GB), and 1.5 GB overhead, the Index's fit rule is

Solving for $N$ gives the largest model that fits:

Example: a 64 GB Mac at the default GPU cap (about 75%) has $U = 48$. At 4.8 bits ($1.05 \times 0.6 = 0.63$ GB per billion parameters) and ignoring the cache, $N_{\max} = (40.8 - 1.5)/0.63 \approx 62$B. With a 70B-class model's 8K-token cache of about 2.7 GB, $N_{\max} \approx 58$B. So 70B misses by roughly 10–17%. With the cap raised, the Index uses $U = 64 - 8 = 56$: $N_{\max} = (47.6 - 1.5)/0.63 \approx 73$B without cache and about 69B with the 8K cache, so a 70B model fits only if its cache stays under $47.6 - 1.5 - 44.1 = 2.0$ GB, roughly 6,000 tokens for that architecture.

Inequality 2: speed

To reach a target decode speed $r$ with $N_a$ active parameters, invert the decode formula:

For 20 tok/s: $1/r - t_0 = 0.0485$ s. A 32B dense model at 4.8 bits (19.2 GB per token) needs $W \ge 19.2 / (0.55 \times 0.0485) \approx 720$ GB/s. A 70B dense model (42 GB per token) needs about 1,575 GB/s, which only the fastest single devices offer. A machine at about 1,200 GB/s would give roughly 15 tok/s on that 70B model.

Inequality 3: context

Once the weights are placed, what remains is your context budget:

where $\kappa = 2 \times \text{layers} \times \text{KV heads} \times \text{head dim} \times \text{bytes per element}$ is the KV cache per token. For a 32B model with 64 layers, 8 KV heads, head dimension 128, and a 16-bit cache, $\kappa = 262{,}144$ bytes. On a 32 GB GPU: $27.2 - 20.16 - 1.5 = 5.54$ GB of budget, or about 21,000 tokens. The same GPU running an 8B model has room for more than 150,000 tokens. Quantizing the KV cache to 8 bits, which several runtimes support, halves $\kappa$ and doubles the budget at a small quality cost.

Putting it in code

def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_el=2):
    return 2 * layers * kv_heads * head_dim * bytes_el

def max_params_b(usable_gb, bits, kv_gb=0.0):
    """Largest model (billions of params) that passes the 85% fit rule."""
    return (0.85 * usable_gb - kv_gb - 1.5) / (bits / 8 * 1.05)

def max_context(usable_gb, params_b, bits, kv_per_token):
    weights = params_b * bits / 8 * 1.05
    budget_gb = 0.85 * usable_gb - weights - 1.5      # what is left for the KV cache
    return max(0, int(budget_gb * 1e9 / kv_per_token))

def bandwidth_needed(active_b, bits, target_tps):
    return active_b * bits / 8 / (0.55 * (1 / target_tps - 0.0015))

print(round(max_params_b(48, 4.8), 1))                              # 64 GB Mac, default GPU cap
print(round(max_params_b(56, 4.8), 1))                              # same Mac, Index assumption (cap raised)
print(max_context(32, 32, 4.8, kv_bytes_per_token(64, 8, 128)))     # 32B on a 32 GB GPU
print(max_context(32, 32, 4.8, kv_bytes_per_token(64, 8, 128, 1)))  # same, 8-bit KV cache
print(round(bandwidth_needed(32, 4.8, 20)))                         # GB/s for 20 tok/s

Output: 62.4 and 73.2 (billion parameters), 21,133 tokens, 42,266 tokens, and 720 GB/s. With these four functions you can screen any shortlist: drop machines that fail the fit inequality for your target model at your target context, then rank the survivors by how far their bandwidth exceeds the requirement.

Energy per token

Power draw matters more once you divide it by speed. Energy per generated token is $E = P / r$ joules, where $P$ is power in watts and $r$ is tokens per second. Using board or adapter ratings as rough upper bounds (real draw during decode is often lower): a 575 W GPU decoding a 32B model at about 48 tok/s uses at most about 12 J per token, or $12 \times 10^6 / 3.6\times10^6 \approx 3.3$ kWh per million tokens; a box limited to 240 W decoding the same model at about 7.7 tok/s uses up to about 31 J per token, or about 8.6 kWh per million. The lower-power machine is not automatically the more efficient one; a fast device that finishes quickly and then idles can use less energy per token. Measure at the wall for real numbers.

A worked buying decision

A user wants a 70B dense model at no less than 10 tok/s. Option A is a 32 GB GPU at 1,792 GB/s; option B is a unified-memory box with about 120 GB usable at 273 GB/s.

1. Fit: about 48 GB needed. A fails ($0.85 \times 32 = 27.2$); B passes.
2. Speed on B: $42/(0.55 \times 273) + 0.0015 \approx 0.281$ s, about 3.6 tok/s, below the target.
3. Bandwidth needed for 10 tok/s: $42/(0.55 \times 0.0985) \approx 775$ GB/s.

Neither machine meets the goal. The professional answer is to change one constraint: accept a mixture-of-experts model with similar quality and far fewer active parameters (B becomes fast enough), choose a 96 GB GPU, or use two 32 GB GPUs. The inequalities make that trade explicit instead of leaving it to marketing.

A final check before buying: run the three inequalities at the context length you will actually use, not the default. A machine that fits a model comfortably at 8K tokens may fail at 64K, and agents and document workflows routinely use long contexts. If the budget is tight, a smaller quantization, an 8-bit KV cache, or a model with fewer KV heads usually buys more room than a hardware upgrade.

In practice

• Write down your target model size and minimum acceptable speed, then filter machines by memory first and bandwidth second.
• Use the decision table for a first cut, then check exact fit and estimated speed in the AI Hardware & Model Index, which tracks current prices.
• Before buying, test the model you care about on hardware you can borrow or rent, so you know it is good enough for your work.