Clustering and Multi-GPU Setups

When a model is too big for one device, the natural idea is to combine several. That works for fitting the model, but it often surprises people on speed. This lesson explains why, so you can tell when a second GPU or a second machine is worth it.

Two ways to split a model

A language model is a stack of layers, each of which transforms the text representation and passes it to the next. There are two main ways to divide that stack.

Pipeline parallelism splits the model by depth. Device 1 holds layers 1–40, device 2 holds layers 41–80. For each token, device 1 does its part, sends a small result to device 2, and waits. Only one device works at a time for a single conversation.

Tensor parallelism splits each layer by width. Every device holds a slice of every layer, and all devices work on the same token at once, then combine their partial results. This happens at every layer, for every token.

An analogy: pipeline parallelism is an assembly line where each station finishes its step before the next starts. Tensor parallelism is several cooks preparing the same dish at once, each chopping part of the ingredients, constantly passing bowls back and forth. The cooks finish faster only if passing bowls is quick.

Why the link matters

Devices exchange data over an interconnect. Compare these with the memory bandwidth figures from earlier lessons (hundreds to thousands of GB/s):

Two properties matter:

• Bandwidth, how much data per second the link carries.
• Latency, the delay before any data arrives, often measured in microseconds.

Pipeline parallelism sends little data per token (the activations at one boundary, often tens of kilobytes), so it works over slower links. Tensor parallelism communicates at every layer, so latency adds up quickly: a model with 80 layers may need well over a hundred small exchanges per token. That is why tensor parallelism has traditionally needed very fast, low-latency links such as NVLink.

RDMA (remote direct memory access) reduces latency by letting one machine write directly into another's memory without involving the operating system on each transfer. It is central to recent desktop clustering.

What clustering helps with

• Fitting bigger models. Two machines with 128 GB each can hold a model that neither can hold alone. For example, NVIDIA's DGX Spark materials say two Sparks connected through their ConnectX-7 ports can work with models up to about 405 billion parameters, versus about 200 billion for one.
• Serving more users. With batching, a cluster processes more conversations at once, raising total throughput.
• Training and fine-tuning. With data parallelism, each machine holds a full copy of the model and trains on different data; Apple's WWDC26 session showed fine-tuning a 9B model at about 600 tokens per second on four M3 Ultra Macs versus about 180 on one.

What it often does not help with

For a single user, pipeline parallelism usually gives no speedup in generation. Each token still passes through every layer in sequence, just on different devices, plus added link latency. If both devices have the same bandwidth, single-user decode is roughly as fast as one device would be if it had enough memory, or slightly slower. Apple's session states this directly: pipeline splitting does not speed up per-token inference.

Tensor parallelism can speed up single-user generation, because each device reads only its slice of the weights per token, but only when the link is fast enough that communication does not eat the gains. Apple reported that in one live demo, four M3 Ultra Macs connected in a full mesh over Thunderbolt 5 with RDMA generated tokens from a 27B model at nearly three times the single-machine rate, and noted the speedup depends on model size and architecture. Results over ordinary Ethernet or without RDMA are usually much weaker.

Current desktop options

As of November 2026:

• Multiple GPUs in one PC. In the RTX 30 series NVIDIA kept NVLink only on the top-end RTX 3090, and it removed the connector entirely from the RTX 40 series. Current consumer multi-GPU setups therefore communicate over PCIe. Pipeline-style splitting works well for fitting models; tensor parallelism works but is limited by PCIe.
• Apple Mac clusters. macOS Tahoe 26.2, released in December 2025, added RDMA over Thunderbolt 5. MLX uses it through Apple's JACCL communication library, and you enable it in system settings. MLX's language-model tools use tensor parallelism by default, with a pipeline option.
• NVIDIA DGX Spark pairing. Each Spark has a ConnectX-7 adapter with QSFP ports supporting 200 Gb Ethernet and RDMA. Two can be cabled directly, or more through a switch, and NVIDIA publishes guides for running vLLM across them.
• Software-only clustering. Tools like llama.cpp's RPC backend can split a model across ordinary networked computers, which helps fit models but is slow over typical home networks.

A worked example

Say you want a model that needs about 200 GB, and you have two machines with 128 GB of usable memory and 273 GB/s each (about 150 GB/s effective, using the 0.55 factor). Neither fits it alone; together they do, since 200 GB is under 85% of 256 GB (about 218 GB). With a pipeline split, single-user decode speed is about what one 273 GB/s machine would give if it had 256 GB, minus link overhead. If the model is dense, a 200 GB need at around 4.8 bits means roughly 300B parameters, or about 180 GB read per token: roughly 1 ÷ (180 ÷ 150 + 0.0015) ≈ 0.8 tok/s. If it is an MoE model with about 10B active parameters (about 6 GB per token), the same estimate gives about 24 tok/s. Clustering made the model possible; the model's design decided whether it was pleasant to use.

Going deeper

What tensor parallelism communicates

In the Megatron-LM scheme (Shoeybi et al., 2019), each transformer layer is split so that devices need to combine results only twice per layer in the forward pass: once after the attention block and once after the feed-forward block. Each combination is an all-reduce: every device contributes a partial result vector, and every device ends up with the sum. For one decoding token the vector is the hidden state, $d$ values at $s$ bytes each, so the message size is

For a 70B-class model with $d = 8192$ and 16-bit activations, $M = 16{,}384$ bytes, only 16 KB. With $L$ layers there are $2L$ all-reduces per token: 160 for an 80-layer model.

The alpha–beta cost model

A standard way to price communication is the alpha–beta model: sending a message of $m$ bytes costs $\alpha + m/\beta$, where $\alpha$ is per-message latency and $\beta$ is link bandwidth. A ring all-reduce over $p$ devices takes $2(p-1)$ steps, each moving $M/p$ bytes:

Per decoded token, tensor parallelism then costs roughly

where $B/(p\,\eta W)$ is each device reading its $1/p$ share of the weights in parallel. Pipeline parallelism over $p$ stages instead costs

because stages run one after another for a single user: memory time is not divided, but there are only $p-1$ hops.

Worked example

Two GPUs, each 1,792 GB/s, a 70B-class dense model at 4.8 bits ($B = 42$ GB, $L = 80$, $d = 8192$). The latency values below are illustrative assumptions; measure your own link.

1. Memory time, TP: $42 / (2 \times 0.55 \times 1792) \approx 21.3$ ms.
2. PCIe 5.0 x16 ($\beta \approx 64$ GB/s), assume $\alpha = 10\,\mu$s. One all-reduce: $2\times10 + 1 \times 16{,}384/64\times10^9 \approx 20 + 0.26 = 20.3\,\mu$s. Times 160: about 3.2 ms.
3. $t_{\text{TP}} \approx 21.3 + 3.2 + 1.5 = 26.0$ ms, about 38 tok/s.
4. A single device of the same bandwidth holding the whole model: $42/(0.55\times1792) + 1.5 \approx 44.1$ ms, about 22.7 tok/s. TP gives about 1.7 times the speed.
5. Over a 25 GB/s network link with $\alpha = 50\,\mu$s: each all-reduce ≈ 100.7 µs, times 160 ≈ 16.1 ms, so $t_{\text{TP}} \approx 38.9$ ms, about 25.7 tok/s. Most of the gain is gone.
6. Pipeline over that same network link: $44.1 + (50 + 0.66)\,\mu$s ≈ 44.2 ms, about 22.6 tok/s, essentially single-device speed, as the earlier section said.

Notice that in every case the bandwidth term is under 1 µs per all-reduce; latency dominates decode communication. Solving $21.3 + 160 \times 2\alpha + 0.04 + 1.5 < 44.1$ (all in ms) gives a break-even latency of about 66 µs: above that, two-way TP over this link is slower than one big device. Prefill is different: messages scale with prompt length ($n d s$), so a 1,000-token prompt sends about 16 MB per all-reduce, and bandwidth matters again.

ETA, T0 = 0.55, 0.0015

def allreduce_s(msg_bytes, p, alpha_s, link_bytes_s):
    """Ring all-reduce, alpha-beta model: 2(p-1) steps, each moving msg/p bytes."""
    return 2 * (p - 1) * alpha_s + 2 * (p - 1) / p * msg_bytes / link_bytes_s

def tp_decode_tps(gb_per_token, bw_gbs, p, layers, hidden, alpha_us, link_gbs, act_bytes=2):
    mem = gb_per_token / p / (ETA * bw_gbs)              # each device reads 1/p of weights
    comm = 2 * layers * allreduce_s(hidden * act_bytes, p, alpha_us * 1e-6, link_gbs * 1e9)
    return 1 / (mem + comm + T0)

def pp_decode_tps(gb_per_token, bw_gbs, p, hidden, alpha_us, link_gbs, act_bytes=2):
    mem = gb_per_token / (ETA * bw_gbs)                  # stages run one after another
    hops = (p - 1) * (alpha_us * 1e-6 + hidden * act_bytes / (link_gbs * 1e9))
    return 1 / (mem + hops + T0)

# 70B-class dense model, 4.8 bits (42 GB read per token), 80 layers, hidden 8192
print(round(tp_decode_tps(42, 1792, 2, 80, 8192, alpha_us=10, link_gbs=64), 1))  # ~38.4
print(round(tp_decode_tps(42, 1792, 2, 80, 8192, alpha_us=50, link_gbs=25), 1))  # ~25.7
print(round(pp_decode_tps(42, 1792, 2, 8192, alpha_us=50, link_gbs=25), 1))      # ~22.6

This model omits real effects (overlapping communication with compute, kernel launch costs that also grow with $p$, uneven splits, and the fact that real libraries switch algorithms for small messages), so use it to reason about direction and order of magnitude, then measure.

Practical constraints on the split

• Tensor-parallel degree normally has to divide the number of attention heads evenly (KV heads are usually split too, or replicated when there are fewer KV heads than devices), so 2, 4, or 8 devices are common and 3 often is not.
• In a pipeline, the slowest stage sets the pace; mixing a fast and a slow device wastes the fast one unless layers are assigned in proportion to speed.
• Mixture-of-experts models add expert parallelism, where different devices hold different experts and tokens are routed between them with all-to-all exchanges; that traffic grows with batch size.

Common pitfalls

• Expecting pipeline splits to speed up one user. They add capacity, not single-stream speed.
• Reading link speeds in the wrong unit. 80 Gb/s is 10 GB/s; 200 Gb Ethernet is 25 GB/s.
• Looking only at link bandwidth. For decode, per-message latency multiplied by $2L$ all-reduces per token usually dominates.
• Mixing unequal devices. The slowest device or stage sets the pace for everyone.
• Quoting aggregate cluster throughput as user speed. Batched totals across users are not what one person experiences.
• Ignoring divisibility. A tensor-parallel degree that does not divide the head count may not be supported by the runtime.

In practice

• Cluster to fit models you cannot otherwise run, or to serve more users, not to make a model that already fits much faster.
• If you want single-user speedups, you need tensor parallelism and a fast, low-latency link with RDMA; check measured results for your exact setup.
• Prefer one machine with enough memory over two when possible: fewer parts, less latency, less setup.
• Pair clusters with MoE models, which read far less data per token.
• Use the AI Hardware & Model Index to check whether one machine already fits your target before planning a cluster.