Datacenter AI Hardware: GPUs, Accelerators, Interconnects and Rack-Scale Systems

Every frontier model is trained and served on datacenter accelerators: GPUs and custom chips packed into liquid-cooled racks and wired together with networks that move terabytes per second. If you build, buy or operate AI systems, you will constantly face questions like "why is my GPU only 30% busy?", "how many chips do we need for this training run?" and "why is decoding slow when the spec sheet promises petaflops?". This lesson gives you the mental model and the arithmetic to answer them.

Intuition: a kitchen with a very slow pantry

Picture a restaurant kitchen. The chefs are the arithmetic units: they can chop extraordinarily fast. Each chef has a tiny cutting board (registers) and the station shares a countertop (on-chip shared memory and caches). Ingredients live in a walk-in pantry down the hall (HBM, the high-bandwidth memory stacked next to the chip). Other kitchens in the building are reachable by a dumbwaiter (NVLink), and kitchens in other buildings by delivery trucks (the datacenter network).

The chefs are so fast that they are almost always waiting for ingredients. A dish that needs a lot of chopping per ingredient (a large matrix multiplication) keeps chefs busy. A dish that needs one chop per ingredient fetched (generating one token for one user) leaves them idle while runners shuttle between pantry and counter. Almost all of AI-hardware engineering is about keeping the chefs fed: bigger, faster pantries, more counter space, smarter recipes that reuse ingredients, and faster dumbwaiters between kitchens.

How a datacenter GPU is organized

Compute: SMs, warps and tensor cores

An NVIDIA GPU is built from many streaming multiprocessors (SMs); AMD calls the equivalent unit a compute unit (CU). An H100 SXM has 132 SMs. Each SM executes groups of 32 threads called warps in lockstep (single instruction, multiple threads). Inside each SM are general-purpose floating-point/integer units ("CUDA cores") and tensor cores, specialized units that perform small matrix multiply-accumulate operations, such as multiplying tiles of BF16 values and accumulating in FP32, in a single instruction. Nearly all of a modern accelerator's headline FLOPS come from tensor cores; the general-purpose units handle element-wise work like activations and normalization at a small fraction of that rate.

Tensor-core throughput roughly doubles each time the number format halves: BF16/FP16 (16-bit), FP8 (8-bit) and, on Blackwell-generation and newer chips, FP4 formats (NVIDIA's NVFP4 and the OCP MXFP4 microscaling format). Two cautions when reading spec sheets:

• Sparsity: many vendor peak numbers assume 2:4 structured sparsity (two of every four weights are zero). Dense throughput is half. Real models are rarely run with 2:4 sparsity, so use the dense figure.
• Peak vs achieved: the peak assumes perfect tiling and no stalls. Well-tuned large-model training typically achieves well under the peak; the ratio of achieved to peak is called model FLOPs utilization (MFU).

Memory hierarchy

HBM (high-bandwidth memory) is DRAM stacked vertically and placed on the same package as the compute die, connected by a very wide bus. It is what gives an H100 SXM about 3.35 TB/s and Blackwell-class parts roughly 8 TB/s. Capacity decides what fits; bandwidth decides how fast you can stream it. Kernels such as FlashAttention are fast precisely because they tile work into shared memory and avoid round trips to HBM.

The roofline model

The roofline model (Williams, Waterman and Patterson, 2009) predicts the best throughput a kernel can reach on a given machine from two hardware numbers and one property of the workload.

Define:

• $P_{\text{peak}}$: peak compute throughput (FLOP/s) for the number format used.
• $B$: memory bandwidth (bytes/s) of the level the data comes from, usually HBM.
• $W$: total floating-point operations the kernel performs (FLOP).
• $Q$: bytes moved between that memory level and the chip.
• $I = W / Q$: arithmetic intensity (FLOP per byte).

Then the attainable throughput is

The two lines meet at the ridge point $I^{*} = P_{\text{peak}} / B$. A kernel with $I < I^{*}$ is memory-bound (more FLOPS would not help; more bandwidth would). A kernel with $I > I^{*}$ is compute-bound.

Worked example 1: the H100 ridge point

For an H100 SXM in dense BF16, take $P_{\text{peak}} \approx 989$ TFLOP/s and $B \approx 3.35$ TB/s:

A kernel must do about 295 floating-point operations for every byte it reads from HBM to keep the tensor cores saturated. That is a high bar, and it rises each generation because compute grows faster than bandwidth.

Worked example 2: a large matrix multiplication

Multiply $A \in \mathbb{R}^{M\times K}$ by $B \in \mathbb{R}^{K\times N}$ in BF16 (2 bytes per element) with $M=N=K=4096$.

• FLOPs: each of the $MN$ outputs needs $K$ multiply-adds, i.e. $W = 2MNK = 2 \times 4096^3 = 1.374 \times 10^{11}$ FLOP.
• Minimum bytes: read $A$ and $B$ once and write $C$ once: $Q = 2(MK + KN + MN) = 2 \times 3 \times 4096^2 = 1.007 \times 10^{8}$ bytes.
• Intensity: $I = 1.374\times10^{11} / 1.007\times10^{8} \approx 1365$ FLOP/byte.

Since $1365 > 295$, the GEMM is compute-bound. Its lower-bound time is$1.374\times10^{11} / 989\times10^{12} \approx 0.14$ ms. This is why training, which is dominated by large GEMMs over big batches, can reach high utilization.

Worked example 3: LLM decoding at batch size 1

Generating one token requires multiplying the activation vector by every weight matrix. For a $d \times d$ weight matrix in BF16 and a single token: $W = 2d^2$ FLOP, and the weights alone are $Q \approx 2d^2$ bytes, so $I \approx 1$ FLOP/byte. Attainable throughput is $1 \times 3.35$ TB/s $= 3.35$ TFLOP/s, about 0.3% of peak.

If the server batches $b$ sequences, the same weights are read once and used $b$ times: $W = 2bd^2$, $Q \approx 2d^2$, so $I \approx b$. You need roughly $b \approx 295$ concurrent tokens per weight read to reach the ridge on an H100. (This simplification ignores the KV cache, which is read per sequence and does not benefit from batching; the next lesson treats it.)

At whole-model level, an 8-billion-parameter model in 16-bit has about 16 GB of weights. Streaming them once per token at the full 3.35 TB/s takes $16 / 3350 \approx 4.8$ ms, an upper bound of about 209 tokens/s for a single stream. Real kernels do not reach full bandwidth; the site's AI Index method uses 55% effective bandwidth plus 1.5 ms overhead:

Either way the conclusion holds: single-stream decoding is limited by memory bandwidth, not FLOPS.

A roofline calculator in Python

def roofline(flops, bytes_moved, peak_flops, bandwidth):
    """Return (intensity, attainable FLOP/s, bound) for one kernel."""
    intensity = flops / bytes_moved
    attainable = min(peak_flops, intensity * bandwidth)
    bound = "compute" if intensity >= peak_flops / bandwidth else "memory"
    return intensity, attainable, bound

def gemm(m, n, k, bytes_per_el=2):
    flops = 2 * m * n * k
    bytes_moved = bytes_per_el * (m * k + k * n + m * n)
    return flops, bytes_moved

H100 = dict(peak_flops=989e12, bandwidth=3.35e12)   # dense BF16, HBM3

for name, (m, n, k) in {
    "training GEMM": (4096, 4096, 4096),
    "decode, batch 1": (1, 4096, 4096),
    "decode, batch 64": (64, 4096, 4096),
    "decode, batch 512": (512, 4096, 4096),
}.items():
    f, q = gemm(m, n, k)
    i, att, bound = roofline(f, q, **H100)
    print(f"{name:18s} I={i:8.1f} FLOP/B  attainable={att/1e12:7.1f} TFLOP/s  ({bound}-bound)")

Here $M$ plays the role of the number of tokens processed together. The output shows intensity rising almost linearly with batch size (about 1, 62 and 410 FLOP/byte) until the decode GEMM crosses the ridge at roughly batch 350 (slightly above 295 because activations also cost bytes) and becomes compute-bound.

Datacenter accelerators as of November 2026

Specifications below are vendor-published figures; check current documentation before buying because configurations and availability change quickly.

Three patterns matter more than any single number:

1. Memory per chip keeps growing (80 GB to 288 GB to 400+ GB), because inference of large models and long contexts is capacity-bound.
2. Low-precision compute grows fastest (FP8, then FP4), so the ridge point rises and memory-bound work gets relatively worse.
3. The unit of design is now the rack or pod, not the chip.

Custom chips (TPU, Trainium) trade ecosystem breadth for cost and integration with one cloud. GPUs offer the broadest software support (CUDA on NVIDIA; ROCm on AMD, which has narrowed but not closed the gap).

Interconnects: scale-up and scale-out

Large models do not fit on one chip, so chips must exchange data constantly. There are two tiers:

• Scale-up links connect a small group of chips with very high bandwidth and low latency, often with load/store memory semantics: NVIDIA NVLink with NVLink Switch (900 GB/s per GPU on Hopper, 1.8 TB/s on Blackwell), AMD Infinity Fabric (and the multi-vendor UALink effort), Google TPU ICI (inter-chip interconnect, 1.2 TB/s bidirectional per Ironwood chip), and AWS NeuronLink.
• Scale-out networks connect thousands of nodes: InfiniBand (NDR 400 Gb/s and XDR 800 Gb/s per port), and Ethernet with RoCE v2 (RDMA over Converged Ethernet), which hyperscalers increasingly prefer. On GB300 NVL72, NVIDIA specifies 800 Gb/s of network connectivity per GPU via ConnectX-8.

Note the units: NVLink is quoted in gigabytes per second, network ports in gigabits. 400 Gb/s is 50 GB/s, roughly 18 times less than Hopper NVLink.

Worked example 4: how long does a gradient all-reduce take?

In data-parallel training, every step ends by averaging gradients across $N$ workers. A ring all-reduce makes each worker send (and receive) about

where $S$ is the gradient size. For an 8-billion-parameter model with BF16 gradients, $S = 16$ GB. With $N = 8$:

• Over Hopper NVLink at about 450 GB/s per direction: $28 / 450 \approx 0.062$ s.
• Over a single 400 Gb/s (50 GB/s) network port: $28 / 50 = 0.56$ s.

A tenfold gap. This simplified estimate ignores latency terms and overlap with computation, but it explains the standard layout: put the most communication-heavy parallelism (tensor parallelism, which all-reduces inside every layer) inside the NVLink domain, and the lighter-weight parallelism (data and pipeline parallelism) across the network.

Rack-scale systems

Since Blackwell, NVIDIA's flagship product is a rack. The GB300 NVL72 couples 72 Blackwell Ultra GPUs and 36 Grace CPUs in a single NVLink domain. NVIDIA's published rack totals include up to 20 TB of HBM with up to 576 TB/s of aggregate HBM bandwidth, 130 TB/s of total NVLink bandwidth, and 1,080 PFLOPS of dense FP4 (1,440 with sparsity). Its predecessor, GB200 NVL72, used the same 72-GPU layout with Blackwell GPUs. Comparable rack- or pod-scale designs: AMD's Helios (72 MI450-series GPUs), Google TPU pods (Ironwood scales to 9,216 chips over ICI and optical circuit switches), and AWS Trn3 UltraServers (up to 144 Trainium3 chips).

Why racks? A 72-GPU NVLink domain lets an entire large mixture-of-experts model run with tensor and expert parallelism over fast links instead of the network. The cost is physical: these racks need direct liquid cooling and power densities that older datacenters cannot supply, which is why facility power has become a gating factor (see AI energy and environment).

What limits training vs inference

Training

Training compute is approximately $C \approx 6ND$ FLOPs for a dense transformer with $N$ parameters trained on $D$ tokens (2 for the forward pass, 4 for the backward).

Worked example 5. A 70B-parameter model on 15 trillion tokens:

On 1,000 H100s at 989 TFLOP/s peak with 40% MFU, the cluster sustains $1000 \times 989\times10^{12} \times 0.4 = 3.96\times10^{17}$ FLOP/s, so the run takes $6.3\times10^{24} / 3.96\times10^{17} \approx 1.59\times10^{7}$ s, about 184 days. With 10,000 GPUs it is about 18 days, if utilization holds, which is the hard part.

Memory is the other wall. Mixed-precision training with Adam typically holds about 16 bytes per parameter (2 for BF16 weights, 2 for gradients, 12 for FP32 master weights and two Adam moments), before activations. For 70B parameters that is $1.12$ TB, which must be sharded (ZeRO/FSDP) across at least 14 80-GB GPUs, and in practice many more.

So training is limited by: total compute (chip count times MFU), memory capacity for optimizer state and activations, communication bandwidth for gradient and tensor-parallel collectives, and reliability: with tens of thousands of chips, hardware failures happen frequently, so checkpointing and fast restart determine effective throughput.

Inference

• Prefill (processing the prompt) is a batch of GEMMs over all prompt tokens, about $2N$ FLOPs per token: compute-bound.
• Decode (generating tokens one at a time) is memory-bandwidth-bound, as worked example 3 showed, and limited by memory capacity for weights plus the KV cache of every active sequence.

That is why inference-oriented chips and racks emphasize HBM capacity and bandwidth, and why serving systems batch aggressively.

Common pitfalls

• Quoting sparse or FP4 peak numbers when the workload is dense BF16. Divide sparse figures by two and use the precision you will actually run.
• Ignoring the ridge point. Buying more FLOPS does nothing for a memory-bound decode workload; bandwidth and batching do.
• Bits vs bytes. Confusing 400 Gb/s with 400 GB/s inflates network capacity eightfold.
• Assuming linear scaling. Doubling GPUs rarely halves time: communication, stragglers and failures grow with scale.
• Forgetting optimizer state. Parameters are only one-eighth of the 16 bytes per parameter needed for Adam training.
• Ignoring the facility. A rack that needs liquid cooling and 100+ kW cannot be dropped into a typical enterprise server room.

In practice

• Start every hardware question with a roofline sketch: compute $W$, $Q$, and $I$, then compare with the ridge point of the candidate chip.
• Measure MFU (training) and achieved bandwidth (decode) with profilers such as Nsight Systems, the PyTorch profiler or vendor equivalents; tune batch size, kernel fusion and parallelism layout before buying hardware.
• Match parallelism to topology: tensor and expert parallelism inside the scale-up domain, pipeline and data parallelism across it.
• For inference fleets, size by memory (weights plus KV cache at target concurrency) first, then by bandwidth for latency, then by compute for prefill-heavy traffic.
• Treat accelerator choice as a software decision as much as a hardware one: framework support, kernels, and your team's experience often outweigh a 20% spec advantage.
• Use the AI Index to compare memory and bandwidth across hardware with consistent assumptions.