Mixture of Experts

Many of the largest open-weight models released in the last two years are mixtures of experts. If you plan to run or pay for AI, this design changes the usual rule that a bigger model is always slower.

The basic idea

A transformer model (the architecture behind today's language models) is a stack of layers. Each layer has an attention part, which lets each token look at earlier tokens, and a feed-forward part, a block of weights that transforms each token's representation. In a dense model, every token passes through the whole feed-forward block of every layer.

A mixture of experts (MoE) replaces that single feed-forward block with many smaller ones called experts, plus a small router (sometimes called a gate). For each token, at each layer, the router scores the experts and sends the token to only the top few. Their outputs are blended and passed on.

An analogy: a large hospital employs hundreds of specialists, but any one patient sees only two or three of them. The hospital still needs room for every specialist, yet each visit takes only the time of the few who are consulted.

A simplification worth flagging: the experts are not neat subject specialists like "the French expert" or "the maths expert". They are learned during training and tend to divide the work in ways that are hard for humans to label.

Total versus active parameters

MoE models come with two numbers:

• Total parameters: every weight in the model, including all experts. All of these must be stored in memory.
• Active parameters: the weights actually used to produce one token: the shared parts (attention, embeddings) plus the chosen experts.

Real examples, as published by their makers:

• Mixtral 8x7B: about 46.7B total, about 12.9B active; 8 experts per layer, 2 chosen per token.
• DeepSeek-V2: 236B total, 21B active.
• gpt-oss-120b: 117B total, 5.1B active.
• Qwen3-Next-80B-A3B: 80B total, about 3B active, with 512 experts per MoE layer, 10 chosen per token plus 1 always-on shared expert.

The "A3B" style of name, meaning "3B active", has become a common way to advertise this.

Why MoE generates faster

Writing each new token (the decode phase) is mostly limited by memory bandwidth: how many gigabytes per second the hardware can read from memory. For every token, the model must read the weights it uses. A dense model reads all of them; an MoE model reads only the active ones. Less data read per token means more tokens per second.

The catch is that you do not know in advance which experts the next token will need, so all of them must sit in fast memory, ready. MoE saves time, not space.

Worked example: 120B MoE versus 70B dense

Take a machine with 128 GB of unified memory (memory shared by the CPU and GPU, as on Apple silicon Macs and some PC chips) and 500 GB/s of memory bandwidth. Compare a 120B-total, 5B-active MoE model with a 70B dense model, both quantized to about 4.5 bits per weight. We use this site's AI Index formulas.

Step 1: usable memory. macOS lets the GPU use about 75% of unified memory by default, so 128 × 0.75 = 96 GB. A model "fits" if it needs no more than 85% of that: 81.6 GB.

Step 2: memory needed (file size ≈ parameters × bits ÷ 8, plus 5%, then KV cache and 1.5 GB overhead):

• MoE: 120 × 4.5 ÷ 8 = 67.5 GB, plus 5% ≈ 71 GB. Add about 3 GB of KV cache for a moderate context and 1.5 GB of overhead: about 75.5 GB. It fits, just.
• Dense: 70 × 4.5 ÷ 8 ≈ 39.4 GB, plus 5% ≈ 41 GB. Add 3 GB and 1.5 GB: about 46 GB. Fits easily.

Step 3: generation speed. The site's estimate is tok/s ≈ 1 ÷ (bytes per token ÷ (0.55 × bandwidth) + 1.5 ms), where bytes per token ≈ active parameters × bits ÷ 8, and 0.55 reflects that real hardware reaches only about half its rated bandwidth.

• MoE: 5 × 4.5 ÷ 8 ≈ 2.8 GB per token. 2.8 ÷ 275 GB/s ≈ 10.2 ms, plus 1.5 ms ≈ 11.7 ms per token: about 85 tokens per second.
• Dense: 39.4 GB per token. 39.4 ÷ 275 ≈ 143 ms, plus 1.5 ms ≈ 145 ms: about 7 tokens per second.

The MoE model needs more memory but writes roughly twelve times faster. Real numbers will differ (routing adds some overhead, and the KV cache depends on the architecture), but the direction holds. The same pair on a 24 GB graphics card tells the other side of the story: neither fits, and the MoE model is the bigger problem.

Trade-offs

• Quality per total size. An MoE model generally does not match a dense model with the same total parameter count; it trades some quality per stored parameter for much cheaper computation. It often matches or beats dense models with similar active-parameter cost.
• Prompt processing. Reading your prompt (the prefill phase) is limited mainly by compute, at about 2 × active parameters in arithmetic operations per token, so MoE helps here too.
• Offloading. Because only a few experts run per token, some runtimes keep rarely-needed expert weights in ordinary system RAM. That works, but speed drops toward system-RAM bandwidth.
• Serving at scale. Providers like MoE because cost per token tracks active parameters, which is part of why large hosted models are often MoE.

Going deeper

The router, formally

Let $x \in \mathbb{R}^d$ be a token's hidden vector entering an MoE layer with $E$ experts $f_1, \dots, f_E$ (each a small feed-forward network) and a router matrix $W_r \in \mathbb{R}^{d \times E}$. The router computes one score per expert and turns them into probabilities with a softmax:

Let $\mathcal{T}$ be the indices of the $k$ largest $p_i$ (top-$k$). The layer output is a gated sum of only those experts:

Renormalising over the chosen experts, as Mixtral does, is the same as taking a softmax over just the top-$k$ logits. Some designs skip renormalisation or add a shared expert that every token uses. Because the top-$k$ choice is not differentiable, the task loss reaches the router only through the gate values $g_i$ of the chosen experts (auxiliary losses, below, add a separate signal).

Worked example. With $E = 4$, $k = 2$ and logits $h = (2, 1, 0, -1)$: the softmax gives $p \approx (0.644, 0.237, 0.087, 0.032)$. Experts 1 and 2 are chosen, and their gates are $0.644 / 0.881 \approx 0.731$ and $0.237 / 0.881 \approx 0.269$, equivalently $1/(1 + e^{-1})$ and $1/(1 + e^{1})$.

Load balancing

Left alone, a router tends to collapse: a few experts get most tokens, train faster, get chosen even more, and the rest are wasted. The Switch Transformer paper added an auxiliary loss. For a batch of $T$ tokens, let $f_i$ be the fraction of routing assignments sent to expert $i$ and $P_i$ the mean router probability for expert $i$:

$f_i$ is not differentiable (it counts discrete choices), but $P_i$ is, so the gradient lowers the probability of overloaded experts. Under perfectly uniform routing $f_i = P_i = 1/E$, giving $E \cdot E \cdot (1/E^2) = 1$, so the unscaled loss is 1 when balanced and grows with imbalance. Switch used $\alpha = 10^{-2}$. Example with $E = 4$, $f = (0.4, 0.3, 0.2, 0.1)$ and $P = (0.35, 0.3, 0.2, 0.15)$: $\sum f_i P_i = 0.14 + 0.09 + 0.04 + 0.015 = 0.285$, so the unscaled loss is $4 \times 0.285 = 1.14$.

Training systems also set an expert capacity, the most tokens an expert may take per batch: $\text{capacity} = \lceil \text{CF} \cdot T k / E \rceil$, where the capacity factor CF is typically a little above 1. Tokens beyond capacity are dropped from that expert and pass through the layer's residual connection unchanged. Many recent inference engines avoid dropping altogether.

Worked example: where Mixtral's 46.7B and 12.9B come from

Mixtral 8x7B uses the same layer shapes as a 7B-class Llama model: $d = 4096$, 32 layers, 8 KV heads of size 128, SwiGLU experts with $d_{ff} = 14336$, a 32,000-token vocabulary, 8 experts and top-2 routing.

• One expert: $3 \cdot 4096 \cdot 14336 = 176{,}160{,}768$ weights.
• All experts: $32 \times 8 \times 176.16\text{M} \approx 45.10\text{B}$.
• Attention: $32 \times 41.94\text{M} \approx 1.34\text{B}$; embeddings and head $2 \times 32000 \times 4096 \approx 0.26\text{B}$; routers about 1M.
• Total ≈ 46.70B. Active: replace 8 experts with 2, so $32 \times 2 \times 176.16\text{M} \approx 11.27\text{B}$, plus the same 1.60B of shared parts: ≈ 12.88B.

Both match the published figures, and the arithmetic shows the name "8x7B" is not $8 \times 7 = 56$B: only the feed-forward blocks are replicated.

Why batching erodes the advantage

The bandwidth saving assumes one token at a time. A server decoding a batch of $B$ tokens at once must read every expert that any of them selects. If routing were uniform and independent, the expected number of distinct experts touched per layer is

For Mixtral ($E = 8$, $k = 2$): 2 experts at $B = 1$, about 5.5 at $B = 4$ and about 7.9 at $B = 16$. By a batch of 16, nearly all expert weights are read every step, so per-step memory traffic approaches that of a dense model of the same total size; the saving that remains is in compute, which still follows the active count. That is why MoE is ideal for single-user local decoding and still cheap per token at scale, but for different reasons.

Code: a top-k router with the balancing loss

import numpy as np

rng = np.random.default_rng(0)
T, d, E, k = 1024, 64, 8, 2              # tokens, model width, experts, top-k
x = rng.standard_normal((T, d))
W_r = rng.standard_normal((d, E)) * 0.1   # router weights
W_e = rng.standard_normal((E, d, d)) / np.sqrt(d)  # toy experts: one matrix each

logits = x @ W_r                                    # (T, E)
p = np.exp(logits - logits.max(axis=1, keepdims=True))
p /= p.sum(axis=1, keepdims=True)                   # router softmax
topk = np.argsort(-p, axis=1)[:, :k]                # chosen experts per token
gates = np.take_along_axis(p, topk, axis=1)
gates /= gates.sum(axis=1, keepdims=True)           # renormalise over top-k

y = np.zeros_like(x)
for slot in range(k):                                # weighted sum of k experts
    e = topk[:, slot]
    y += gates[:, slot:slot + 1] * np.einsum("td,tdo->to", x, W_e[e])

f = np.bincount(topk.ravel(), minlength=E) / (T * k)  # share of assignments
P = p.mean(axis=0)                                     # mean router probability
aux = E * np.sum(f * P)                                # equals 1.0 when uniform
print("load per expert:", np.round(f, 3))
print("aux loss (unscaled):", round(float(aux), 4))

The loop runs each token through its chosen experts and weights the results by the renormalised gates; the last lines compute the Switch-style loss. Real implementations group tokens by expert so each expert does one large matrix multiply, and, when experts are spread over GPUs (expert parallelism), exchange tokens with an all-to-all communication step, which is often the bottleneck in serving.

Compute

Prefill and training compute follow the active count: about $2 N_{\text{active}}$ FLOPs per token forward. Mixtral therefore computes like a roughly 13B dense model while storing about 47B parameters.

Common pitfalls

• Sizing hardware from the active count. Every expert must be resident (or streamed at a heavy speed cost).
• Reading "8x7B" as 56B. Attention and embeddings are shared; only the feed-forward blocks are copied.
• Assuming experts are topic specialists. Routing patterns are learned and mostly not human-interpretable.
• Ignoring batch effects. At large batch sizes, different tokens hit different experts, so a server reads most experts every step and the bandwidth advantage of sparsity shrinks.
• Fine-tuning without balancing. Dropping the auxiliary loss (or its equivalent) during fine-tuning can collapse routing.

In practice

• When you see an MoE model, plan memory on the total parameter count and expect speed in line with the active count.
• MoE models suit machines with a lot of memory but modest bandwidth, such as large unified-memory computers.
• On a small GPU, a dense model that fits entirely in video memory may beat a larger MoE that has to spill into system RAM.
• Check the model card for both numbers; names like "A3B" or "A22B" tell you the active count.