Parameters and Model Size

Model names often carry a number such as 8B, 70B or 1T. That number is the single best predictor of how much memory a model needs and what hardware can run it, so it is worth knowing exactly what it counts.

What a parameter is

A parameter (also called a weight) is one adjustable number inside a neural network. During training, each of these numbers is nudged, billions of times, so that the model's predictions of the next token get a little better. When training ends, the parameters are frozen, and together they are the model. Running the model means multiplying your input through all those stored numbers.

An analogy: think of a huge mixing desk with billions of tiny knobs. Training is the long process of setting every knob. The finished model is just the knob positions written down. A bigger desk can capture subtler patterns, but only if someone takes the time (and has the material) to set the knobs well.

What 7B, 70B and 1T mean

The letter is a count, not a quality score:

• B means billion, so a 7B model has about seven billion parameters.
• T means trillion, so a 1T model has about a thousand billion.
• M (million) shows up on small models such as text classifiers or on-device helpers.

The numbers are often rounded. A model called "8B" might have 8.03 billion parameters.

Bytes per parameter

Each parameter is stored as a number with a fixed precision, measured in bits (a bit is a single 0 or 1; eight bits make a byte). Fewer bits means a smaller file and a little less accuracy per number. Reducing the bits after training is called quantization, covered in its own lesson.

Why "~4.5–4.8" rather than exactly 4? Real 4-bit files also store small scaling numbers for each block of weights, and some sensitive parts of the model are usually kept at higher precision, so the average comes out a bit above 4.

The memory math

The rule this site uses, in its AI Index methodology, is:

Memory needed ≈ file size + KV cache + about 1.5 GB of runtime overhead, where file size ≈ parameters × bits per weight ÷ 8, plus about 5%.

The KV cache is working memory the model builds up as it reads your conversation; it grows with context length and has its own lesson. For short chats it is usually a few GB or less.

Worked example 1, an 8B model at about 4.8 bits:

• File: 8 billion × 4.8 ÷ 8 = 4.8 GB, plus 5% ≈ 5 GB.
• KV cache for an 8,000-token conversation on a typical 8B design: roughly 1 GB at 16-bit.
• Overhead: 1.5 GB.
• Total: about 7.5 GB. On a GPU with 12 GB of video memory, the site counts a model as fitting when it needs no more than 85% of usable memory (10.2 GB here), so this fits comfortably.

Worked example 2, a 70B model:

• At 16-bit: 70 billion × 2 bytes ≈ 140 GB for the weights alone. That is beyond any single consumer graphics card.
• At about 4.5 bits: 70 × 4.5 ÷ 8 ≈ 39 GB, plus 5% ≈ 41 GB. Add a few GB of KV cache and 1.5 GB of overhead and you need roughly 45 GB. A 24 GB card cannot hold it. Two 24 GB cards (48 GB) or a 64 GB Mac (about 48 GB usable by default) come close, but 45 GB is over this site's 85% "fits" threshold of about 41 GB, so they would need a smaller quant or a short context. A machine with 80 GB or more of usable memory runs it comfortably.

Worked example 3, a 1T model at about 4.5 bits: 1,000 × 4.5 ÷ 8 ≈ 560 GB, plus 5% ≈ 590 GB before the KV cache. That is server or multi-machine territory.

These are estimates. Actual file sizes vary by format, and some runtimes need more overhead, but the method gets you within a few GB.

Bigger is not automatically better

Parameter count sets the model's capacity, not its quality. Two things matter just as much:

• Training data. In 2022 DeepMind's Chinchilla study showed that a 70B model trained on about 1.4 trillion tokens of text outperformed its own 280B predecessor, Gopher, which used the same training compute but far less data. Since then, labs have often trained smaller models on far more data than that study's compute-optimal ratio; Meta reported training even its 8B Llama 3 model on about 15 trillion tokens, which makes small models run cheaply while performing well.
• Training method. Data quality, filtering, fine-tuning and reinforcement learning after pre-training often matter more than a few extra billion parameters. A well-trained newer 8B model can beat an older 70B model on many tasks.

So compare models on results for your task, not on the size label.

Dense versus sparse

In a dense model, every parameter is used to produce every token. A 70B dense model does about 70 billion parameters' worth of work for each word it writes.

In a sparse model, only part of the network is used for each token. The most common sparse design today is the mixture of experts (MoE), where a small routing network picks a few specialist sub-networks per token. For these models you will see two numbers: total parameters (everything that must be stored) and active parameters (what is actually used per token). Memory follows the total; speed follows the active count. The next lesson explains how.

Going deeper

Counting parameters from a config file

You do not have to trust the size label: you can count parameters exactly from a model's configuration. Take a Llama-style decoder (the layout used by most open models) with model width $d$ (the length of each token's hidden vector), $L$ layers, $n_h$ query heads and $n_{kv}$ key/value heads of size $d_h$, feed-forward width $d_{ff}$ and vocabulary size $V$. Each layer holds:

• Attention: the query projection $W_Q \in \mathbb{R}^{d \times n_h d_h}$, key and value projections $W_K, W_V \in \mathbb{R}^{d \times n_{kv} d_h}$ and the output projection $W_O \in \mathbb{R}^{n_h d_h \times d}$, giving $2 d\, n_h d_h + 2 d\, n_{kv} d_h$ weights.
• Feed-forward (SwiGLU, with a gate, an up and a down matrix): $3\, d\, d_{ff}$ weights.
• Normalisation: two RMSNorm weight vectors, $2d$.

Add the input embedding table and the output head ($V d$ each, unless they are tied, meaning shared) and one final norm:

Worked example: Llama 3 8B. Its published configuration has $d = 4096$, $L = 32$, $n_h = 32$, $n_{kv} = 8$, $d_h = 128$, $d_{ff} = 14336$, $V = 128256$, untied embeddings.

• Attention per layer: $2 \cdot 4096 \cdot 4096 + 2 \cdot 4096 \cdot 8 \cdot 128 = 33{,}554{,}432 + 8{,}388{,}608 = 41{,}943{,}040$.
• Feed-forward per layer: $3 \cdot 4096 \cdot 14336 = 176{,}160{,}768$.
• Per layer with norms: $218{,}112{,}000$; times 32 layers: $6{,}979{,}584{,}000$.
• Embedding plus output head: $2 \cdot 128256 \cdot 4096 = 1{,}050{,}673{,}152$; final norm 4,096.
• Total: 8,030,261,248, which is the "8.03 billion" behind the 8B label.

Two lessons fall out of the arithmetic. First, about 81% of each layer is the feed-forward block, which is why mixture-of-experts designs target it. Second, the vocabulary costs about 13% of this model; in very small models embeddings can be a quarter or more of all parameters, so "1B" models differ a lot in how much of their budget does actual computation.

Compute per token

A matrix–vector product with an $m \times n$ weight matrix costs $mn$ multiplications and $mn$ additions, about $2mn$ floating-point operations (FLOPs). Summed over the network, a forward pass costs roughly

ignoring the attention-score term, which grows with context (see the KV-cache lesson). Training also needs a backward pass, about twice the forward cost, giving the widely used approximation $C_{\text{train}} \approx 6ND$, where $D$ is the number of training tokens. For Llama 3 8B on about 15 trillion tokens: $6 \times 8 \times 10^9 \times 15 \times 10^{12} \approx 7.2 \times 10^{23}$ FLOPs.

The Chinchilla analysis found that, for a fixed compute budget, parameters and training tokens should grow in roughly equal proportion, which works out to about 20 tokens per parameter. By that yardstick an 8B model would be "compute-optimal" at about 160 billion tokens; training it on 15 trillion (about 94 times more) spends extra training compute to get a smaller model that is cheaper at inference, the trade-off modern labs usually choose.

Training memory is a different number

The inference rule of thumb (2 bytes per parameter at 16-bit) badly underestimates training. A common accounting for mixed-precision training with the Adam optimiser is about 16 bytes per parameter: 2 for 16-bit weights, 2 for 16-bit gradients, and 12 for 32-bit optimiser state (a master copy of the weights plus Adam's two moment estimates). An 8B model therefore needs about 128 GB before activations, which is why full fine-tuning needs several data-centre GPUs while parameter-efficient methods (such as LoRA, which trains small added matrices) fit on one.

Code: count and size a model

def count_params(d, n_layers, n_heads, n_kv_heads, head_dim, d_ff, vocab,
                 tied_embeddings=False):
    """Parameter count for a Llama-style decoder (GQA attention, SwiGLU FFN,
    RMSNorm without biases)."""
    attn = d * n_heads * head_dim            # W_Q
    attn += 2 * d * n_kv_heads * head_dim    # W_K and W_V
    attn += n_heads * head_dim * d           # W_O
    ffn = 3 * d * d_ff                       # gate, up and down projections
    norms = 2 * d                            # two RMSNorm weight vectors
    per_layer = attn + ffn + norms
    embed = vocab * d * (1 if tied_embeddings else 2)  # input + output head
    return n_layers * per_layer + embed + d            # + final norm

def memory_gb(n_params, bits, kv_gb=0.0, overhead_gb=1.5):
    file_gb = n_params * bits / 8 / 1e9 * 1.05
    return file_gb, file_gb + kv_gb + overhead_gb

n = count_params(d=4096, n_layers=32, n_heads=32, n_kv_heads=8,
                 head_dim=128, d_ff=14336, vocab=128256)
print(f"{n:,} parameters")                       # 8,030,261,248
for bits in (16, 8, 4.8):
    f, total = memory_gb(n, bits, kv_gb=1.0)
    print(f"{bits:>4} bits: file {f:5.1f} GB, total {total:5.1f} GB")

The function implements the formula above; memory_gb applies this site's methodology (file size plus 5%, plus KV cache, plus 1.5 GB overhead). Change the arguments to any config.json to size a model before downloading it.

Common pitfalls

• Confusing total and active parameters. For MoE models, memory follows the total and speed follows the active count; quoting only one misleads.
• Forgetting the KV cache and overhead. A file that "just fits" will fail as soon as the context grows.
• Using decimal and binary units interchangeably. 1 GB is $10^9$ bytes and 1 GiB is $2^{30}$ bytes (about 7% more); tools differ in which they report.
• Applying inference memory rules to training. Full training with Adam needs roughly eight times the 16-bit weight memory, plus activations.
• Reading size as quality. A size label predicts hardware needs, not results on your task.

In practice

• Multiply parameters by bits per weight and divide by 8 to estimate the file size in GB, then add about 5%, the KV cache and 1.5 GB to estimate total memory.
• Remember the anchors: 16-bit is about 2 GB per billion parameters, 8-bit about 1 GB, and a typical 4-bit file about 0.6 GB.
• Treat the size label as a hardware requirement, not a quality ranking; check benchmarks and your own tests.
• For a sparse (MoE) model, look for both the total and the active parameter count before deciding what hardware you need.