KV Cache and Context Cost

A model that fits easily with a short chat can run out of memory when you paste in a long document. The reason is the KV cache, and knowing how to estimate it tells you how much context your hardware can really handle.

What the KV cache is

Language models write one token at a time. Through the attention mechanism, each new token looks back at every earlier token in the context. To do that, each layer turns every token into two vectors (lists of numbers): a key, which describes what the token offers, and a value, the information it carries.

Those keys and values do not change once computed, so instead of recalculating them for the whole conversation at every step, the model stores them. That store is the key-value (KV) cache. It makes generation fast, but it grows with every token in the context.

An analogy: a reader who keeps a sticky note for every sentence so far, instead of rereading the whole book before each new sentence. The notes save enormous time, but a long book means a big pile of notes.

The formula

For a standard transformer:

KV cache (bytes) = 2 × layers × KV heads × head dimension × bytes per value × tokens

• 2: one key and one value.
• Layers: how many transformer layers the model has.
• KV heads: attention is split into parallel "heads"; this is how many distinct key/value sets each layer keeps.
• Head dimension: the length of each key or value vector, often 128.
• Bytes per value: 2 at 16-bit, 1 at 8-bit.
• Tokens: the length of the context, including the reply being written.

These numbers are in a model's configuration file, usually named config.json.

Worked example: a 70B model at long context

Llama 3 70B has 80 layers, 8 KV heads and a head dimension of 128 (from its published configuration). At 16-bit:

• Per token: 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, about 0.33 MB.
• At 8,000 tokens: about 2.6 GB.
• At 32,768 tokens: about 10.7 GB.
• At 131,072 tokens (a "128K" context): about 43 GB.

At full 128K context, the KV cache alone (about 43 GB) is larger than the 4-bit weights (about 41 GB). Using this site's method, total memory is weights + KV cache + about 1.5 GB of overhead, so roughly 86 GB, nearly double the roughly 45 GB the same model needs for an 8,000-token chat. That is why 128K context can need tens of gigabytes on its own.

Why it is not even worse: grouped-query attention

Llama 3 70B has 64 attention heads but only 8 KV heads. That is grouped-query attention (GQA): several query heads share one set of keys and values. If every head kept its own (the original "multi-head attention" design), the cache would be 8 times larger, about 340 GB at 128K. GQA is now standard in most open models, with only a small effect on quality.

Other ways designers shrink the cache

• Sliding-window attention. Some layers only look at the most recent tokens, say the last 1,024, so their cache stops growing. Google's Gemma 3 interleaves five such local layers for every one full "global" layer; only the global layers keep the whole context.
• Multi-head latent attention (MLA). Introduced by DeepSeek, MLA compresses keys and values into a smaller shared "latent" vector and reconstructs them when needed. DeepSeek reported that DeepSeek-V2 cut KV cache size by 93.3% compared with its earlier 67B model.
• Hybrid and linear-attention layers. Some models replace most attention layers with layers that keep a fixed-size running summary instead of a growing cache. Qwen3-Next, for example, uses three such "Gated DeltaNet" layers for every standard attention layer, so only a quarter of its layers keep a KV cache. The trade-off is that a fixed-size summary can forget fine details, which is why a few full-attention layers are kept.

The formula above applies to standard attention layers; for these designs, read the model card or let your runtime report the cache size.

Quantizing the KV cache

Just as weights can be quantized, so can the cache. Runtimes such as llama.cpp and vLLM can store keys and values at 8-bit (and llama.cpp lower still). At 8-bit, the 128K example drops from about 43 GB to about 21 GB. The vLLM documentation notes that FP8 cache works best with calibration, and some sensitive layers may need to stay at full precision. Test quality on long documents before relying on it.

Context also costs time

Two separate costs grow with context:

• Prefill: reading the prompt. The model processes every prompt token, so a 100,000-token document takes far longer to start answering than a short question.
• Decode: writing each new token. Every step reads the whole KV cache as well as the weights, so generation slows as the context fills.

Hosted services feel this too, which is one reason long prompts cost more and why many offer discounted "cached" input for repeated prompt prefixes.

Going deeper

Deriving the cache from attention

In one attention head of one layer, token $t$ with hidden vector $x_t$ produces a query, key and value:

and the head's output is a weighted average of all earlier values:

where $d_h$ is the head dimension. Because the model is causal, $k_i$ and $v_i$ depend only on tokens up to $i$, so they never change once computed. Without a cache, step $t$ would recompute $t$ keys and values in every layer, and generating $n$ tokens would redo $1 + 2 + \dots + n \approx n^2/2$ projections. With a cache, each step computes one new key and value per KV head and appends them.

What is stored per token, per layer, is one key and one value vector for each KV head: $2\, n_{kv} d_h$ numbers. Multiply by $L$ layers, $b$ bytes per number, $T$ tokens and $B$ sequences served together (the batch):

This is the formula from earlier in the lesson, with the batch term that matters for servers: a server holding 16 conversations of 8K tokens needs as much cache as one 128K conversation.

Attention variants as choices of $n_{kv}$. Multi-head attention (MHA) has $n_{kv} = n_h$; grouped-query attention (GQA) has $n_{kv} = n_h / g$ for group size $g$; multi-query attention (MQA) has $n_{kv} = 1$. For an 80-layer model with $d_h = 128$ at 16-bit, the per-token cache is 2.62 MB with 64 KV heads, 0.33 MB with 8 and 41 KB with 1.

Multi-head latent attention stores something else entirely: a compressed latent vector of size $d_c$ plus a small decoupled positional key of size $d_h^R$ per token per layer. DeepSeek-V2 uses $d_c = 512$ and $d_h^R = 64$, so it caches 576 numbers per token per layer. With 60 layers at 16-bit that is $60 \times 576 \times 2 = 69{,}120$ bytes per token, against about 3.9 MB for full multi-head attention with its 128 heads of size 128.

Why decode is memory-bound

At each decode step, every layer reads its whole cache. For one sequence the attention computation does about $4\, n_h\, d_h\, T$ FLOPs per layer ($2 n_h d_h T$ for the scores $q^\top k$ and the same for the weighted sum of values), while reading $2\, n_{kv}\, d_h\, T\, b$ bytes. The ratio, the arithmetic intensity, is

With GQA group size $g = 8$ at 16-bit this is 8 FLOPs per byte. Modern accelerators can do hundreds of FLOPs per byte of bandwidth, so attention during decode is limited by memory reads, not arithmetic. Batching does not help here, because each sequence has its own cache.

That means long context slows decoding in the same way bigger weights do. Extending this site's speed formula so bytes per token include the cache (an extension of the AI Index method, not part of it): Llama 3 70B at about 4.5 bits on a 500 GB/s machine reads 39.4 GB of weights per token. At a full 128K context it also reads about 42.9 GB of cache, so the estimate falls from $1 / (39.4/275 + 0.0015) \approx 6.9$ to $1 / (82.3/275 + 0.0015) \approx 3.3$ tokens per second.

Prefill grows quadratically

During prefill, attention scores are computed for every pair of prompt tokens. With causal masking about half the pairs are needed, giving roughly $2\, L\, n_h\, d_h\, T^2$ FLOPs for scores and value sums together, on top of about $2N$ FLOPs per token for the weights. For Llama 3 70B at $T = 131{,}072$: the attention term is $2 \times 80 \times 64 \times 128 \times 131072^2 \approx 2.3 \times 10^{16}$ FLOPs, larger than the weight term $2 \times 70 \times 10^9 \times 131072 \approx 1.8 \times 10^{16}$. At 8K tokens the attention term is 256 times smaller, under a tenth of the weight term. Kernels such as FlashAttention reduce memory traffic for this step but not the quadratic FLOP count.

Serving: paging the cache

Servers do not know in advance how long each reply will be. Reserving the maximum context for every request wastes memory that could serve other users. PagedAttention, introduced with vLLM, stores each sequence's cache in fixed-size blocks allocated on demand, like virtual-memory pages, and lets sequences that share a prompt prefix share blocks. Reusing a cached prefix in this way is also what makes discounted cached-input pricing possible: the prefix's keys and values need not be recomputed.

Sliding windows in the formula

For a layer that attends only to the last $W$ tokens, the cache holds $\min(T, W)$ tokens instead of $T$. Gemma 3's pattern of five local layers (window 1,024) per global layer therefore stores, on average per layer, $\left(5\min(T, 1024) + T\right)/6$ tokens. At $T = 131{,}072$ that is about 22,700 tokens per layer, roughly 17% of what the same model would need if every layer were global.

Code: a cache calculator

def kv_cache_bytes(n_layers, n_kv_heads, head_dim, tokens, batch=1,
                   bytes_per_elem=2):
    """Standard (GQA/MHA) attention: one key and one value vector per
    KV head, per layer, per token, per sequence in the batch."""
    return 2 * n_layers * n_kv_heads * head_dim * bytes_per_elem * tokens * batch

def max_context(budget_gb, n_layers, n_kv_heads, head_dim, bytes_per_elem=2,
                batch=1):
    per_token = kv_cache_bytes(n_layers, n_kv_heads, head_dim, 1, batch,
                               bytes_per_elem)
    return int(budget_gb * 1e9 // per_token)

llama3_70b = dict(n_layers=80, n_kv_heads=8, head_dim=128)
for ctx in (8_192, 32_768, 131_072):
    gb16 = kv_cache_bytes(**llama3_70b, tokens=ctx) / 1e9
    gb8 = kv_cache_bytes(**llama3_70b, tokens=ctx, bytes_per_elem=1) / 1e9
    print(f"{ctx:>7} tokens: {gb16:5.1f} GB at 16-bit, {gb8:5.1f} GB at 8-bit")

# How much context fits in 20 GB of spare memory, at 16-bit and 8-bit?
print(max_context(20, **llama3_70b), max_context(20, **llama3_70b,
                                                 bytes_per_elem=1))

The output reproduces the worked example (about 2.7, 10.7 and 42.9 GB at 8K, 32K and 128K tokens at 16-bit) and shows that 20 GB of spare memory holds about 61,000 tokens of Llama 3 70B context at 16-bit, or about 122,000 with an 8-bit cache.

Common pitfalls

• Using query heads instead of KV heads. With GQA, plugging in $n_h$ overstates the cache by the group factor (8 times for Llama 3 70B).
• Forgetting the batch. Serving many users multiplies the cache; it, not the weights, usually limits concurrency.
• Applying the formula to MLA, sliding-window or hybrid layers. Use the architecture's own accounting or the runtime's report.
• Reserving maximum context by default. Many runtimes allocate the configured context up front.
• Ignoring speed. A long context costs decode speed (cache reads) and time-to-first-token (quadratic prefill), not just memory.

In practice

• Before loading a long document, estimate the KV cache from the formula and add it to the model's file size and about 1.5 GB of overhead.
• Set your local runtime's context length to what you actually need; reserving 128K when you use 8K wastes memory.
• If memory is tight, try an 8-bit KV cache before dropping to a smaller or more heavily quantized model.
• When choosing a model for long documents, look for GQA, sliding-window, MLA or hybrid designs, which keep the cache small.