Context Windows and Memory

Every AI chat has a limit on how much text the model can consider at once. Knowing that limit, and what really happens when a chat app "remembers" you, explains a lot of odd behaviour, from forgotten instructions to answers that get worse in very long conversations.

What a context window is

The context window is the maximum amount of text a model can take in and work with in a single request. It includes everything: the app's hidden instructions, your messages, the model's earlier replies, any documents you attached, and the answer the model is writing now.

An analogy: the context window is the model's desk. Whatever is on the desk, it can look at. Anything that does not fit on the desk simply is not there, no matter how important it was earlier. The model does not have a filing cabinet behind it that it can rummage through on its own.

Measured in tokens

Context windows are measured in tokens, the small chunks of text a model actually reads. A token is often a whole short word, part of a longer word, or a punctuation mark. As a rough rule of thumb for English, one token is about three-quarters of a word, so 1,000 tokens is roughly 750 words. Other languages, code and unusual words often use more tokens per word, so treat this as an estimate.

Worked example. A 300-page novel has roughly 100,000 words. At about 0.75 words per token, that is around 133,000 tokens. A model with a 128,000-token window could not hold the entire book plus your question; a model with a 1-million-token window could, with plenty of room left.

As of November 2026, leading hosted models commonly advertise context windows from a few hundred thousand tokens up to around a million. For example, Google lists a 1-million-token window for Gemini 4 Argon, and Anthropic lists a 1-million-token window for Claude Opus 5.5. Smaller models, and models run on your own computer, often work with much less. These figures change frequently, so check the current documentation for the model you use.

What "memory" in a chat app actually is

The model itself does not remember anything between requests. Each time you send a message, the app sends the model a fresh bundle of text. What feels like memory is built from a few tricks:

• Re-sent history. Within one conversation, the app re-sends earlier messages along with your new one. That is why the model "remembers" what you said ten messages ago: it is reading it again.
• Summaries. When a conversation gets too long to fit, some apps quietly compress older parts into a summary, or drop them entirely. Details can be lost in that compression.
• Saved memories and personalisation. If your app has a memory or personalisation feature, it typically stores short notes about you (for example, "prefers metric units") and inserts them into the context of new chats. The model has not learned these facts permanently; they are just added text.
• Custom instructions. Standing instructions you set in settings are added to every conversation in the same way.

So a model can "forget" an instruction from early in a long chat because it was summarised away, dropped, or simply given less weight among a lot of other text.

Long documents

Large context windows let you paste in long reports, contracts or transcripts and ask questions about them. That is genuinely useful, but there are trade-offs:

• Cost and speed. Every token in the window has to be processed. Longer inputs mean slower responses and, on paid services that charge per token, higher bills. Part of the reason is that the model keeps a working record of everything it has read, called the KV cache, which grows with the length of the context and takes up memory. (The lesson on KV cache and context cost covers this in detail.)
• Usage limits. Many consumer plans cap how much you can use in a period, and long contexts use that allowance up faster.
• Not always necessary. If you only need one section, paste only that section.

Why quality can drop deep into long contexts

A model accepting a million tokens does not mean it pays equal attention to every one of them. Research has repeatedly found that reliability falls as inputs get longer:

• A 2023 study, "Lost in the Middle", found that models answered questions best when the relevant information was near the start or the end of a long input, and noticeably worse when it was buried in the middle.
• A 2025 technical report from the company Chroma tested 18 models and found that performance declined as input length grew, even on simple tasks such as finding and repeating a piece of text.

Newer models have improved, but the general lesson still holds: more context is not free, and the most important material can get diluted by everything around it.

Going deeper

Where tokens come from

Most modern models split text with a subword method descended from byte-pair encoding (BPE), adapted for language models by Sennrich, Haddow and Birch (2015). The idea: start from single characters (or bytes), then repeatedly merge the most frequent adjacent pair into a new token until the vocabulary reaches a target size. Common words end up as single tokens; rare words are built from several pieces. That is why "the" is one token, an unusual surname may be four, and languages or code that were less common in the tokenizer's training text need more tokens per word. The 0.75-words-per-token rule is an average for English prose; for precise counts, run the model's own tokenizer.

Budgeting the window

Everything shares one window of $W$ tokens:

$T_{\text{system}} + T_{\text{history}} + T_{\text{documents}} + T_{\text{output}} \le W$

The output counts too, so you must reserve room for the answer before filling the rest.

Worked example. A model has $W = 128{,}000$. The app's system prompt uses 2,000 tokens, the conversation so far 30,000, and you reserve 8,000 for the reply. The room left for documents is $128{,}000 - 2{,}000 - 30{,}000 - 8{,}000 = 88{,}000$ tokens, about $88{,}000 \times 0.75 = 66{,}000$ words. A 100,000-word document will not fit; you would trim it, summarise parts, or start a fresh chat to free the 30,000 tokens of history.

Why long contexts cost more

In a transformer, each new token is compared against every earlier token by the attention mechanism. Reading a prompt of $n$ tokens therefore involves roughly $n^2/2$ pairwise comparisons per attention layer. Doubling $n$ roughly quadruples that work, which is one reason very long prompts are slow to start answering. (Real systems use many optimisations, and other parts of the model grow only linearly with $n$, so total cost grows somewhat less than fourfold; the direction of the effect is the point.)

The model also stores, for every token it has read, a key and a value vector in every layer so it does not recompute them: the KV cache. Its size grows linearly with context length:

$\text{KV bytes per token} = 2 \times L \times H_{kv} \times d_{\text{head}} \times b$

where $L$ is the number of layers, $H_{kv}$ the number of key/value heads, $d_{\text{head}}$ the size of each head, $b$ the bytes per stored number, and the 2 counts keys and values.

Worked example. A configuration typical of 8-billion-parameter models has $L = 32$, $H_{kv} = 8$, $d_{\text{head}} = 128$, stored at 16-bit precision ($b = 2$):

$2 \times 32 \times 8 \times 128 \times 2 = 131{,}072 \text{ bytes} = 128 \text{ KiB per token.}$

At 100,000 tokens that is $131{,}072 \times 100{,}000 \approx 13.1$ GB, more than the model's own weights at 4-bit precision. This is why local tools often default to short contexts, and why hosted providers price long inputs carefully.

Advertised versus effective context

A model accepting a million tokens tells you what it can read, not what it can use well. Researchers measure this with tests such as "needle in a haystack" (hide one fact at different depths of a long filler text and ask for it) and harder suites such as RULER (Hsieh et al., 2024), which adds multi-step tracing and aggregation tasks. RULER's authors found that almost all models aced the simple needle test, yet performance dropped substantially on the harder tasks as length grew, and only about half of the models that claimed windows of 32,000 tokens or more held up satisfactorily at 32,000. The useful idea is effective context length: the longest input at which a model still performs your kind of task reliably. It is often well below the advertised figure, and you can measure it for your own task.

Trimming history in code

Chat applications must decide what to drop when a conversation outgrows its budget. The simplest policy keeps the most recent messages that fit:

def est_tokens(text):
    # Rough English estimate (~0.75 words per token).
    # For real counts, use the model's own tokenizer.
    return round(len(text.split()) / 0.75)

def fit_history(system, messages, window=128_000, reserve_out=8_000):
    budget = window - reserve_out - est_tokens(system)
    kept = []
    for msg in reversed(messages):        # walk from newest to oldest
        cost = est_tokens(msg)
        if cost > budget:
            break
        kept.append(msg)
        budget -= cost
    return list(reversed(kept)), budget   # restore chronological order

msgs = ["word " * 3000, "word " * 59000, "word " * 1500]   # three old-to-new messages
kept, left = fit_history("You are helpful.", msgs, window=90_000)
print(len(kept), "messages kept;", left, "tokens left for documents")

With a 90,000-token window, the newest two messages (about 2,000 and 78,700 estimated tokens) fit and the oldest (about 4,000) is dropped. Notice the weakness: if the oldest message held the original task statement, it is now gone, which is exactly the "forgotten instruction" behaviour described above. Production systems therefore pin important messages, summarise old ones, or retrieve relevant pieces on demand (the idea behind retrieval-augmented generation).

Common pitfalls

• Treating the advertised window as usable capacity. Test your task at the lengths you actually need.
• Forgetting to reserve space for the answer. A prompt that fills the window leaves the model no room to reply, or causes the reply to be cut off.
• Assuming the 0.75 rule holds everywhere. Code, tables, non-English text and unusual names can need far more tokens.
• Believing memory means learning. Saved memories are inserted text; delete them if they are wrong, or they will keep steering answers.
• Pasting everything to save effort. Long inputs cost money and time and can dilute the part that matters.
• Dropping the oldest messages blindly in your own apps; pin the task statement and key constraints.

In practice

• Start fresh for new topics. A new chat gives the model a clean desk with no leftover clutter from earlier tasks.
• Restate key instructions in long conversations, or put them in custom instructions so they are included every time.
• Put the important part where it is easy to find. Place your question and key instructions at the end, after a long document, and say which section matters most.
• Trim what you paste. Remove boilerplate, unrelated sections and duplicate copies.
• Ask for a summary, then continue from it. For very long projects, ask the model to summarise decisions so far, then start a new chat with that summary.
• Check what your app remembers. Look in settings for memory or personalisation features, see what is stored, and delete anything you do not want carried into future chats.