Context engineering: designing what the model sees

A chat with a model is one prompt. An application or agent is a machine that assembles a prompt on every call from many parts: standing instructions, tool definitions, retrieved documents, stored memories, conversation history and the newest message. Context engineering is the discipline of deciding what goes into that window, in what form, in what order and at what size. Anthropic's engineering team described it in 2025 as the natural evolution of prompt engineering: curating the smallest set of high-signal tokens that makes the desired behaviour most likely. In production systems it usually matters more than the wording of any single instruction.

Intuition: packing a briefcase for a long trip

The model is a traveller who can carry exactly one briefcase per call, of fixed size, and who forgets everything not in it. Pack too little and they lack what they need. Pack too much and they cannot find the important paper among the clutter, and the briefcase costs more to carry. Good packing means: essentials always in the same pocket (so security can wave them through quickly, which is what caching does), today's papers on top, old receipts summarised onto one page, and a note telling them where the filing cabinet is so they can fetch the rest when needed.

What goes into the context window

The context window is a hard limit on input plus output (for most APIs, reasoning tokens count against it too). But the practical limit is lower, for three reasons: cost (you pay for every input token on every call), latency (prefill time grows with input length), and quality.

Context rot and position effects

Long-context models can accept hundreds of thousands of tokens, but that does not mean they use all of them equally well.

• Lost in the middle. Liu et al. (2023) showed that on multi-document question answering, models used relevant information best when it sat at the beginning or end of the context and worst when it was in the middle.
• Context rot. A 2025 Chroma technical report tested 18 models and found that performance degraded as input length grew, even on simple retrieval and text-replication tasks; distracting passages that resembled the answer made it worse; and on a conversational memory benchmark, giving models a focused prompt with only the relevant part beat giving them the full history of roughly 113,000 tokens.

Anthropic's framing is an attention budget: every token added competes for the model's attention, so each one should earn its place. Practical consequences: retrieve fewer, better chunks; remove stale tool output; summarise old turns; and test your application at the context lengths it will actually reach, not just on short demos.

Ordering: quality and caching

Two ordering principles apply, and fortunately they mostly agree.

For quality: put long reference material near the top and the specific question and instructions for this turn at the end. Anthropic's documentation reports that placing the query after long documents (20,000+ tokens) can improve response quality by up to 30% in its tests. Asking the model to quote the relevant passages before answering also helps it find what matters.

For caching: prompt caching stores the model's processed state for a prompt prefix so later calls sharing that exact prefix skip recomputation and pay a reduced rate. As of November 2026, OpenAI documents that caching requires the entire prefix to match exactly, that the minimum cacheable prefix is 1,024 tokens on its recent models, and that cached reads are billed at a steep discount (0.1 times the normal input rate on most of its newest models, and less on some). Other providers, including Anthropic and Google, offer caching with explicit cache markers or automatic detection, each with its own prices and lifetimes; check current docs.

The layout that serves both goals:

[1] System prompt                (static: identical for every user)
[2] Tool definitions             (static)
[3] Few-shot examples            (static)
--- cache boundary A: shared by everyone ---
[4] User memory / project notes  (static within a session)
--- cache boundary B ---
[5] Conversation history         (append-only; grows at the end)
[6] Retrieved documents for this turn
[7] Current user message + turn-specific instructions

The rule: static first, dynamic last, never edit the middle. A timestamp in the system prompt, tools reordered between calls, or a user's name injected into line 1 breaks the cache for everything after it. Placing retrieved documents in [6] keeps them near the question, which helps quality. (If you need very large shared documents, put them in the static block instead.)

Worked example: a token budget for a support agent

Suppose a model with a 200,000-token context window, used as a customer-support agent.

Step 1: reserve output. The agent uses a reasoning model and sometimes writes long answers. Reserve 32,000 tokens. Input budget: $200{,}000 - 32{,}000 = 168{,}000$.

Step 2: fixed components. System prompt 4,000 tokens. Tools: 25 tools at about 350 tokens each $= 8{,}750$. User memory 2,000. Fixed total $= 4{,}000 + 8{,}750 + 2{,}000 = 14{,}750$.

Step 3: per-turn components. Retrieval returns 8 chunks of about 1,000 tokens $= 8{,}000$. Current message up to 2,000. Per-turn total $= 10{,}000$.

Step 4: history. The hard ceiling for history is $168{,}000 - 14{,}750 - 10{,}000 = 143{,}250$ tokens. But because of context rot, cost and latency, the team sets a soft target of 60,000 total input tokens per call, leaving $60{,}000 - 24{,}750 = 35{,}250$ tokens for history. Policy: keep the most recent turns verbatim; when history exceeds 35,250 tokens, replace the oldest turns with a summary of at most 2,000 tokens.

Step 5: cost with and without caching. Take a hypothetical price of 3 US dollars per million input tokens, cache reads at 0.1 times that rate, and cache writes at 1.25 times (a premium some providers charge; others charge none). Over a 40-turn session, the static prefix (14,750 tokens) is sent 40 times.

• Without caching: $40 \times 14{,}750 = 590{,}000$ tokens, $\times \text{\textdollar}3/10^6 = \text{\textdollar}1.77$.
• With caching: first call writes at $1.25 \times 14{,}750 = 18{,}437.5$ token-equivalents; 39 reads at $0.1 \times 14{,}750 = 1{,}475$ each, $39 \times 1{,}475 = 57{,}525$. Total $75{,}962.5$ token-equivalents $\times \text{\textdollar}3/10^6 \approx \text{\textdollar}0.23$.

That is about an 87% saving on the static part alone, before counting the growing history, which can also be cached incrementally because it is append-only. Real numbers depend on the provider's prices and on the cache expiring between turns.

Step 6: tool bloat check. The 8,750 tokens of tools are 59% of the fixed budget. If analytics show that 10 of the 25 tools are almost never used, moving them behind a single "search more tools" tool, or into a sub-agent, saves around 3,500 tokens per call and reduces confusion between similar tools.

Managing conversation history

Three standard strategies, often combined:

1. Truncation (sliding window): keep the last $N$ turns or last $T$ tokens. Simple and predictable, but every time the window slides the start of the history changes, which breaks the cache for it, and early facts (the user's name, the original goal) are lost.
2. Summarisation (compaction): replace old turns with a model-written summary. Preserves key facts at a fraction of the size, at the cost of an extra call and some information loss. Anthropic describes clearing old tool results as the lightest form of compaction: the raw 30,000-token log the agent already read rarely needs to be shown again.
3. Pinning: always keep certain items, such as the user's original request, decisions made and open to-dos, regardless of age.

def est_tokens(text: str) -> int:
    return max(1, len(text) // 4)        # rough heuristic: ~4 characters per token in English

def summarize(turns):                    # stand-in; in production, call a model
    return "Summary of earlier conversation: " + " | ".join(t["content"][:40] for t in turns)

def build_history(turns, budget, keep_recent=6, pinned=None):
    # return a history list that fits in `budget` tokens
    pinned = pinned or []
    recent = turns[-keep_recent:]
    older = turns[:-keep_recent]
    used = sum(est_tokens(t["content"]) for t in pinned + recent)
    out = list(pinned)
    if older:
        summary = {"role": "user", "content": summarize(older)}
        if used + est_tokens(summary["content"]) <= budget:
            out.append(summary)
            used += est_tokens(summary["content"])
    while recent and used > budget:      # still too big: drop oldest recent turns
        used -= est_tokens(recent.pop(0)["content"])
    return out + recent

turns = [{"role": "user" if i % 2 == 0 else "assistant",
          "content": f"turn {i} " + "x" * 400} for i in range(20)]
hist = build_history(turns, budget=1000)
print(len(hist), sum(est_tokens(t["content"]) for t in hist))

The heuristic of about four characters per token is only an English-text approximation; use the provider's tokenizer or token-counting endpoint for real budgets, especially for code and non-English text. To stay cache-friendly, recompute the summary only occasionally (for example, when history crosses the budget) rather than on every turn, so the prefix stays stable between compactions.

Context for agents: just-in-time and beyond

Long-running agents can generate far more context than any window holds: file contents, search results, command output. Patterns that work:

• Just-in-time retrieval: keep lightweight references (file paths, URLs, record IDs, queries) in context and load full content with tools only when needed, instead of pre-loading everything. Many coding agents use a hybrid: a few key files loaded up front, the rest explored on demand.
• Compaction: when nearing the limit, summarise the trajectory (decisions, open problems, key findings) and continue in a fresh window.
• Structured note-taking: the agent writes progress notes or a to-do file outside the context window and reads it back later, so state survives compaction.
• Sub-agents: a coordinator delegates focused sub-tasks to sub-agents with clean windows, which return condensed results (Anthropic mentions summaries of roughly 1,000 to 2,000 tokens) instead of their full exploration.
• Tool design: tools should return concise, relevant output (paginated, filtered, truncated with a note) rather than dumping 50,000 tokens of raw data into context.

Tools are context too

Every tool definition is sent on every call, and the model must choose among them on every step, so tool design is context design. Good tools have distinct purposes, unambiguous names (search_orders_by_customer_id rather than lookup), descriptions that say when to use and when not to use them, minimal required parameters and concise outputs. If a human engineer could not say for certain which of two tools fits a situation, the model will not either. When an agent needs dozens of tools, consider exposing a small core set plus a tool-search tool that loads others on demand, or splitting responsibilities across sub-agents that each see only their own tools. Measure tool-selection accuracy on an eval set just as you would measure answer quality, because a wrong tool call wastes a full round trip and pollutes the history with irrelevant output.

Memory

Memory is information persisted outside the model and selectively re-inserted: user preferences, project facts, prior decisions. Inserting all of it every time wastes budget and can mislead; better systems retrieve the relevant memories per turn, keep them short and dated, and treat them as data, not instructions. The AI 320 lesson ai-memory-systems covers designs in depth.

Common pitfalls

• Dumping everything in because the window is big. More context often lowers accuracy and always raises cost and latency.
• Breaking the cache by accident: timestamps, random IDs, unstable JSON key order or reordered tools near the top of the prompt.
• Unbounded tool output: a single log dump can push the system prompt's instructions out of effective attention.
• Lossy summaries of critical facts: summarisation silently drops the account number or the constraint the user stated in turn 2. Pin critical facts.
• Forgetting reasoning and output tokens in the budget, then getting truncated answers.
• Testing only short conversations. Bugs from context rot appear at turn 40, not turn 3.
• Overlapping tools with similar names and vague descriptions, which make the agent choose wrongly.

In practice

Treat the context as a designed artifact with an explicit budget per component, logged on every call. Engineers instrument token counts per section, cache-hit rates and quality metrics at different conversation lengths, and alert when a component grows unexpectedly. Start with a stable, cache-friendly layout (static first, dynamic last), retrieve narrowly, summarise or clear old material, and for agents prefer references plus just-in-time loading over pre-loading. Re-measure whenever you add a tool, change retrieval settings or switch models, because each model has different long-context behaviour and each provider prices caching differently.