RAG vs Long Context vs Fine-Tuning: Choosing an Architecture

Every team building on language models eventually asks the same question: our model does not know our material, so should we retrieve it, paste all of it into a very long prompt, or train it in? Since context windows grew into the hundreds of thousands of tokens and beyond, the question has become sharper, and it is often argued as a matter of taste. It should not be. Each approach changes something different about the system, and the right choice follows from the data, the query volume, the latency budget and the governance requirements. This lesson gives you a framework and the arithmetic to decide.

Intuition: three ways to prepare for an exam, plus a phone

Imagine a consultant preparing to answer clients' questions about a 2,000-page regulation.

• Long-context prompting is reading the entire regulation immediately before every single client call. Nothing is missed in principle, but it is slow and expensive each time, and attention flags in the middle of a long read.
• RAG is keeping a well-indexed binder and flipping to the three relevant sections when a question comes in. It is fast and cheap, and you can show the client the page, but if the index is bad you look in the wrong place.
• Fine-tuning is a training course that changes how the consultant thinks and talks: the jargon, the style of the memo, the habit of always checking exemptions. It is poor at memorising every clause reliably, and when the regulation is amended the course is out of date.
• Tool use is picking up the phone to the registry for things that change by the minute, such as the status of a particular filing. No amount of reading or training substitutes for a live lookup.

The central insight: fine-tuning mostly changes behaviour; retrieval and long context supply knowledge; tools supply live state and actions. Most confusion comes from using fine-tuning to inject facts or using retrieval to change style.

What each approach actually does

Long-context prompting puts the whole relevant corpus, or a large part of it, into the prompt on each request. Nothing is built offline. The model can reason across the whole text at once, which helps with global questions ("summarise all the changes across these contracts") that chunked retrieval handles poorly. Costs scale with prompt length on every call, though prompt caching (reusing the computed key-value state of an unchanged prompt prefix across requests, offered by many providers at a discounted price for cache hits) changes the economics substantially for a fixed corpus. Quality can degrade when relevant facts sit in the middle of long inputs (Liu et al., 2023), although models have improved on this.

Retrieval-augmented generation (Lewis et al., 2020, for the original formulation) indexes the corpus offline, retrieves a few relevant chunks per query and puts only those in the prompt. Cost per query is small and nearly independent of corpus size; updates mean re-indexing changed documents; and answers can cite exact passages. Its weakness is retrieval misses: if the right chunk is not retrieved, the model cannot use it.

Fine-tuning updates model weights on examples. Full fine-tuning updates all weights; parameter-efficient methods such as LoRA (Hu et al., 2021) train small low-rank adapter matrices instead. Fine-tuning is excellent for format, tone, domain vocabulary, classification and extraction tasks, tool-calling conventions, and distilling a big model's behaviour into a small one. It is unreliable for injecting large numbers of new facts: Ovadia et al. (2023) compared the two for knowledge injection and found that retrieval consistently outperformed unsupervised fine-tuning, both for knowledge seen in pre-training and for entirely new knowledge. Fine-tuned facts also cannot be cited, and removing a fact requires retraining.

Tool use lets the model call APIs and databases at run time: order status, inventory, prices, calendars, calculations, SQL over live tables. It is the only option for data that changes continuously or must be exact and authoritative, and the only one that can take actions.

The comparison

Worked cost and latency comparison

Scenario: a customer-support assistant over a product knowledge base of 300,000 tokens (small enough to fit in a large context window), handling 10,000 questions per day. Each answer is about 300 output tokens.

We use hypothetical prices chosen for easy arithmetic: USD 3.00 per million input tokens, USD 0.30 per million cached input tokens, and USD 15.00 per million output tokens. Real prices vary by provider and model and change often; substitute your own. Output cost is the same for every option: $300 \times 15 / 10^6 = 0.0045$ USD per query.

Option 1: long context, no caching. Input = 300,000 corpus + 200 question tokens = 300,200 tokens.

Times 10,000 queries: about USD 9,051 per day.

Option 2: long context with prompt caching, assuming every request hits the cache for the 300,000-token prefix (optimistic: real systems pay a premium on cache writes and lose the cache when it expires or the corpus changes):

About USD 951 per day, roughly a tenfold reduction.

Option 3: RAG, retrieving 8 chunks of 500 tokens plus 500 tokens of instructions and question, so 4,500 input tokens:

About USD 180 per day, plus the cost of embedding queries (negligible at these sizes) and running the vector index.

Option 4: fine-tuned model with a 700-token prompt, assuming (for simplicity) the same per-token price, although many providers charge more for custom models:

About USD 66 per day, plus training costs and, crucially, no citations and stale knowledge after every documentation change.

Latency and memory. Using the site's methodology, prefill costs about $2 \times$ (active parameters) FLOPs per prompt token. For a self-hosted model with 8 billion active parameters on hardware sustaining an effective 100 TFLOPS:

for the uncached long prompt, versus $2 \times 8 \times 10^9 \times 4{,}500 / 10^{14} \approx 0.72$ s for RAG. This counts only the linear term; attention adds a cost that grows with the square of the length, so the long-context figure is an underestimate. Caching avoids recomputing the prefix, which is why cached long-context requests have a much shorter time to first token.

The KV cache (the stored keys and values for every token in the context) must also fit in memory. For a common 8B configuration with 32 layers, 8 KV heads and head dimension 128 at 2 bytes per element:

so 300,200 tokens need about $300{,}200 \times 131{,}072 \approx 39.3$ GB per concurrent request, against about 0.59 GB for a 4,500-token RAG prompt. On self-hosted hardware, long context limits how many users you can serve at once.

The calculation in code, so you can plug in your own prices and model shapes:

PRICE = {"in": 3.00, "cached_in": 0.30, "out": 15.00}   # hypothetical $ per 1M tokens

def cost_per_query(fresh_in, cached_in=0, out=300, p=PRICE):
    return (fresh_in * p["in"] + cached_in * p["cached_in"] + out * p["out"]) / 1e6

def prefill_seconds(fresh_tokens, active_params=8e9, eff_flops=100e12):
    """Linear-term estimate; cached prefix tokens are not recomputed."""
    return 2 * active_params * fresh_tokens / eff_flops

def kv_cache_gb(tokens, layers=32, kv_heads=8, head_dim=128, bytes_per=2):
    return 2 * layers * kv_heads * head_dim * bytes_per * tokens / 1e9

strategies = {                     # (fresh input tokens, cached input tokens)
    "long context, no cache": (300_200, 0),
    "long context, cached":   (200, 300_000),
    "RAG (8 x 500-token chunks)": (4_500, 0),
    "fine-tuned, short prompt":   (700, 0),
}
for name, (fresh, cached) in strategies.items():
    c = cost_per_query(fresh, cached)
    print(f"{name:28s} ${c:.4f}/q  ${c * 10_000:>7,.0f}/day  "
          f"prefill~{prefill_seconds(fresh):6.3f}s  KV~{kv_cache_gb(fresh + cached):5.2f} GB")

Running it reproduces the figures above (USD 0.9051, 0.0951, 0.0180 and 0.0066 per query) and shows the cached long-context prefill dropping to milliseconds while its KV memory stays at about 39 GB.

What the numbers do not say. Cost is only one axis. If the support corpus changes daily and answers must cite the article, fine-tuning is ruled out no matter how cheap it is. If many questions need cross-document synthesis, RAG's savings may come with an accuracy loss that costs more in escalations. Always pair the cost model with measured accuracy on your evaluation set.

Accuracy: what the research suggests

Results comparing RAG and long context depend heavily on model, task and retrieval quality, so treat any single study as indicative. Li et al. (2024) compared the two across several benchmarks and found that sufficiently capable long-context models outperformed RAG on average, while RAG was far cheaper. They proposed Self-Route: the model first tries to answer from retrieved chunks and declares whether the question is answerable from them; only questions it judges unanswerable are routed to the full long-context prompt. Because most queries were answerable from retrieval, this recovered most of the long-context quality at a fraction of the cost. The broader lesson is that the choice need not be global: it can be made per query.

Hybrid patterns

The strongest production systems usually combine approaches, each doing what it is best at:

• RAG plus fine-tuning. Fine-tune for behaviour (output schema, tone, citing chunk IDs, refusing when context is insufficient) and use retrieval for facts. RAFT (Zhang et al., 2024) fine-tunes models specifically to answer from retrieved documents while ignoring distractor documents, improving domain RAG performance.
• Retrieve, then read long. Use retrieval to select whole documents or large sections rather than small chunks, then rely on a long context to reason across them. This reduces the chunk-boundary problems of classic RAG while keeping prompts far below the full corpus.
• Self-Route and escalation. Answer cheaply with RAG first, and escalate to long context, more retrieval rounds or a human when the model reports insufficient evidence.
• Cached long context for small, stable corpora. A product manual or a codebase of moderate size can live in a cached prompt prefix, with no index to maintain.
• Tools for live and exact data. Route prices, stock, account status and aggregates to APIs or SQL, and keep retrieval for unstructured text.
• Distillation. Use a large long-context or agentic system to generate high-quality answers, then fine-tune a small model on them for a high-volume, narrow task, keeping retrieval for facts.

A decision procedure in words

Work through these questions in order. The first decisive answer usually settles the main architecture; later questions refine it.

1. Is the problem behaviour or knowledge? If the model already knows enough but answers in the wrong format, tone or task framing, try prompting first and then fine-tune. Retrieval will not fix style.
2. Does the information change continuously, or must it be exact and authoritative (prices, balances, inventory, status)? Use tool use or SQL, not documents.
3. Do answers need citations, per-user permissions, or the ability to delete specific facts quickly? That rules out fine-tuning as the knowledge source; use RAG or long context built per user.
4. Does the relevant corpus fit comfortably in the context window (say, well under half of it, leaving room for instructions and output)?
   • If no, use RAG, possibly retrieving whole documents into a long context.
   • If yes, go to 5.
5. What is the query volume and latency budget?
   • Low volume, tolerant latency, or many global cross-document questions: long context, with prompt caching if the corpus is stable.
   • High volume or tight latency: RAG, or Self-Route, with long context as an escalation path.
6. After building the knowledge path, are there remaining behaviour gaps (schema compliance, refusal behaviour, domain jargon, cost of a large model)? Add fine-tuning or distillation on top.
7. Validate with your evaluation set: measure accuracy, faithfulness, cost and latency for the top two candidates before committing.

Decision table

Common pitfalls

• Fine-tuning to teach facts, then discovering the model mixes them up, cannot cite them and is stale after the next document update.
• Assuming long context makes retrieval obsolete without measuring cost at production volume, latency, and quality on facts buried mid-context.
• Comparing uncached long-context cost with a cached deployment or ignoring cache-write premiums and expiry when estimating savings.
• Ignoring access control: a single long prompt or a fine-tuned model shared across users can leak documents a user is not entitled to see.
• Treating the choice as global instead of routing per query by type and difficulty.
• Deciding on cost alone, without measured accuracy and faithfulness on a golden set.
• Forgetting maintenance: fine-tuned models must be retrained and re-evaluated when the base model is updated or deprecated, and RAG indexes need ingestion monitoring.

In practice

Begin with prompting plus the simplest knowledge path that satisfies your constraints: cached long context for small stable corpora, RAG for anything large, changing, permissioned or requiring citations, and tools for live data. Build the evaluation set first, then measure accuracy, faithfulness, cost per query and p95 latency for the two most plausible designs on it. Add per-query routing when your traffic mixes easy lookups with hard synthesis questions, and add fine-tuning last, for behaviour that prompting cannot reliably produce or to cut cost on a narrow high-volume task. Revisit the decision whenever prices, context limits or model capabilities change, because the break-even points move, while the underlying logic of knowledge versus behaviour versus live state does not.