Embeddings and Retrieval-Augmented Generation

A model only knows what was in its training data and what you put in its prompt. If you want it to answer questions about your company handbook, last week's tickets or a 10,000-page archive, you need a way to find the few relevant passages and hand them to the model. That is what retrieval-augmented generation does.

Embeddings: meaning as coordinates

An embedding is a list of numbers (a vector) that an embedding model produces from a piece of text. The model is trained so that texts with similar meaning end up with similar vectors. "How do I reset my password?" and "I forgot my login credentials" share few words, but their vectors land close together.

An analogy: imagine every sentence pinned on an enormous map where nearby pins mean similar things. A real embedding space has hundreds or thousands of dimensions instead of two, but the idea is the same. "Closeness" is usually measured with cosine similarity, which compares the direction of two vectors and gives a score near 1 for very similar meaning.

Embedding models are separate from chat models. They are typically much smaller and cheaper to run, and they output only vectors, never text.

Vector search

To search, you embed every passage in your collection once and store the vectors in a vector index (a vector database, or a vector extension to a database you already run). At query time you embed the question and ask the index for the nearest vectors. For large collections, indexes use approximate nearest neighbour algorithms that trade a tiny bit of accuracy for very large speed gains.

Storage is predictable. A 1,024-dimension vector stored as 32-bit floats takes 1,024 × 4 = 4,096 bytes, about 4 KB. One million chunks therefore need about 4 GB for the vectors alone, before index overhead and the original text.

Chunking

You rarely embed whole documents. A single vector for a 200-page manual blurs everything together. Instead you split documents into chunks, typically a few hundred tokens each, often with a small overlap so a sentence cut at a boundary still appears whole in one chunk.

Worked example: a 200-page handbook of roughly 100,000 tokens, split into 500-token chunks with 50 tokens of overlap, gives about 100,000 ÷ 450 ≈ 220 chunks. Good chunking follows the document's structure (headings, sections, table rows) rather than cutting blindly every N tokens, and stores useful metadata with each chunk: title, section, date, URL, and who is allowed to see it.

The RAG pipeline

Retrieval-augmented generation (RAG) combines search with generation:

1. Ingest: split documents into chunks, embed them, store vectors plus text and metadata.
2. Retrieve: embed the user's question and fetch the top matching chunks (say, 20 to 50).
3. Rerank (optional but valuable): re-score those candidates more carefully and keep the best few.
4. Generate: put the selected chunks into the prompt with instructions such as "answer only from these sources; if they don't contain the answer, say so; cite the source ID for each claim."
5. Cite: show the user which documents the answer came from so they can check.

The model is not learning your data; it is reading it on the spot, like an open-book exam. That means updates take effect as soon as you re-index a document, and you can delete data cleanly.

Hybrid search and reranking

Pure vector search is good at meaning but weak at exact strings: product codes, error numbers, names, rare jargon. Classic keyword search (typically an algorithm called BM25, which scores documents by how often the query's words appear, weighting rare words more) is the opposite. Hybrid search runs both and merges the results; a simple, popular merge method is reciprocal rank fusion, which rewards chunks that rank highly in either list.

A reranker is a second model that reads the question and each candidate chunk together and scores how well the chunk answers the question. It is slower than vector search, so you only run it on the top few dozen candidates, but it is usually more accurate at picking the final handful.

Citing sources

Give each chunk an ID in the prompt, ask the model to tag claims with those IDs, and then verify in code that every cited ID was actually provided. Show citations as links back to the original document and section. Citations do not guarantee correctness, since a model can cite a real source for a claim the source doesn't make, but they make checking fast.

Common failure modes

• Retrieval misses: the right passage exists but isn't in the top results, often because of poor chunking, a vocabulary mismatch, or missing keyword search. The model then answers from general knowledge or guesses.
• Lost context: a chunk says "this limit applies" without saying which product. Prepending the document title and section heading to each chunk helps.
• Too much context: stuffing 40 chunks into the prompt can bury the relevant one; research has found models attend less reliably to information in the middle of long prompts.
• Stale or duplicate data: old versions of a policy outrank the new one. Track dates and remove superseded documents.
• Permission leaks: retrieval returns a chunk the user isn't allowed to see. Filter by access rights at retrieval time, not after generation.
• Injected instructions: a retrieved document contains text like "ignore previous instructions." Treat retrieved text as data, never as commands.

Going deeper

This lesson opens the course on knowledge systems. The later lessons take each stage apart in detail; here is the rigorous core that all of them build on.

Similarity, precisely

Let an embedding model map a text to a vector $\mathbf{v} \in \mathbb{R}^d$. For a query vector $\mathbf{q}$ and a chunk vector $\mathbf{c}$, cosine similarity is

which ranges from $-1$ to $1$. If you normalize every vector to unit length when you store it, cosine similarity becomes a plain dot product, which is what most vector indexes compute fastest. For unit vectors it is also tied to Euclidean distance by $\lVert \mathbf{q} - \mathbf{c} \rVert^2 = 2 - 2\cos(\mathbf{q}, \mathbf{c})$, so ranking by smallest distance and ranking by largest cosine give the same order.

Worked example in three dimensions (real embeddings have hundreds or thousands). Query $\mathbf{q} = (0.8, 0.6, 0)$, which already has length 1. Three chunks:

So the ranking is B, A, C. Here the raw dot products happen to give the same order, but in general they need not: with unnormalized vectors a long vector can outscore a better-aligned short one, so normalize before ranking by dot product.

How embedding models learn "closeness"

Retrieval embedding models are commonly trained with a contrastive objective. Given a query $q$, a passage $p^{+}$ known to answer it and a set of other passages treated as negatives, the model is pushed to score the positive above the negatives. With similarity $s(\cdot,\cdot)$ and a temperature $\tau$, a standard form of the loss is

which is a softmax cross-entropy over "which passage is the right one". Dense Passage Retrieval (Karpukhin et al., 2020) trained separate query and passage encoders in this spirit, using the other questions' passages in the same training batch as cheap negatives. Two practical consequences follow. First, an embedding model is only as good as the kinds of query and passage pairs it was trained on, so a general model can underperform on specialized jargon. Second, queries and documents must be embedded with the same model (and, for models that expect them, the same query and document prefixes); vectors from different models live in unrelated spaces.

Keyword scoring: BM25

BM25 scores a document $D$ for a query with terms $t_1, \dots, t_n$ as

where $f(t_i, D)$ is how often the term occurs in $D$, $|D|$ is the document's length, $\text{avgdl}$ is the average length in the collection, $\text{IDF}$ rewards rare terms, $k_1$ (typically around 1.2 to 2) controls how quickly repeated occurrences stop adding score, and $b$ (typically 0.75) controls how much long documents are penalized. The saturation in $k_1$ is why stuffing a page with a keyword stops helping, and the IDF weight is why an exact error code such as "E-4021" is so decisive for keyword search.

Fusing rankings: reciprocal rank fusion

BM25 scores and cosine similarities live on different scales, so you can't simply add them. Reciprocal rank fusion (RRF) ignores scores and combines ranks:

with $k = 60$ a common default from the original RRF paper (Cormack, Clarke and Büttcher, 2009). Worked example: keyword search returns A, B, C and vector search returns C, A, D.

• A: $1/61 + 1/62 = 0.01639 + 0.01613 = 0.03252$
• C: $1/63 + 1/61 = 0.01587 + 0.01639 = 0.03227$
• B: $1/62 = 0.01613$; D: $1/63 = 0.01587$

Fused order: A, C, B, D. Documents found by both retrievers rise to the top, which is exactly the behaviour you want from hybrid search.

import numpy as np

def normalize(X):
    X = np.asarray(X, dtype=np.float32)
    return X / np.linalg.norm(X, axis=-1, keepdims=True)

def top_k(query_vec, doc_matrix, k=3):
    """Exact cosine search; rows of doc_matrix are already normalized."""
    scores = doc_matrix @ normalize(query_vec)
    idx = np.argsort(-scores)[:k]
    return [(int(i), float(scores[i])) for i in idx]

def rrf(rankings, k=60):
    """Reciprocal rank fusion. rankings: list of ranked doc-id lists."""
    fused = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(fused, key=fused.get, reverse=True)

docs = normalize([[0.9, 0.1, 0.0], [0.7, 0.7, 0.1], [0.0, 0.2, 0.98]])
print(top_k([0.8, 0.6, 0.0], docs, k=2))       # chunk 1 (B) then chunk 0 (A)
print(rrf([["A", "B", "C"], ["C", "A", "D"]]))  # ['A', 'C', 'B', 'D']

Exact search like top_k costs $O(Nd)$ per query, about a billion multiply-adds for a million 1,024-dimension chunks. That is fine for small collections and is the ground truth you measure approximate indexes against; the next lesson, on vector databases and indexing, covers the approximate structures (such as HNSW graphs) that make search over millions of vectors fast.

Measuring retrieval

Because most RAG failures are retrieval failures, measure retrieval on its own with a set of questions labelled with the chunks that should be found:

• Recall@k: the fraction of relevant chunks that appear in the top $k$. If the right passage is not retrieved, the generator cannot use it.
• Mean reciprocal rank (MRR): the average over queries of $1/\text{rank}$ of the first relevant result (0 if none is found).

Worked example: four questions whose first relevant chunk appears at ranks 1, 3, not found, and 2. MRR $= (1 + 1/3 + 0 + 1/2)/4 \approx 0.458$. If each question has a single relevant chunk and the top 5 are kept, Recall@5 $= 3/4 = 0.75$. The lesson on evaluating RAG extends this to answer faithfulness and end-to-end metrics.

Where this course goes next

Each stage of the pipeline above has its own lesson in this course:

• Vector databases and indexing: approximate nearest-neighbour indexes, filtering, and the memory and speed trade-offs.
• Document processing and chunking: parsing PDFs, tables and slides, and structure-aware chunking strategies.
• Advanced retrieval: query rewriting, rerankers, contextual chunk enrichment and multi-vector methods.
• GraphRAG and knowledge graphs: retrieving over entities and relationships for questions that span many documents.
• Agentic RAG: letting a model decide when and what to search, over several rounds.
• Evaluating RAG: retrieval metrics, faithfulness and answer quality, and how to build the test set.
• AI memory systems: storing and recalling what an assistant has learned about users and tasks over time.
• RAG vs long context vs fine-tuning: choosing between retrieval, very long prompts and training.

Common pitfalls

• Mixing embedding models. Re-embedding documents with a new model but not the queries (or the reverse) silently destroys search quality. Version the model alongside the index.
• Skipping normalization. If the index computes dot products, unnormalized vectors let long vectors win regardless of meaning.
• Tuning on vibes. Changing chunk size or top-$k$ without a labelled retrieval set means you cannot tell whether it helped.
• Ignoring keyword search. Vector-only retrieval routinely misses identifiers, part numbers and names.
• Retrieving too little or too much. Too few candidates and the answer is missing; too many and the relevant passage is diluted. Retrieve broadly, rerank, then pass a focused handful.

In practice

• Start simple: structure-aware chunking, one embedding model, hybrid search, a reranker, and a prompt that requires citations and permits "I don't know."
• Build a small test set of real questions with the passages that should be retrieved, and measure retrieval separately from answer quality.
• Store metadata and enforce permissions in the retrieval step.
• When answers are wrong, look at what was retrieved first; most RAG problems are search problems.