Recommender Systems: Factorization, Two-Tower Retrieval and Evaluation

Recommender systems decide what each person sees next: the next video, product, song, article, job posting or ad. They are among the most commercially important uses of machine learning, because in large catalogs most items are discovered through recommendations rather than search. They are also an unusual ML problem: the model's outputs change the data it will be trained on next, so engineering and ethics are inseparable.

The picture in your head

Picture a giant spreadsheet with one row per user and one column per item. A cell holds a rating, a purchase or a click if the user interacted with that item; almost every cell is empty (a typical user touches far less than 1% of the catalog). Recommendation is the art of filling in the blanks: guessing which empty cells in your row would be bright if you tried those items. Two ways to guess: "people whose rows look like yours liked this" (collaborative filtering) and "this item looks like the items you liked" (content-based). Industrial systems combine both and add a second stage that carefully re-ranks a shortlist.

Feedback: explicit versus implicit

• Explicit feedback is a deliberate judgment: 1–5 stars, thumbs up or down. It is clear in meaning but scarce, and biased toward people who rate and items they chose to consume.
• Implicit feedback is behavior: clicks, plays, watch time, purchases, add-to-cart, skips. It is abundant but ambiguous. A play might be a misclick; a missing interaction usually means "never saw it," not "dislikes it." There are no true negatives, only positives and unknowns.

Most production systems run on implicit feedback. Hu, Koren and Volinsky (2008) formalized a standard treatment: convert raw counts $r_{ui}$ into a binary preference $p_{ui} = 1$ if $r_{ui} > 0$ else $0$, and a confidence $c_{ui} = 1 + \alpha r_{ui}$ that grows with how much the user engaged, then fit factors to the preferences weighted by confidence. Unobserved cells are treated as weak negatives rather than ignored.

Collaborative filtering

Collaborative filtering (CF) uses only the interaction matrix, no item descriptions.

Neighborhood methods. User-based CF finds users similar to you (by cosine similarity or Pearson correlation of their rows) and recommends what they liked. Item-based CF computes item–item similarity from co-interaction ("people who bought this also bought…"), then scores each candidate by its similarity to items in your history. Item-based CF became an industry staple because item similarities are more stable than user similarities and can be precomputed.

Matrix factorization (MF). Assume each user $u$ and item $i$ can be described by $k$ hidden factors, vectors $\mathbf{p}_u, \mathbf{q}_i \in \mathbb{R}^k$, with $k$ much smaller than the number of users or items (often tens to a few hundred). For movies, one factor might loosely capture "action versus drama," another "mainstream versus arthouse," though factors are learned, not named. The predicted rating is

where $\mu$ is the global mean rating, $b_u$ the user's bias (a generous or harsh rater) and $b_i$ the item's bias (broadly liked or not). With $\mathcal{K}$ the set of observed (user, item) pairs, we minimize regularized squared error:

with $\lambda$ the regularization strength. Note the sum covers only observed ratings; filling empty cells with zeros and running a classic SVD would teach the model that unseen means disliked. This formulation, with biases, was popularized by the Netflix Prize era and described by Koren, Bell and Volinsky (2009).

Two standard optimizers:

• Stochastic gradient descent (SGD). For each observed rating compute the error $e_{ui} = r_{ui} - \hat r_{ui}$ and update, with learning rate $\eta$: $b_u \leftarrow b_u + \eta(e_{ui} - \lambda b_u)$, $b_i \leftarrow b_i + \eta(e_{ui} - \lambda b_i)$, $\mathbf{p}_u \leftarrow \mathbf{p}_u + \eta(e_{ui}\mathbf{q}_i - \lambda \mathbf{p}_u)$, $\mathbf{q}_i \leftarrow \mathbf{q}_i + \eta(e_{ui}\mathbf{p}_u - \lambda \mathbf{q}_i)$ (using the old values on the right-hand sides).
• Alternating least squares (ALS). Fix all item vectors and the problem becomes an independent ridge regression per user, solvable in closed form; then fix users and solve for items; alternate. ALS parallelizes well and is the usual choice for the implicit-feedback formulation, where the sum runs over all cells.

Worked example: one SGD step

Let $k = 2$, $\mu = 3.5$, $b_u = 0.2$, $b_i = -0.1$, $\mathbf{p}_u = (0.5, 0.2)$, $\mathbf{q}_i = (0.4, 0.6)$, the true rating $r_{ui} = 5$, $\eta = 0.1$ and $\lambda = 0.05$.

1. Dot product: $0.5 \times 0.4 + 0.2 \times 0.6 = 0.20 + 0.12 = 0.32$.
2. Prediction: $\hat r_{ui} = 3.5 + 0.2 - 0.1 + 0.32 = 3.92$.
3. Error: $e_{ui} = 5 - 3.92 = 1.08$.
4. User bias: $0.2 + 0.1(1.08 - 0.05 \times 0.2) = 0.2 + 0.1(1.07) = 0.307$.
5. Item bias: $-0.1 + 0.1(1.08 - 0.05 \times (-0.1)) = -0.1 + 0.1(1.085) = 0.0085$.
6. User vector: $\mathbf{p}_u + 0.1\big(1.08 (0.4, 0.6) - 0.05 (0.5, 0.2)\big) = (0.5, 0.2) + 0.1\big((0.432, 0.648) - (0.025, 0.010)\big) = (0.5407, 0.2638)$.
7. Item vector: $\mathbf{q}_i + 0.1\big(1.08 (0.5, 0.2) - 0.05 (0.4, 0.6)\big) = (0.4, 0.6) + 0.1\big((0.540, 0.216) - (0.020, 0.030)\big) = (0.452, 0.6186)$.
8. New prediction: $3.5 + 0.307 + 0.0085 + (0.5407 \times 0.452 + 0.2638 \times 0.6186) = 3.8155 + 0.4076 \approx 4.22$.

One step moved the prediction from 3.92 toward 5. Training repeats this over all observed ratings for many epochs.

The following NumPy implementation trains on a toy matrix with two obvious taste groups and fills in the missing cells:

import numpy as np

def train_mf(R, k=2, epochs=200, lr=0.02, lam=0.1, seed=0):
    """R: (n_users, n_items) ratings with np.nan for missing entries."""
    rng = np.random.default_rng(seed)
    n_u, n_i = R.shape
    P, Q = 0.1 * rng.standard_normal((n_u, k)), 0.1 * rng.standard_normal((n_i, k))
    bu, bi = np.zeros(n_u), np.zeros(n_i)
    obs = np.argwhere(~np.isnan(R))                 # observed (user, item) pairs only
    mu = np.nanmean(R)
    for _ in range(epochs):
        rng.shuffle(obs)
        for u, i in obs:
            e = R[u, i] - (mu + bu[u] + bi[i] + P[u] @ Q[i])
            bu[u] += lr * (e - lam * bu[u])
            bi[i] += lr * (e - lam * bi[i])
            P[u], Q[i] = (P[u] + lr * (e * Q[i] - lam * P[u]),
                          Q[i] + lr * (e * P[u] - lam * Q[i]))
    return np.clip(mu + bu[:, None] + bi[None, :] + P @ Q.T, 1, 5)

nan = np.nan
R = np.array([[5, 4, nan, 1, 1],
              [4, nan, 5, 1, 2],
              [1, 1, 2, 5, nan],
              [nan, 2, 1, 4, 5]], dtype=float)
print(np.round(train_mf(R), 1))

Users 0 and 1 like items 0–2 and dislike 3–4; users 2 and 3 are the opposite. The model predicts about 5.0 for user 0 on item 2, about 3.4 for user 1 on item 1, about 4.9 for user 2 on item 4 and about 2.0 for user 3 on item 0 (exact values depend on the random seed), which is what a person would guess from the pattern.

Content-based recommendation

Content-based methods describe items by their attributes (genre, text, price, brand, image embeddings, audio features) and build a profile of each user from the items they engaged with, for example the average of those items' feature vectors. Candidates are ranked by similarity to the profile. Strengths: they can recommend a brand-new item on day one and explain themselves ("because you watched…"). Weaknesses: they over-specialize (more of the same), and they need good item features. Pretrained text, image and audio encoders from the previous lessons now supply those features almost for free.

Industrial architecture: retrieval, then ranking

With hundreds of millions of items you cannot score every item with an expensive model for every request within a ~100 ms budget. Production systems therefore run a funnel, described publicly in YouTube's two-stage design (Covington, Adams and Sargin, 2016):

1. Candidate generation (retrieval). Cheaply select a few hundred to a few thousand plausible items from the whole catalog, usually from several sources merged together: embedding retrieval, item-to-item CF, trending items, items from followed creators.
2. Ranking. Score each candidate with a rich model that uses many features (user history, item features, context such as time and device, cross features) and predicts one or more outcomes (click, watch time, purchase, satisfaction surveys). Gradient-boosted trees and deep networks are both common.
3. Re-ranking and business rules. Enforce diversity, freshness, deduplication, eligibility, fairness and policy constraints, then return the final list.

Two-tower retrieval

The standard retrieval model is the two-tower (dual-encoder) network: a user tower maps user features and history to an embedding $\mathbf{u} \in \mathbb{R}^d$, an item tower maps item features to $\mathbf{v} \in \mathbb{R}^d$, and relevance is their dot product $s(u, i) = \mathbf{u}^\top \mathbf{v}_i$. Because the towers do not interact until the final dot product, all item embeddings can be precomputed and stored in an approximate nearest-neighbor (ANN) index (the same vector-search technology used for RAG); at request time you compute one user embedding and retrieve the top items in milliseconds. Matrix factorization is the special case where each tower is just an embedding lookup.

Training typically uses observed (user, positive item) pairs and in-batch negatives: the other items in the mini-batch serve as negatives, and a softmax over the batch is trained with cross-entropy. Popular items appear as negatives more often, which biases the model against them, so practical systems apply a correction such as subtracting the log of each item's sampling probability from its logit (the approach described by Yi et al., 2019).

import torch, torch.nn as nn, torch.nn.functional as F

class Tower(nn.Module):
    def __init__(self, n_ids, dim=64):
        super().__init__()
        self.emb = nn.Embedding(n_ids, dim)
        self.mlp = nn.Sequential(nn.Linear(dim, dim), nn.ReLU(), nn.Linear(dim, dim))
    def forward(self, ids):
        return F.normalize(self.mlp(self.emb(ids)), dim=-1)   # unit-length embeddings

user_tower, item_tower = Tower(n_ids=10_000), Tower(n_ids=50_000)
opt = torch.optim.Adam(list(user_tower.parameters()) + list(item_tower.parameters()), lr=1e-3)

def train_step(user_ids, pos_item_ids, temperature=0.05):
    u = user_tower(user_ids)                    # (B, d)
    v = item_tower(pos_item_ids)                # (B, d)
    logits = u @ v.T / temperature              # (B, B): row b's positive is column b
    labels = torch.arange(len(user_ids))        # the other B-1 items are in-batch negatives
    loss = F.cross_entropy(logits, labels)
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

print(train_step(torch.randint(0, 10_000, (256,)), torch.randint(0, 50_000, (256,))))

Real towers take many features (IDs plus categorical and text features, recent-history sequences), not only an ID, and duplicates of the same item within a batch should be masked.

Cold start

New users and new items have no interactions, so pure CF cannot place them. Standard remedies:

• New items: content features in the item tower (so a new item gets a sensible embedding from its title, image and category), plus deliberate exploration traffic to collect early feedback.
• New users: onboarding questions, context (location, device, referral source), popularity within a segment, and fast session-based models that adapt within the first few clicks.
• Bandits: treat recommendation as explore–exploit (for example Thompson sampling or upper-confidence-bound methods), giving uncertain items some exposure to learn their value.

Offline evaluation

Hold out interactions, ideally the latest ones for each user (a time-based split, since random splits leak the future), and check whether the model ranks them highly. For a ranked list of length $k$ and a user with a set of relevant held-out items:

• Precision@k = (relevant items in the top $k$) ÷ $k$.
• Recall@k = (relevant items in the top $k$) ÷ (total relevant items for that user).
• NDCG@k (normalized discounted cumulative gain, Järvelin and Kekäläinen, 2002) rewards placing the most relevant items near the top. With graded relevance $\text{rel}_j$ at rank $j$,

where IDCG@k is the DCG of the ideal ordering, so NDCG lies in $[0, 1]$. (Some libraries use the linear gain $\text{rel}_j$ instead of $2^{\text{rel}_j} - 1$; state which you use.)

Worked example (precision and recall). A user has 4 relevant held-out items. The top 5 recommendations contain relevant items at ranks 1, 3 and 5. Precision@5 $= 3/5 = 0.60$; recall@5 $= 3/4 = 0.75$.

Worked example (NDCG). The top 5 items have graded relevance $(3, 2, 0, 1, 2)$ and these are all the relevant items for the user.

DCG@5 $= 10.484$. The ideal order is $(3, 2, 2, 1, 0)$: $7 + 3(0.631) + 3(0.500) + 1(0.431) + 0 = 10.823$. NDCG@5 $= 10.484 / 10.823 = 0.969$. The list is nearly ideal; it loses a little because a relevance-2 item sits at rank 5 behind a non-relevant item.

import numpy as np

def dcg(rels):
    rels = np.asarray(rels, dtype=float)
    return float(np.sum((2 ** rels - 1) / np.log2(np.arange(2, len(rels) + 2))))

def ndcg_at_k(ranked_rels, all_rels, k):
    ideal = sorted(all_rels, reverse=True)[:k]
    return dcg(ranked_rels[:k]) / dcg(ideal) if dcg(ideal) > 0 else 0.0

print(round(ndcg_at_k([3, 2, 0, 1, 2], all_rels=[3, 2, 2, 1], k=5), 3))   # 0.969

Also track coverage (what fraction of the catalog ever gets recommended), diversity and novelty; a model can score well on NDCG by recommending only blockbusters.

Online A/B testing

Offline metrics are computed on logs produced by the old system, so they only partly predict live behavior; the final judge is a randomized online controlled experiment (A/B test). Users are randomly assigned to control (current system) or treatment (new model), and you compare a pre-registered primary metric (for example, long-term engagement or purchases per user) plus guardrail metrics (latency, complaints, unsubscribes, revenue). Good practice, as laid out by Kohavi, Tang and Xu (2020): randomize by user, not by request; compute the sample size needed for the effect you care about before starting; run for full weekly cycles; beware novelty effects; check that the observed traffic split matches the intended one (a sample ratio mismatch signals a bug); and do not stop the test the moment results look significant ("peeking" inflates false positives unless you use sequential-testing methods).

Feedback loops and filter bubbles

A recommender trains on interactions that it caused. Items it shows get clicks; items it hides get none, so the next model is even more confident they are bad. This feedback loop produces popularity bias (the rich get richer), position bias (top slots get clicked because they are on top) and, for users, filter bubbles, where exposure narrows toward what they already consume. Optimizing short-term clicks can also reward sensational or low-quality content. Mitigations include logging the propensity (probability) with which each item was shown and using inverse-propensity weighting during training and evaluation, reserving a small amount of randomized exploration traffic, explicit diversity and freshness constraints in re-ranking, optimizing longer-term or survey-based satisfaction targets, and giving users controls ("not interested," topic preferences). Regulators also care: the EU Digital Services Act requires online platforms to set out the main parameters of their recommender systems in their terms and conditions (Article 27), and requires very large online platforms and search engines to offer at least one recommender option not based on profiling (Article 38).

Common pitfalls

• Random train/test splits that let the model see the future; split by time.
• Treating missing as negative in explicit-rating MF, or ignoring negatives entirely with implicit data.
• Evaluating with sampled negatives (ranking the true item among 100 random items) and reporting the result as if it were full-catalog ranking; sampled metrics can reorder models.
• Popularity baseline neglected. Always compare against "most popular" and "recently popular"; they are surprisingly hard to beat.
• Training–serving skew: features computed differently offline and online, or the retrieval index built from stale embeddings.
• Optimizing the wrong objective: click-through rate rewards clickbait; choose targets that reflect user value.

In practice

Teams start with strong baselines (popularity, item-to-item CF, ALS on implicit feedback), then add a two-tower retrieval model with content features to handle cold start and a feature-rich ranker trained on multiple objectives. They evaluate offline with time-split recall@k and NDCG@k plus coverage and diversity, ship only through A/B tests with guardrails, and keep a small exploration budget to fight feedback loops. Infrastructure matters as much as models: a feature store for consistent online and offline features, an ANN index refreshed as embeddings change, logging of what was shown (not only what was clicked) with propensities, and dashboards that slice metrics by new versus returning users and by item popularity tier. Increasingly, sequence models (transformers over a user's recent interaction history) and LLM-generated item features are added to the same funnel rather than replacing it.