Pretraining and Data: How a Base Model Is Made

Every capability you see in a modern chat model starts with pretraining: months of next-token prediction over trillions of tokens of text and code. Pretraining is where almost all of the compute budget goes and where the model acquires its vocabulary, world knowledge, coding ability and the raw material for reasoning. If you understand the objective, the data pipeline and the arithmetic of training at scale, you can read a model's technical report critically, estimate what a training run costs, and make sensible decisions when you continue-pretrain or fine-tune a model yourself.

Intuition: compressing a library by predicting the next word

Imagine being handed a library of a few billion web pages, books and code repositories and told to write the shortest possible encoding of all of it. The best way to compress text is to predict it well: if you can guess the next character with high confidence, you need very few bits to encode what actually comes next. A language model is a learned predictor of this kind. To predict the next token of a Python file it has to "know" syntax; to predict the next token of a chemistry paper it has to absorb some chemistry; to predict the end of a proof it has to follow the argument. Nobody labels any of this. The supervision is the text itself, which is why the method is called self-supervised learning and why it scales to the whole web.

This picture also explains why data quality matters so much. A predictor trained on spam, boilerplate and duplicated pages spends capacity memorising junk. A predictor trained on carefully filtered, diverse, educational text learns more per token.

The pretraining objective

A tokenizer turns text into a sequence of integer token ids $x_1, x_2, \dots, x_T$ drawn from a vocabulary $V$. A causal (decoder-only) transformer with parameters $\theta$ outputs, at each position $t$, a probability distribution $p_\theta(\cdot \mid x_{<t})$ over the next token. Training minimises the average negative log-likelihood, which is the cross-entropy between the one-hot "true next token" and the model's distribution:

$\mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})$

Here $T$ is the number of predicted positions in the batch, and $x_{<t}$ means all tokens before position $t$. The model produces a vector of logits $z \in \mathbb{R}^{|V|}$ and the probability of token $k$ is the softmax $p_k = e^{z_k} / \sum_j e^{z_j}$. Two derived quantities show up everywhere:

• Perplexity $= e^{\mathcal{L}}$ (with natural logs): the effective number of tokens the model is choosing between. A loss of 2.0 nats/token is a perplexity of about 7.4.
• Bits per byte: loss converted to bits ($\div \ln 2$) and normalised by the number of UTF-8 bytes rather than tokens. Because it does not depend on the tokenizer, it is the fair way to compare models with different vocabularies.

All positions in a sequence are trained in parallel thanks to the causal attention mask, so one forward pass over a 4,096-token sequence gives 4,095 training signals. Documents are usually packed: concatenated with an end-of-document token and cut into fixed-length windows so no compute is wasted on padding.

Worked example: one position's loss

Suppose the vocabulary has four tokens and at some position the logits are $z = (2.0, 1.0, 0.1, -1.0)$, with the true next token being index 0.

1. Exponentiate: $e^{2.0} = 7.389$, $e^{1.0} = 2.718$, $e^{0.1} = 1.105$, $e^{-1.0} = 0.368$.
2. Sum: $7.389 + 2.718 + 1.105 + 0.368 = 11.580$.
3. Probability of the true token: $7.389 / 11.580 = 0.638$.
4. Loss: $-\ln 0.638 = 0.449$ nats.

If the true token had been index 3, the loss would be $-\ln(0.368/11.580) = -\ln 0.0318 = 3.45$ nats. Confident wrong predictions are punished hard, which is exactly the pressure that forces the model to learn.

Tokenizer training with BPE

Before pretraining you must train a tokenizer, and it is frozen for the life of the model. Most LLMs use byte-pair encoding (BPE), introduced for neural machine translation by Sennrich, Haddow and Birch (2016), or a close relative (WordPiece, Unigram via SentencePiece). Byte-level BPE, popularised by GPT-2, starts from the 256 possible bytes so that any string can be encoded and nothing is ever "unknown".

The BPE training algorithm:

1. Split the training corpus into words (a regex pre-tokenizer separates spaces, punctuation and digits) and represent each word as a sequence of bytes.
2. Count every adjacent pair of symbols across the corpus, weighted by word frequency.
3. Merge the most frequent pair into a new symbol and record the merge rule.
4. Repeat until the vocabulary reaches the target size (commonly 32k to 256k tokens).

At encoding time the merges are applied in the order they were learned. The resulting vocabulary size is a real trade-off: a larger vocabulary shortens sequences (cheaper attention, more text per context window) but enlarges the embedding and output matrices and leaves rare tokens under-trained. Tokenizers trained mostly on English also spend more tokens per word on other languages, which raises cost and reduces effective context for those users. Digits are often split into single characters or groups of up to three, because inconsistent number tokenization measurably hurts arithmetic.

from collections import Counter

def train_bpe(words, num_merges):
    # words: dict mapping a word (tuple of symbols) to its corpus count
    vocab = dict(words)
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for word, freq in vocab.items():
            for a, b in zip(word, word[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = max(pairs, key=pairs.get)
        merges.append(best)
        new_vocab = {}
        for word, freq in vocab.items():
            out, i = [], 0
            while i < len(word):
                if i < len(word) - 1 and (word[i], word[i + 1]) == best:
                    out.append(word[i] + word[i + 1]); i += 2
                else:
                    out.append(word[i]); i += 1
            new_vocab[tuple(out)] = new_vocab.get(tuple(out), 0) + freq
        vocab = new_vocab
    return merges, vocab

corpus = {tuple("low"): 5, tuple("lower"): 2, tuple("newest"): 6, tuple("widest"): 3}
merges, vocab = train_bpe(corpus, 4)
print(merges)  # [('e','s'), ('es','t'), ('l','o'), ('lo','w')]

This toy works on characters; production tokenizers (for example the Hugging Face tokenizers library) do the same thing on bytes, in Rust, over hundreds of gigabytes.

Data sources and the pipeline

Where the tokens come from

• Web crawl. The backbone of nearly every open pretraining corpus is Common Crawl, a US 501(c)(3) nonprofit founded in 2007 that has collected web data since 2008. Its archive runs to petabytes, a single crawl typically covers billions of pages, and new crawls are released regularly. Raw crawl is mostly unusable: navigation menus, spam, near-duplicates, machine-generated pages and many languages mixed together.
• Curated text: books, encyclopedias, academic papers, patents, legal and government documents, forums and Q&A sites.
• Code: public repositories, deduplicated and license-filtered, plus issues, notebooks and documentation.
• Math and science: arXiv-style papers and math web pages extracted with their LaTeX intact. DeepSeekMath (Shao et al., 2024), for example, mined about 120 billion math tokens from Common Crawl with an iteratively trained classifier.
• Synthetic data: text generated by other models, increasingly used for reasoning and instruction-like content (covered in the distillation lesson).

A reference pipeline: FineWeb

The best documented open example is FineWeb (Penedo et al., NeurIPS 2024 Datasets and Benchmarks), a 15-trillion-token English corpus built from 96 Common Crawl snapshots and released under the ODC-By 1.0 license. Its steps are a good template:

1. Text extraction from the raw WARC HTML with the trafilatura library rather than using Common Crawl's pre-extracted WET text, which kept more menus and boilerplate.
2. URL filtering against a blocklist of adult domains.
3. Language identification with a fastText classifier, keeping English with score at least 0.65.
4. Quality heuristics: the MassiveText repetition and quality rules (from the Gopher work), most of the C4 heuristics, and some custom line-level rules (for example, dropping documents where too many lines are very short or duplicated).
5. Deduplication with MinHash on word 5-grams, applied within each snapshot. Interestingly, the authors found that deduplicating globally across all snapshots gave worse downstream results than per-snapshot deduplication, because it disproportionately kept lower-quality leftovers from older crawls.
6. PII handling: email addresses and public IP addresses are anonymised.

A companion set, FineWeb-Edu (1.3 trillion tokens), adds a model-based quality filter: Llama-3-70B-Instruct rated several hundred thousand pages for educational value on a 0 to 5 scale, a small regression head on top of a text-embedding model was trained on those ratings, and only pages scoring 3 or more were kept. Models trained on it improve markedly on knowledge and reasoning benchmarks for the same token budget. This "LLM-as-annotator, small classifier at scale" pattern is now standard.

Deduplication in more detail

Exact deduplication (hashing whole documents or long substrings) is cheap but misses near-duplicates such as the same article with different ads. MinHash LSH estimates the Jaccard similarity of two documents' shingle sets: for each of $k$ random hash functions, keep the minimum hash value over the document's shingles; the probability that two documents share a minimum equals their Jaccard similarity. Grouping hashes into bands and flagging documents that collide in any band finds candidate duplicates in near-linear time. Duplicates matter because they waste compute, skew the data mixture toward whatever is copied most, and increase verbatim memorisation (a privacy and copyright risk).

Toxicity and safety filtering

Teams remove or down-weight hateful, sexual and violent content with blocklists and classifiers, and remove known personal data. Filtering is a trade-off: aggressive toxicity classifiers disproportionately remove dialects and minority-group discussions, and a model that has never seen toxic text can be worse at recognising it. Many teams therefore filter the worst content, keep some borderline material, and rely on post-training for behaviour.

Data mixtures and annealing

The final corpus is a weighted mixture of sources, for example a large share of filtered web, plus code, math, books, multilingual text and reference material. Weights are set with small proxy-model ablations: train 1B-parameter models for a few billion tokens on candidate mixtures and compare benchmark scores. High-quality sources are often upsampled (seen two to four times) while web text is seen about once. Many recent recipes end with an annealing phase: as the learning rate decays to zero, the mixture shifts toward the highest-quality math, code and curated data, which gives a disproportionate boost for the tokens spent. Benchmark test sets must be decontaminated out of the corpus by n-gram matching, or later evaluations become meaningless.

Training at scale

A model with tens of billions of parameters does not fit on one accelerator once you add optimizer state. With mixed precision and Adam, each parameter typically costs about 16 bytes of training state: 2 bytes for BF16 weights, 2 for BF16 gradients, and 12 for the FP32 master weights and the two Adam moments. A 70B model therefore needs roughly $70\times10^9 \times 16 \approx 1.1$ TB before counting activations. Four parallelism strategies, usually combined (so-called 3D or 4D parallelism), solve this:

ZeRO (Rajbhandari et al., 2019) observed that plain data parallelism stores redundant copies of the optimizer state on every GPU. Stage 1 shards optimizer state, stage 2 also shards gradients, stage 3 also shards parameters. PyTorch's Fully Sharded Data Parallel (FSDP) implements the stage-3 idea natively. Tensor parallelism is bandwidth-hungry, so it stays inside a node; pipeline parallelism introduces "bubbles" of idle time that micro-batching reduces. The headline efficiency metric is model FLOPs utilisation (MFU): achieved useful FLOPs divided by the hardware's peak. Well-tuned dense runs often reach roughly 35 to 50 percent.

The compute estimate $C \approx 6ND$

For a dense transformer with $N$ parameters trained on $D$ tokens, total training compute is approximately

$C \approx 6ND \ \text{FLOPs}$

The 6 comes from about $2N$ FLOPs per token for the forward pass (one multiply and one add per weight) and about $4N$ for the backward pass (gradients with respect to both activations and weights). The approximation ignores the attention-score computation, which grows with context length and is modest at typical lengths. Kaplan et al. (2020) wrote it as $C \approx 6NBS$ with batch size $B$ and steps $S$, so $D = BS$. Sanity check with published numbers: Chinchilla had $N = 70\times10^9$ and $D = 1.4\times10^{12}$, so $6ND = 5.88\times10^{23}$ FLOPs, within about 2 percent of the $5.76\times10^{23}$ reported by Hoffmann et al. (2022).

Worked example: what does a 7B model on 2T tokens cost?

Assumptions (state yours explicitly when you do this for real): dense 7B model, 2 trillion tokens, H100 GPUs with about 989 TFLOP/s dense BF16 peak, 40 percent MFU, and an illustrative rental price of 2 US dollars per GPU-hour.

1. Compute: $C = 6 \times 7\times10^9 \times 2\times10^{12} = 8.4\times10^{22}$ FLOPs.
2. Effective throughput per GPU: $0.40 \times 989\times10^{12} = 3.956\times10^{14}$ FLOP/s.
3. GPU-seconds: $8.4\times10^{22} / 3.956\times10^{14} = 2.12\times10^{8}$ s.
4. GPU-hours: $2.12\times10^{8} / 3600 \approx 59{,}000$.
5. Wall clock on 1,024 GPUs: $59{,}000 / 1{,}024 \approx 57.6$ hours, about 2.4 days.
6. Cost: about 59,000 GPU-hours at 2 US dollars each, roughly 118,000 US dollars for the final run.

The final run is only part of the bill. Data processing, tokenizer and mixture ablations, failed runs, evaluation and the people involved routinely multiply it several times. Frontier runs use roughly two to four orders of magnitude more compute.

Training instabilities and checkpoints

Large runs fail in recognisable ways:

• Loss spikes and divergence. The loss jumps suddenly and may never recover. Causes include bad data batches, too high a learning rate, and attention logits growing without bound. Mitigations: learning-rate warm-up then cosine or warmup-stable-decay schedules, gradient-norm clipping (commonly at 1.0), AdamW with tuned $\beta_2$ and $\epsilon$, QK-normalisation of attention, an auxiliary z-loss that keeps the softmax normaliser near 1, and keeping sensitive operations such as softmax and normalisation in FP32. The PaLM team reported handling spikes by restarting from a checkpoint shortly before the spike and skipping the data batches around it.
• Hardware failures. On thousands of GPUs, a node failure every few hours is normal. Runs need automatic detection, elastic restart, and frequent checkpoints of model weights, optimizer state, the data-loader position and random-number-generator state. Without the optimizer state and data position you cannot resume the same run exactly.
• Silent data corruption. Faulty chips occasionally produce wrong numbers without crashing; teams watch for anomalous gradient norms per rank.

Checkpoints also serve science: intermediate checkpoints let you measure learning dynamics and serve as starting points for continued pretraining.

Common pitfalls

• Comparing perplexity across tokenizers. Per-token loss is not comparable between vocabularies; use bits per byte or downstream tasks.
• Benchmark contamination. Test questions leak into crawls; decontaminate before training and report how.
• Over-deduplicating or over-filtering. Very aggressive filters can leave too few tokens, remove whole dialects, or skew topics.
• Forgetting the optimizer in memory estimates. Weights alone are about one eighth of training memory with Adam in mixed precision.
• Treating $6ND$ as exact. It ignores attention FLOPs, recomputation for activation checkpointing (which adds roughly another forward pass), and MFU losses.
• Changing the tokenizer after training. Adding tokens requires new embeddings that start untrained; plan the vocabulary up front.
• Saving only weights. You cannot resume a run faithfully without optimizer state and data-loader position.

In practice

Very few organisations pretrain frontier models; many more do continued pretraining on domain data (legal, biomedical, a new language) or train small models from scratch for edge deployment. The workflow is the same at any scale: build and inspect the data pipeline first, run small proxy models to choose the mixture, estimate compute with $6ND$ and a realistic MFU, use a mature training framework (Megatron-LM, DeepSpeed, PyTorch FSDP, or higher-level wrappers) rather than writing parallelism yourself, checkpoint often, and monitor loss, gradient norm and per-domain validation loss continuously. For learning, the Hugging Face datasets library can stream FineWeb samples, and small GPT-style training scripts reproduce the whole loop on a single GPU in hours. Record data provenance and licenses as you go: downstream users and regulators increasingly ask what a model was trained on.