The LLM Era: Transformers to Agents (2017–2026)

The years since 2017 are the most compressed period in AI history. A single architecture, the transformer, scaled from a translation model to systems that write code, pass professional exams, see images, reason through multi-step problems and operate software tools. Language models moved from research curiosity to a product with hundreds of millions of weekly users. This lesson gives you an accurate map of that period, verified as of November 2026, and then steps back to ask what seventy years of AI history teach about where the field goes next.

Intuition: one recipe, scaled

The LLM era runs on one recipe with three steps:

1. Pre-train a large transformer to predict the next token (a word or word piece) on a huge body of text. To predict well, the model must absorb grammar, facts, styles and some reasoning patterns.
2. Post-train it to follow instructions and be helpful and safe, using curated examples and human or AI feedback.
3. Scale model size, data and compute together, because performance improves predictably when you do.

Most of the milestones below are refinements of one of these steps, or new ways to use the result (tools, images, longer reasoning).

Timeline

The transformer (2017)

Recurrent networks (lesson 5) process a sentence one token at a time, which is slow to train because each step waits for the previous one, and they struggle to carry information across long distances. In June 2017 eight Google researchers (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin) published "Attention Is All You Need." Their transformer dropped recurrence entirely and used self-attention: every token computes a relevance-weighted mix of every other token's representation, all in parallel. The core operation is

where $Q$, $K$ and $V$ are matrices of query, key and value vectors computed from the tokens, $d_k$ is the key dimension, and softmax turns each row of scores into weights that sum to 1. Because every position is computed at once, transformers train efficiently on GPUs, and attention connects distant tokens directly. The paper set new translation records; its greater impact was that the architecture scaled. (The full mechanics are in the AI 210 lesson on transformers.)

Pre-training: GPT and BERT (2018–2019)

Two families appeared in 2018:

• GPT (OpenAI, June 2018; Alec Radford and colleagues) pre-trained a transformer decoder to predict the next token, then fine-tuned it for specific tasks. Generative, left-to-right.
• BERT (Google, October 2018; Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova) pre-trained a transformer encoder by masking words and predicting them from both sides. It set records across many language-understanding benchmarks and was deployed in Google Search in 2019.

For a few years BERT-style encoders dominated classification and search tasks. But generative pre-training scaled more naturally into general-purpose systems. GPT-2 (February 2019, 1.5 billion parameters) generated coherent paragraphs, and OpenAI initially withheld the full model, citing misuse concerns, before releasing it in stages through November 2019, the first major public debate about releasing model weights. GPT-3 (May 2020, 175 billion parameters, trained on about 300 billion tokens) showed in-context learning: given a few examples in the prompt, it could perform new tasks without any weight updates. Prompting became a way of programming.

Scaling laws (2020–2022)

Jared Kaplan and colleagues at OpenAI ("Scaling Laws for Neural Language Models," January 2020) found that test loss falls as a smooth power law in model size, dataset size and compute, over many orders of magnitude. For model size $N$ (parameters, with enough data), loss behaves roughly as

where $N_c$ is a fitted constant and $\alpha_N \approx 0.076$ in their fits. This turned AI progress into something like an engineering forecast: spend more compute, get a predictable improvement.

DeepMind's Chinchilla paper (Hoffmann et al., March 2022) corrected the recipe: for a fixed compute budget, parameters and training tokens should grow in roughly equal proportion, and many large models had been under-trained. Their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, outperformed much larger models. A widely used rule of thumb drawn from this work is about 20 training tokens per parameter for compute-optimal training (in practice, models meant for cheap inference are now trained on far more tokens than that).

Worked example: estimating training compute

A standard approximation for transformer training compute is

FLOPs (floating-point operations), where $N$ is the number of parameters and $D$ the number of training tokens (about 2 FLOPs per parameter per token forward, 4 backward).

• GPT-3: $6 \times 1.75\times10^{11} \times 3\times10^{11} = 3.15 \times 10^{23}$ FLOPs.
• Chinchilla: $6 \times 7\times10^{10} \times 1.4\times10^{12} = 5.88 \times 10^{23}$ FLOPs.

Chinchilla used about 1.9× GPT-3's compute with 40% of its parameters, spending the difference on almost five times as much data. Tokens per parameter: GPT-3 about $3\times10^{11}/1.75\times10^{11} \approx 1.7$; Chinchilla $1.4\times10^{12}/7\times10^{10} = 20$.

def train_flops(params, tokens):
    return 6 * params * tokens

for name, n, d in [("GPT-3", 175e9, 300e9), ("Chinchilla", 70e9, 1.4e12)]:
    c = train_flops(n, d)
    print(f"{name:10s} {c:.2e} FLOPs, {d / n:.1f} tokens/param")

# Power-law effect of doubling model size (alpha_N ~ 0.076):
print(f"loss ratio when N doubles: {2 ** -0.076:.3f}")  # ~0.949

Doubling model size cuts loss by only about 5% under this fit. Power laws give reliable but diminishing returns per doubling, which is why frontier training budgets grew by orders of magnitude.

From predictor to assistant: instruction tuning and RLHF

A raw pre-trained model continues text; it does not reliably follow instructions. Two techniques changed that:

• Instruction tuning: fine-tune on many tasks phrased as instructions with good responses (for example Google's FLAN work, 2021).
• Reinforcement learning from human feedback (RLHF): humans compare pairs of model outputs; a reward model learns to predict their preferences; the language model is then optimized with reinforcement learning to produce outputs the reward model scores highly. The method came from Paul Christiano and colleagues (2017), was applied to summarization by OpenAI in 2020, and to instruction following in InstructGPT (Ouyang et al., March 2022), where labellers preferred outputs of a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3.

Variants followed: Anthropic's Constitutional AI (December 2022) used a written set of principles and AI feedback to reduce reliance on human labels, and Direct Preference Optimization (DPO, 2023) learned from preference pairs without a separate reinforcement-learning loop.

ChatGPT and the adoption shock (2022–2023)

OpenAI released ChatGPT, a chat interface to an RLHF-tuned GPT-3.5 model, on 30 November 2022 as a "research preview." An analyst estimate from UBS, based on Similarweb data, put it at about 100 million monthly active users in January 2023, which would make it the fastest-growing consumer application to that point. In October 2025, OpenAI's chief executive Sam Altman said ChatGPT had about 800 million weekly active users. The technology was not new (GPT-3 had existed for over two years); the interface and post-training were. A free, conversational product made capabilities visible to the public, and competitors, investors and regulators responded within months.

GPT-4 (14 March 2023) accepted images as well as text and scored highly on professional and academic exams in OpenAI's technical report, which, notably, disclosed no details of model size, data or training compute, a sign of the shift from open research to competitive secrecy at the frontier.

The open-weight movement

In parallel, a movement grew to release model weights so anyone could run, study and fine-tune models:

• LLaMA (Meta, February 2023) was released to researchers under a non-commercial licence; its weights leaked publicly within about a week, and a wave of community fine-tunes followed. Llama 2 (July 2023) came with a licence permitting most commercial use.
• Mistral 7B (September 2023) was released under the permissive Apache 2.0 licence by the French start-up Mistral AI, followed by the mixture-of-experts Mixtral.
• Qwen (Alibaba, open releases from 2023) and DeepSeek became leading Chinese open-weight families. DeepSeek-R1 (20 January 2025, MIT licence) matched leading proprietary reasoning models on several benchmarks according to its developers, and its reported training efficiency briefly shook technology stock markets.
• gpt-oss (OpenAI, 5 August 2025), two models under Apache 2.0, marked OpenAI's first open-weight language models since GPT-2.

"Open weights" is not the same as "open source": training data and code are often not released, and licences vary (see the AI 300 lesson on open weights and licensing). Open weights made local AI practical, and they made the frontier gap between proprietary and open models a recurring news story.

Multimodal models

Models learned to handle images, audio and video alongside text. OpenAI's CLIP (2021) learned a shared embedding space for images and captions; text-to-image systems (DALL·E, Stable Diffusion, Midjourney) arrived in 2021–2022; GPT-4 accepted images in 2023; Google's Gemini (December 2023) was designed as multimodal from the start; GPT-4o (May 2024) handled real-time voice conversation. As of November 2026, frontier models commonly accept text, images, audio and documents in one context, and many generate images or speech.

Reasoning models (2024–)

Chain-of-thought prompting (Wei et al., 2022) showed that asking a model to write intermediate steps improves multi-step problem solving. OpenAI's o1 (12 September 2024) trained models with reinforcement learning to produce long internal reasoning before answering, and showed accuracy rising with more test-time compute (thinking longer). DeepSeek-R1 openly described a similar approach. In July 2025, Google DeepMind reported that an advanced Gemini Deep Think model achieved a gold-medal standard at the International Mathematical Olympiad, scoring 35 of 42 points with natural-language proofs written within the time limit and graded by IMO coordinators; OpenAI announced a similar 35-point result for an experimental model, graded by three former IMO medalists rather than through the official IMO coordination process. Note the historical echo: AlphaGo combined learning with search, and reasoning models combine learned language models with a form of search over their own reasoning at inference time.

Agents and MCP (2023–2026)

An agent is a model placed in a loop where it plans, calls tools (search, code execution, APIs, a browser or a desktop), observes results and continues until a goal is met. Research on ReAct (2022) and Toolformer (2023), and vendor function-calling APIs (2023), made tool use routine; coding agents and computer-use agents followed in 2024–2025.

Integrations multiplied, and every application needed custom connectors for every model. Anthropic introduced the Model Context Protocol (MCP) on 25 November 2024: an open protocol, built on JSON-RPC 2.0, through which AI applications connect to servers exposing tools, resources and prompts. OpenAI adopted MCP in March 2025, and other major providers followed. On 9 December 2025 the Linux Foundation announced the Agentic AI Foundation (AAIF), anchored by three contributed projects: MCP (from Anthropic), goose (an agent framework from Block) and AGENTS.md (a convention for instructions to coding agents, from OpenAI). Its platinum members at launch were Amazon Web Services, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft and OpenAI. Handing a protocol to a neutral foundation mirrors how earlier internet and cloud standards matured. As of November 2026, MCP is widely supported across major AI providers and developer tools, though agent reliability, security (for example prompt injection through tool outputs) and evaluation remain open problems.

Analysis: recurring patterns across AI history

Looking back across all six lessons, five patterns recur.

1. Hype cycles

Every era has followed a cycle of real breakthrough, inflated promise, disappointment and recalibration: the 1955 proposal's single summer; the 1958 perceptron press coverage; the 1980s expert-systems boom and the Lisp machine crash; the 2010s claims about imminent self-driving cars. The current era shows the same signs (very large investment, sweeping timelines), alongside genuinely broad adoption that earlier booms never achieved. History does not say whether a downturn will come; it says that capability and adoption are different from promises, and that the gap between them is what triggers winters.

2. Compute

Progress repeatedly followed cheaper computation: stored-program computers made AI possible in the 1950s; commodity workstations killed Lisp machines in 1987; GPUs enabled AlexNet in 2012; massive accelerator clusters enabled GPT-3 and beyond. Richard Sutton's 2019 essay "The Bitter Lesson" argued that general methods that exploit growing computation (search and learning) eventually beat methods built on human knowledge. The LLM era is the strongest evidence for his thesis, and it raises new questions about energy use and who can afford frontier research.

3. Data

Each paradigm moved the source of knowledge further from human experts and closer to data: rules written by experts (1970s–80s), parameters estimated from labelled data (1990s–2000s), features learned from large labelled datasets (2010s), and knowledge absorbed from most of the public web (2020s). The knowledge acquisition bottleneck of lesson 2 was largely solved by data, but it created new problems: copyright disputes, privacy, bias absorbed from text, and concern that high-quality human-written data is finite.

4. Benchmarks

Benchmarks have repeatedly decided arguments: NIST speech evaluations, ILSVRC in 2012, CASP in 2018–2020, and today's exam, coding and maths evaluations. They also saturate and get overfitted, and success on a benchmark has repeatedly overstated real-world reliability. Good practice is to treat benchmarks as evidence, not proof, and to evaluate on your own task.

5. Old ideas, new scale

Neural networks (1943–1958), backpropagation (1970–1986), convolution (1980–1998), reinforcement learning from self-play (1992), attention (2014) and even the dream of a learning "child machine" (Turing, 1950) were all old before they were triumphant. Ideas that fail at one scale can succeed at another, so dismissing an approach because it failed before is risky, and so is assuming that scaling alone will solve every remaining problem.

Common pitfalls

• Dating ChatGPT's capabilities to November 2022. The underlying models existed earlier; the product and post-training made them usable.
• Equating open weights with open source. Check the licence and what was actually released.
• Treating scaling laws as laws of nature. They are empirical fits over a range; they predict loss, not specific capabilities, and they can bend.
• Repeating unverified recent claims. For 2025–2026 events, cite primary sources (company announcements, papers, foundation press releases) and date your statements.

In practice

As a practitioner in 2026: build on the pre-train/post-train/scale recipe rather than fighting it; choose between proprietary and open-weight models on capability, cost, privacy and licence; expose tools through standard protocols such as MCP instead of bespoke integrations; and evaluate agents on your own tasks, with attention to security. Above all, read announcements as a historian would: ask what was measured, against what baseline, at what cost, and what the system still cannot do.