Connectionism, Backpropagation and the AI Winters

Neural networks are the foundation of almost every AI system you use today, yet for most of AI's first fifty years they were a minority interest, at times close to a career risk. This lesson tells the story of connectionism (the idea that intelligence emerges from many simple connected units that learn by adjusting connection strengths), of the books, reports and markets that twice froze AI funding, and of the handful of researchers who kept neural networks alive. The story matters to practitioners because the same forces, inflated promises, unmet benchmarks and hardware economics, still move money in AI.

Intuition: knobs instead of rules

Symbolic AI wrote knowledge down as rules. Connectionism stores knowledge in weights: thousands or millions of numerical knobs on the connections between simple units. Nobody sets the knobs by hand. Instead the system sees an example, compares its output with the right answer, and nudges each knob a little in the direction that would have reduced the error. Learning is the accumulation of many small nudges. The history of connectionism is largely the history of figuring out how to compute the right nudge for knobs buried deep inside a network.

Timeline

The perceptron (1957–58)

Frank Rosenblatt, a psychologist at the Cornell Aeronautical Laboratory, simulated the perceptron on an IBM 704 in 1957 and described it in "The perceptron: a probabilistic model for information storage and organization in the brain" in Psychological Review (1958). The Mark I Perceptron, a custom machine with a 20×20 array of photocells connected through randomly wired units to motor-driven potentiometers that stored the weights, was built in 1959 and publicly demonstrated on 23 June 1960. A 1958 US Navy press event produced a New York Times story describing an "embryo of an electronic computer" that the Navy expected would eventually walk, talk, see and be conscious of its existence, an early example of AI hype.

The learning rule

A perceptron computes

where $\mathbf{x}$ is the input vector, $\mathbf{w}$ the weight vector, $b$ a bias, and $\hat{y}$ the prediction. After each example with true label $y \in \{0,1\}$:

where $\eta > 0$ is the learning rate. If the prediction is right, $y - \hat{y} = 0$ and nothing changes. If the unit failed to fire when it should have, weights move toward $\mathbf{x}$; if it fired wrongly, they move away. The perceptron convergence theorem (proofs by Rosenblatt and others; Novikoff's 1962 proof is the standard one) guarantees the rule finds a separating line in a finite number of mistakes if the data are linearly separable.

Worked example: learning OR

Inputs and targets: (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→1. Start with $w_1 = w_2 = 0$, $b = 0$, $\eta = 1$.

Epoch 1:

• (0,0): $0 > 0$? No,$\hat{y}=0$, correct.
• (0,1): $0 > 0$? No,$\hat{y}=0$, target 1, error$+1$. Update$w_2 = 1$,$b = 1$. (Now$\mathbf{w}=(0,1)$,$b=1$.)
• (1,0): $0 + 0 + 1 = 1 > 0$, $\hat{y}=1$, correct.
• (1,1): $0+1+1 = 2 > 0$, correct.

Epoch 2:

• (0,0): $b = 1 > 0$, $\hat{y}=1$, target 0, error $-1$. Update $\mathbf{w}=(0,1)$, $b=0$.
• (0,1): $1 > 0$, correct. (1,0):$0 > 0$? No,$\hat{y}=0$, error$+1$:$\mathbf{w}=(1,1)$,$b=1$.
• (1,1): $3 > 0$, correct.

Epoch 3:

• (0,0): $1 > 0$, wrong, error$-1$:$\mathbf{w}=(1,1)$,$b=0$.
• (0,1): $1>0$ correct; (1,0): $1>0$ correct; (1,1): $2>0$ correct.

Epoch 4: (0,0): $0>0$? No, correct; all others correct. Converged with $\mathbf{w}=(1,1)$, $b=0$: the unit fires whenever at least one input is 1. That is OR.

Why XOR is impossible for one perceptron

XOR's targets are (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→0. A single perceptron needs $b \le 0$ (from (0,0)), $w_2 + b > 0$ and $w_1 + b > 0$ (from the middle two), and $w_1 + w_2 + b \le 0$ (from (1,1)). Adding the middle two gives $w_1 + w_2 + 2b > 0$, so $w_1 + w_2 + b > -b \ge 0$, contradicting the last condition. No line separates XOR's classes. A network with one hidden layer can compute XOR easily (for example, OR and NAND units feeding an AND unit), but in 1969 there was no widely known way to train hidden units.

This short numpy program runs the perceptron rule on OR and XOR and shows the difference:

import numpy as np

def train_perceptron(X, y, epochs=20, lr=1.0):
    w, b = np.zeros(X.shape[1]), 0.0
    for _ in range(epochs):
        mistakes = 0
        for xi, yi in zip(X, y):
            pred = 1 if xi @ w + b > 0 else 0
            if pred != yi:
                w += lr * (yi - pred) * xi
                b += lr * (yi - pred)
                mistakes += 1
        if mistakes == 0:
            return w, b, True
    return w, b, False

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
for name, y in [("OR", [0, 1, 1, 1]), ("XOR", [0, 1, 1, 0])]:
    w, b, ok = train_perceptron(X, np.array(y))
    print(name, "converged" if ok else "did not converge", w, b)
# OR converged [1. 1.] 0.0
# XOR did not converge ...

Minsky and Papert's Perceptrons (1969)

Marvin Minsky and Seymour Papert of MIT published Perceptrons: An Introduction to Computational Geometry in 1969. It was a rigorous mathematical analysis of what single-layer perceptrons (with fixed feature detectors feeding one trainable layer) can compute. They showed that some simple-looking predicates, including XOR-like parity and whether a figure is connected, require resources that grow impractically with the size of the input for such machines.

The book did not prove that multilayer networks were useless; it expressed scepticism that the approach would extend, while acknowledging the multilayer question as open. But it was widely read as a refutation of neural networks. Combined with the shift of funding toward symbolic AI, and with Rosenblatt's death in a boating accident in 1971, neural-network research in the US dwindled through the 1970s. Some work continued elsewhere, notably Kunihiko Fukushima's neocognitron (1980) in Japan and Teuvo Kohonen's associative-memory work in Finland (which led to his self-organizing maps in 1982). Historians still debate how much the book caused the decline versus documented it.

The first AI winter (~1974–1980)

An AI winter is a period of reduced funding and interest after expectations collapse. The term itself was popularized at the 1984 AAAI conference, where Roger Schank and Marvin Minsky warned that the expert-systems boom would end the same way. The first winter had several causes:

• Machine translation disappointment. The 1966 ALPAC report concluded that machine translation was slower, less accurate and more expensive than human translation, and US funding for it largely stopped.
• The Lighthill report (UK). The Science Research Council asked the applied mathematician Sir James Lighthill to assess AI. His report, Artificial Intelligence: A General Survey, dated July 1972 and published in 1973, divided the field into advanced automation, computer-based studies of the central nervous system, and the "bridge" between them (robotics and general reasoning), and judged that the bridge had failed to deliver. Its central technical criticism was combinatorial explosion: methods that worked on toy problems would not scale to real ones. UK government funding for AI was cut sharply, leaving only a few centres such as Edinburgh. In June 1973 the BBC televised a debate at the Royal Institution between Lighthill and Donald Michie, Richard Gregory and John McCarthy.
• DARPA's shift. In the US, the Mansfield Amendments (from 1969) required defence research to have direct military relevance, and DARPA moved from open-ended grants to mission-oriented projects. Its Speech Understanding Research programme (1971–76) was judged disappointing and not renewed.

Historians add a caveat: academic AI did not vanish in the 1970s (the expert-systems work of lesson 2 grew in exactly this period, and AI society membership grew), so the "winter" was severe in some funding streams and countries more than others.

The connectionist revival (1980s)

Hopfield networks (1982)

John Hopfield, a physicist then at Caltech, published "Neural networks and physical systems with emergent collective computational abilities" in PNAS in 1982. A Hopfield network is a fully connected network of binary units with symmetric weights. It has an energy function

(ignoring bias terms), where $s_i \in \{-1, +1\}$ is the state of unit $i$ and $w_{ij}=w_{ji}$ is the connection weight. Updating units one at a time never increases $E$, so the network settles into local minima. If memories are stored as minima (for example with the Hebbian rule $w_{ij} \propto \sum_{\mu} \xi^\mu_i \xi^\mu_j$ over stored patterns $\xi^\mu$), presenting a corrupted pattern lets the network roll downhill to the nearest stored memory: associative memory. Hopfield's work made neural networks respectable to physicists and brought the tools of statistical physics into the field.

Boltzmann machines (1985)

Geoffrey Hinton and Terrence Sejnowski, with David Ackley, introduced the Boltzmann machine ("A learning algorithm for Boltzmann machines", Cognitive Science, 1985). It is a Hopfield-like network with stochastic units and hidden units, whose states follow the Boltzmann distribution from statistical mechanics. Its learning rule could train hidden units to capture the statistical structure of data, a key step toward learning internal representations. It was slow, but its restricted form (the restricted Boltzmann machine) later became the building block of the 2006 deep belief networks (lesson 5). Hopfield and Hinton shared the 2024 Nobel Prize in Physics for these foundational contributions.

Backpropagation: a tangled history

Backpropagation computes how much each weight in a multilayer network contributed to the output error, by applying the chain rule of calculus backward from the output layer to the input. For a weight $w$ feeding a unit deep in the network, the gradient of the loss $L$ factorizes along the path:

where $\hat{y}$ is the network output and $h$ an intermediate unit's activation. Each factor is computed once and reused, which makes the whole gradient roughly as cheap as one forward pass. Credit for the idea is spread over decades:

• Control theory (1960s). Gradient methods through multistage systems appeared in optimal-control work by Henry Kelley, Arthur Bryson and others.
• Seppo Linnainmaa (1970). His master's thesis at the University of Helsinki described the reverse mode of automatic differentiation, the general, efficient algorithm that backpropagation is a special case of, motivated by analysing rounding errors. He published it in BIT in 1976.
• Paul Werbos (1974). His Harvard PhD thesis, "Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences," proposed training networks with backpropagated derivatives; he applied it to neural networks explicitly in a 1982 paper. The work drew little attention at the time.
• Rumelhart, Hinton and Williams (1986). David Rumelhart, Geoffrey Hinton and Ronald Williams published "Learning representations by back-propagating errors" in Nature in 1986, with a longer chapter in the Parallel Distributed Processing volumes edited by Rumelhart and James McClelland. Their crucial contribution was demonstrating that backprop makes hidden units learn useful internal representations (for example, discovering features of family-tree relationships), and they reached a wide audience. David Parker and Yann LeCun also described versions in the mid-1980s.

The fair summary: Rumelhart, Hinton and Williams popularized backpropagation for neural networks and showed what it could do; they did not invent the underlying algorithm. Through the late 1980s, backprop-trained networks achieved real results, such as NETtalk (Sejnowski and Rosenberg, 1987) learning to pronounce English text and LeCun's convolutional networks reading handwritten ZIP codes at Bell Labs (1989).

Why the revival stalled

Backpropagation worked for shallow networks, but deeper networks were hard to train: gradients shrank or blew up as they passed through many layers (the vanishing/exploding gradient problem, analysed by Sepp Hochreiter in 1991 and by Bengio, Simard and Frasconi in 1994). Datasets were small and computers slow. In the 1990s, support vector machines and other methods with cleaner theory and fewer tuning headaches often matched or beat neural networks (lesson 4). Connectionism did not die, but by the late 1990s it was again out of fashion in mainstream machine learning.

The second AI winter (~1987–1990s)

The second winter hit symbolic AI's commercial wave:

• Lisp machine collapse (1987). Companies such as Symbolics and Lisp Machines Inc. sold specialized workstations for AI development. General-purpose workstations from Sun and others, and increasingly powerful PCs, ran Lisp well enough at a fraction of the price. The specialized market collapsed around 1987.
• Expert-system disappointment. Systems proved costly to maintain, brittle and hard to update (lesson 2). Many were quietly retired or rebranded as "business rules" or "knowledge management."
• Government retrenchment. DARPA's Strategic Computing Initiative cut AI spending from 1988, and Japan's Fifth Generation project ended in the early 1990s without meeting its goals.

Researchers responded by avoiding the label. Through the 1990s much good work was published under names like machine learning, informatics, knowledge-based systems or computational intelligence, partly to avoid the stigma of "AI."

Analysis: the anatomy of a winter

Both winters followed a similar sequence:

1. A real result (the perceptron, early theorem provers, XCON) triggers interest.
2. Promises outrun the evidence, often in press coverage and funding proposals ("will walk, talk, see"; "a summer will make significant advances").
3. Scaling fails: methods hit combinatorial explosion, linear separability, brittleness or the knowledge bottleneck.
4. A credible external assessment (ALPAC, Lighthill, internal DARPA reviews) crystallizes doubt.
5. Funding and markets contract, and hardware economics (cheap general-purpose computers) can wipe out specialized vendors.
6. Ideas survive underground and return when data and compute catch up.

Step 6 is the most important for practitioners. The perceptron of 1958, backpropagation of 1970–86, convolutional networks of 1989 and Boltzmann machines of 1985 were all, with more data and faster hardware, the ingredients of the 2010s deep-learning boom.

Common pitfalls

• "Minsky and Papert proved neural networks can't work." They analysed single-layer perceptrons; the multilayer case was left open.
• "Hinton invented backpropagation." He co-authored the paper that popularized it in 1986; reverse-mode differentiation dates to Linnainmaa (1970) and its application to networks to Werbos (1974/1982). Hinton himself has said as much.
• Treating the winters as total. Funding fell sharply in particular countries and programmes; research continued, often under different names.
• Thinking winters were caused by bad science. Mostly they were caused by the gap between good but limited science and the promises made about it.

In practice

When you evaluate an AI claim today, apply the winter checklist: Is the result real but narrow? Does the claim assume it will scale? Has an independent evaluation tested it? Are vendors' economics dependent on specialized hardware that commodity hardware could undercut? In your own work, report limitations as clearly as results; the field has twice paid for promises outrunning evidence. And remember that "out of fashion" is not "wrong": several of today's core techniques spent decades as unfashionable ideas.