AI WINTER
The Mark I Perceptron
20 × 20 photocells
Rosenblatt’s perceptron (1958) learns a linear threshold rule from examples: whenever it errs, it nudges its weights toward the right answer. The Mark I built it in hardware at the Cornell Aeronautical Laboratory: a 20 × 20 eye of photocells, wired at random to 512 association units, its weights held on motor-driven potentiometers.
AI WINTER
The XOR Problem
no single line will do
Minsky and Papert proved in 1969 that a single-layer perceptron cannot compute exclusive-or: its two true cases sit at opposite corners of a square, and no straight line separates them from the other two. One hidden layer solves it with two lines, and backpropagation can learn them.
AI WINTER
Samuel’s Checkers
machine learning, named 1959
Arthur Samuel’s checkers program at IBM improved by playing against itself, adjusting the weights of its board evaluation and remembering positions it had seen, until it played better than he did. His 1959 paper, “Some studies in machine learning using the game of checkers”, is usually credited with naming the field.
AI WINTER
The Hopfield Network
E = −½ ∑ wijsisj
Hopfield showed in 1982 that a network of on-off units with symmetric connections has an energy, like a magnet’s, that every update lowers. Stored patterns sit at the energy minima, so a corrupted pattern slides down to the memory nearest it. The work shared the 2024 Nobel Prize in Physics.
AI WINTER
The Palmer Penguins
344 birds, 3 species
From 2007 to 2009, researchers at Palmer Station, Antarctica, measured the bill length and depth, flipper length and body mass of 344 Adélie, Chinstrap and Gentoo penguins. The data were published in 2014 and released in 2020 as an alternative to the iris data for teaching classification: three species, four measurements each.
LOCAL MINIMUM
Backpropagation
the chain rule, run backward
A network’s error depends on each weight through a chain of functions, so its derivative is a product of local derivatives, the chain rule. Backpropagation computes them all in one backward pass, reusing every product it has formed. Linnainmaa described this reverse mode in 1970; Rumelhart, Hinton and Williams showed in 1986 that it teaches hidden layers useful features.
LOCAL MINIMUM
Gradient Descent
Cauchy, 1847
To minimize a function, step downhill along the negative gradient. Cauchy proposed it in 1847 for solving systems of equations. In a long, narrow bowl, steepest descent with exact line searches turns a right angle at every step and zigzags slowly to the bottom, which is why momentum and good conditioning matter.
LOCAL MINIMUM
The Rosenbrock Valley
minimum at (1, 1)
Rosenbrock’s 1960 test function, f = (1 − x)² + 100(y − x²)², has a long, curved, nearly flat valley along y = x² with its single minimum at (1, 1). Finding the valley is easy; following it to the bottom is hard. It remains the first stress test for a new optimizer, and the ground this page’s background descends.
LOCAL MINIMUM
The ReLU and the Sigmoid
max(0, x) and 1/(1 + e−x)
A neuron sums its inputs and passes the sum through a nonlinearity. The logistic sigmoid, smooth and S-shaped, was the classic choice, but its slope vanishes at both ends. The rectified linear unit, max(0, x), keeps a slope of one for every positive input, and made deep networks far easier to train after 2010.
LOCAL MINIMUM
Dropout
half the units, every step
During training, dropout switches off each hidden unit at random, typically with probability one half, on every step, so no unit can rely on particular others; at test time every unit is used, its output scaled to match. Proposed by Hinton’s group in 2012 and published by Srivastava and colleagues in 2014, it trains a huge ensemble of thinned networks at once.
NO FREE LUNCH
The Maximum Margin
margin = 2/‖w‖
Of all the lines that separate two classes, the support vector machine picks the one with the widest margin, the empty band between them. Only the points on the band’s edges, the support vectors, fix where it lies: move any other point and nothing changes. Boser, Guyon and Vapnik (1992) and Cortes and Vapnik (1995) made it practical.
NO FREE LUNCH
The Kernel Trick
z = x² + y²
Points inside a circle and points outside it cannot be split by a straight line. Lift every point to a height equal to its squared distance from the center, z = x² + y², and a flat plane splits them cleanly. Kernel methods do this without ever computing the new coordinates, using only inner products in the lifted space.
NO FREE LUNCH
The Decision Tree
two questions, three species
A decision tree asks one yes-or-no question about one feature at a time, cutting the data into rectangles. Two well-chosen questions can sort three species: the first splits one off, the second divides the other two, and only the few points near a boundary land on the wrong side. CART (1984) and ID3 (1986) grow such trees from data.
NO FREE LUNCH
k-Means
each center at its centroid
k-means splits points into k groups by alternating two steps: assign each point to its nearest center, then move each center to the mean of its points. Lloyd devised it at Bell Labs in 1957; it was published in 1982. At convergence the cells form a centroidal Voronoi tessellation: every center sits at its own cell’s center of mass.
NO FREE LUNCH
Overfitting
ten points, degree nine
Fit ten noisy points taken from one smooth curve with polynomials of rising degree. A straight line misses the shape; a cubic follows it; a ninth-degree polynomial passes through every point exactly while swinging wildly between them. Its training error is zero and its error on new data is huge: the standard textbook picture.
LATENT SPACE
The Convolution
3 × 3 weights, shared
A convolutional layer slides one small grid of weights across an image, multiplying and summing at every position, so the same detector is used everywhere. A vertical-edge kernel such as Sobel’s lights up only where brightness changes from left to right. LeCun’s networks for handwritten zip codes (1989) learned their kernels instead.
LATENT SPACE
The First Layer
edges and blobs, learned
Train a deep network on millions of photographs and its first layer learns, unprompted, small filters that detect oriented edges at every angle and scale, and patches of opposing color. They resemble the Gabor-like receptive fields Hubel and Wiesel found in the visual cortex. Krizhevsky, Sutskever and Hinton showed their 96 in 2012.
LATENT SPACE
The t-SNE Map
van der Maaten & Hinton, 2008
t-SNE lays high-dimensional data out on a plane so that points close in the original space stay close. It matches neighbor probabilities, with a heavy-tailed Student-t on the map so separate clusters pull apart. Run on handwritten digits, it draws ten islands, one per digit, with a few strays between them.
LATENT SPACE
The Swiss Roll
Isomap and LLE, 2000
A sheet rolled into a spiral is two-dimensional, but points on neighboring layers are close in space and far apart along the sheet. Isomap, by Tenenbaum, de Silva and Langford, and locally linear embedding, by Roweis and Saul, published side by side in Science in December 2000, unroll it by measuring distances along the sheet.
LATENT SPACE
Diffusion
noise in, picture out
A diffusion model is trained by adding Gaussian noise to images step by step until only noise is left, and learning to undo each step. To generate, it runs backward: start from pure noise and remove it, step by step, until a picture appears. Sohl-Dickstein and colleagues introduced it in 2015; Ho, Jain and Abbeel’s DDPM (2020) made it work at scale.
CONTEXT WINDOW
The Attention Matrix
softmax(QKT/√d)V
Attention lets each position in a sequence look at the earlier ones and take a weighted average of what it finds. The weights, a softmax of query–key dot products, fill a square matrix whose rows each sum to one; in a model that writes left to right everything above the diagonal is masked to zero. Bahdanau, Cho and Bengio introduced attention for translation in 2014.
CONTEXT WINDOW
The Transformer Block
Vaswani et al., 2017
A transformer block does two things in turn: multi-head self-attention, which mixes information across positions, and a small feed-forward network applied to each position, four times wider inside. Each is wrapped in a residual connection and a layer normalization. Stacking such blocks, with no recurrence at all, gave the 2017 transformer.
CONTEXT WINDOW
The LSTM Cell
Hochreiter & Schmidhuber, 1997
A long short-term memory cell carries its memory along a straight path, the cell state, where information can travel many steps without fading. Gates, each a sigmoid between 0 and 1, decide what to forget, what to write and what to read out. Hochreiter and Schmidhuber designed it in 1997 to beat the vanishing gradient; Gers, Schmidhuber and Cummins added the forget gate in 2000.
CONTEXT WINDOW
The Cart-Pole
Barto, Sutton & Anderson, 1983
A cart on a track must keep a hinged pole upright by pushing left or right, learning only from when the pole falls or the cart hits the end. Michie and Chambers used it in 1968; Barto, Sutton and Anderson’s actor–critic learned to balance it in 1983, and it has been the first test of reinforcement learning ever since.
CONTEXT WINDOW
Value Iteration
Bellman, 1957
In a gridworld the value of a square is the best immediate reward plus the discounted value of where you land: Bellman’s equation. Sweeping it until nothing changes gives every square its value and its best move. In Russell and Norvig’s 4 × 3 world, the square just below the penalty takes the long way around.