AI & Machine Learning · Subject 23

AI & Machine Learning

From the perceptron rule to early stopping, lettered on button-ups and ties, with socks to match.

AI & Machine Learning

Scrolling is the history of machine learning, from Rosenblatt’s 1958 perceptron to the models of today. The perceptron’s retina learns a letter; Minsky and Papert’s XOR stops it, and the first AI winter frosts the screen; backpropagation sends errors backward through a network; LeNet-5 reads a digit and AlexNet’s filters appear; word2vec’s king − man + woman lands on queen; a transformer’s attention arcs between the words of a sentence while the context window grows from 512 tokens to a million; the GPU wall fills to Llama 3.1’s 16,384 H100s as training compute climbs a power law; and at the end the model writes, about the page and about you.

This week in AI & Machine Learning,

An AI pitches over 4,400 research ideas to 100-plus lab heads, a climate emulator learns more from one odd scenario than from six standard ones, and a new score shows when merging cell data pays off.

Get it by email

Download MP3

Welcome in

For anyone who has watched a loss curve flatten

You know the quiet of a training run at three in the morning, the loss curve you refresh more often than your email, and the moment validation error turns up while training error keeps falling. You have held out a test set and promised not to peek. You have chased a learning rate down by factors of ten. You remember the first network that did something you never told it to: a filter that found edges, a cluster that meant something, a prediction that was right for the right reason.

Short-sleeve button-up · XS–4XL

EQUATION STRIPE

From the perceptron rule to Bellman’s equation, every line labeled.

Machine learning keeps its ideas in a line each: an update rule, a loss, a bound, a recursion. The Equation Stripe sets forty-five of them in rows, each introduced by its name in capitals, so PERCEPTRON RULE, BACKPROPAGATION and ATTENTION read at a glance. Together they make a working practitioner’s commonplace book: how a unit fires and a network remembers, how losses are measured and minimized, how the classical models draw their boundaries, how representations are learned, and how models that live in time keep state and decide.

Twenty-three rows carry the science of the collection’s five picture designs. The other twenty-two are results every practitioner meets: the McCulloch–Pitts neuron, perceptron convergence, Hebbian storage and Hopfield’s capacity, softmax, cross-entropy, mean squared error, the Robbins–Monro conditions, momentum, Adam, the universal approximator, the ridge penalty, the SVM primal, the Gaussian kernel, information gain, the nearest-neighbor bound, principal components, the denoising loss, positional encoding, layer norm, the LSTM output and Q-learning.

  • McCulloch–Pitts Neurony = H(∑iwixi − θ)
  • Perceptron Rulew ← w + η(y − ŷ)x
  • Perceptron Convergencemistakes ≤ (R/γ)2
  • XORx1 ⊕ x2 = (x1 ∨ x2) ∧ ¬(x1 ∧ x2)
  • Logistic RegressionP(y = 1 | x) = σ(wTx + b)
  • Hopfield EnergyE = −½∑ijwijsisj
Shop now · Regular price $65 Sale price $58.50

Leave your email and we’ll write the day it’s ready.

Short-sleeve button-up · XS–4XL

CONCEPT STRIPE

From the imitation game to self-play, the field in short lines.

The Concept Stripe carries the field’s working vocabulary: its dated history, the kinds of learning, the craft of training, the architectures, the classical models and the decisions, each after a short heading in capitals. About every fourth unit is an equation from the Equation Stripe, so the PERCEPTRON RULE and the BELLMAN EQUATION turn up between DARTMOUTH, 1956 and NO FREE LUNCH.

  • THE IMITATION GAMETuring, 1950: machine or person?
  • DARTMOUTH, 1956where artificial intelligence was named
  • AI WINTERwhen the funding froze, twice
  • TRAIN, VALIDATE, TESTnever grade on the homework
  • EARLY STOPPINGstop when validation error turns up
  • NO FREE LUNCHno learner wins on every problem
Shop now · Regular price $65 Sale price $58.50

Leave your email and we’ll write the day it’s ready.

In production.

AI WINTER

The first learning machines, the problem that froze them, and the physics that thawed them.

The field’s first decades in four objects, and the data set students meet first today. Arthur Samuel’s checkers program improved by playing against itself, and his 1959 paper is usually credited with naming machine learning. Rosenblatt’s perceptron (1958) learned a linear threshold rule from examples, and the Mark I built it in hardware: a 20 × 20 eye of photocells, wired at random to 512 association units, its weights held on motor-driven potentiometers.

In 1969 Minsky and Papert proved that a single-layer perceptron cannot compute exclusive-or: its two true cases sit at opposite corners of a square, and no straight line separates them from the other two. In 1982 Hopfield gave a network of on-off units an energy, like a magnet’s, that every update lowers, so a corrupted memory slides down to the stored pattern nearest it; the work brought physicists in and shared the 2024 Nobel Prize in Physics. Beside them stand the Palmer penguins: 344 birds of three species, measured from 2007 to 2009 and released in 2020 for teaching classification. AI winter is the insiders’ word for the funding freezes of the 1970s and the late 1980s.

  • The Mark I PerceptronThe Mark I’s eye of 400 photocells, its random patch-cord wiring and its motor-driven weight knobs.
  • The XOR ProblemFour spheres on a stone tile, the two vermilion ones diagonal: no one rod can split them off, but two parallel rods can.
  • Samuel’s CheckersA checkerboard in mid-game, two kings crowned: the game a program learned to play better than its author.
  • The Hopfield NetworkTwo panels of flip-dots, up or down like spins: a ring with twenty dots wrong, and the same ring recalled whole.
  • The Palmer PenguinsOne Adélie, one Chinstrap and one Gentoo, side by side: the three species of the Palmer data.
Tell me when it’s ready

Leave your email and we’ll write the day it’s ready.

In production.

LOCAL MINIMUM

How a network learns: the slope, the step, the valley and the tricks.

Optimization is the engine of deep learning. To minimize a function, step downhill along the negative gradient, as Cauchy proposed in 1847; in a long, narrow bowl, steepest descent with exact line searches turns a right angle at every step and zigzags slowly to the bottom, which is why momentum and good conditioning matter. Rosenbrock’s 1960 function, f = (1 − x)² + 100(y − x²)², is the standard stress test: finding its curved valley is easy, following it to the minimum at (1, 1) is hard. The background of this page runs that descent, with momentum.

A network’s error depends on each weight through a chain of functions, so backpropagation computes every derivative in one backward pass of the chain rule; Linnainmaa described this reverse mode in 1970, and Rumelhart, Hinton and Williams showed in 1986 that it teaches hidden layers useful features. The sigmoid’s slope vanishes at both ends, while the rectified linear unit, max(0, x), keeps a slope of one for every positive input. Dropout switches off each hidden unit at random on every training step. The name is every practitioner’s fear, and the joke is that in high dimensions it is rarely the real problem.

  • BackpropagationA brass gear train whose ratios multiply, 3 × 2 × 4: the chain rule, which backpropagation runs in reverse.
  • Gradient DescentGold beads zigzagging down an oval porcelain bowl, every step at right angles to the last: steepest descent.
  • The Rosenbrock ValleyA plaster model of Rosenbrock’s surface: a banana-shaped valley, nearly flat along its floor, a pin at (1, 1).
  • The ReLU and the SigmoidTwo neon tubes: the amber S of the sigmoid and the blue-white hinge of the ReLU, sharing one invisible square.
  • DropoutA glass-and-brass network with half its hidden spheres gone dark and every rod to them removed: one thinned network.
Tell me when it’s ready

Leave your email and we’ll write the day it’s ready.

In production.

NO FREE LUNCH

Margins, kernels, splits and clusters, and the price of fitting too well.

The classical models every data scientist reaches for first. Of all the lines that separate two classes, the support vector machine picks the one with the widest margin, 2/‖w‖, and only the points on its edges, the support vectors, fix where it lies (Boser, Guyon and Vapnik, 1992; Cortes and Vapnik, 1995). Points inside a circle and points outside it cannot be split by a straight line until each is lifted to a height z = x² + y²; kernel methods do this using only inner products in the lifted space.

A decision tree asks one yes-or-no question about one feature at a time, cutting the data into rectangles; CART (1984) and ID3 (1986) grow such trees from data. k-means alternates two steps, assigning each point to its nearest center and moving each center to the mean of its points; Lloyd devised it at Bell Labs in 1957. And a ninth-degree polynomial passes through ten noisy points exactly while swinging wildly between them: training error zero, error on new data huge. Wolpert and Macready’s no-free-lunch theorems (1996–97) say no one learner wins on every problem, which is why all five are still used.

  • The Maximum MarginIvory and vermilion spheres on stone, split by a brass rod with a steel rod either side: three spheres touch the margin.
  • The Kernel TrickA glass paraboloid: vermilion beads low in the bowl, ivory beads high up its wall, and a flat glass plate between.
  • The Decision TreeA marquetry panel cut twice by brass strips into three woods: two questions that sort three species.
  • k-MeansA round stained-glass window of twelve cells, each with a jewel at its exact center of mass: k-means at rest.
  • OverfittingThe same ten pearls on three boards: a straight wire misses, a gentle curve fits, a wild one hits every pearl.
Tell me when it’s ready

Leave your email and we’ll write the day it’s ready.

In production.

LATENT SPACE

What a network sees, and the space where it keeps it.

A convolutional layer slides one small grid of weights across an image, so the same detector is used everywhere; LeCun’s networks for handwritten zip codes (1989) learned their kernels. Train a deep network on millions of photographs and its first layer learns, unprompted, filters for oriented edges and patches of opposing color, much like the receptive fields Hubel and Wiesel found in the visual cortex; Krizhevsky, Sutskever and Hinton showed their 96 in 2012.

t-SNE lays high-dimensional data out on a plane so that neighbors stay neighbors, with a heavy-tailed Student-t on the map so separate clusters pull apart (van der Maaten and Hinton, 2008). Isomap and locally linear embedding, published side by side in Science in December 2000, unroll a Swiss roll by measuring distances along the sheet. A diffusion model learns to undo noise added step by step, then runs backward from pure noise to a picture (Sohl-Dickstein and colleagues, 2015; Ho, Jain and Abbeel, 2020). The latent space is where all of them live.

  • The ConvolutionA glass mosaic of a white disc and, beside it, what a vertical-edge kernel sees: only its left and right rims.
  • The First LayerForty-eight back-lit glass tiles, each a striped blob at its own angle or a soft patch of opposing colors.
  • The t-SNE MapA slab of glass holding ten glowing clouds of points in ten colors: islands of similar things, pulled apart.
  • The Swiss RollA glass ribbon rolled into a spiral, its color running through the spectrum along its length: the Swiss roll.
  • DiffusionSix glass mosaics in a row: a golden disc dissolving step by step into noise, the path a diffusion model reverses.
Tell me when it’s ready

Leave your email and we’ll write the day it’s ready.

In production.

CONTEXT WINDOW

Models that live in time: what they keep, what they attend to, and what they do next.

Attention lets each position in a sequence take a weighted average of what it finds at the others. The weights, a softmax of query–key dot products, fill a square matrix whose rows each sum to one, and a model that writes left to right masks everything above the diagonal; Bahdanau, Cho and Bengio introduced it for translation in 2014. A transformer block does two things in turn, multi-head self-attention and a feed-forward network four times wider inside, each wrapped in a residual connection and a layer normalization. Stacking such blocks, with no recurrence at all, gave the 2017 transformer.

Before it, the long short-term memory cell carried its memory along a straight path, with sigmoid gates that decide what to forget, what to write and what to read out (Hochreiter and Schmidhuber, 1997; the forget gate, 2000). Reinforcement learning’s two textbook problems close the design: a cart that must keep a hinged pole upright, which Barto, Sutton and Anderson’s actor–critic learned to balance in 1983, and a 4 × 3 gridworld solved by sweeping Bellman’s equation until nothing changes. The context window is how far back a model can look.

  • The Attention MatrixAn 8 × 8 grid of glass cells glowing amber below the diagonal and dark above it: each row’s light adds to one.
  • The Transformer BlockA glass-and-brass model of one transformer block: eight attention heads, a feed-forward layer, two bypass pipes.
  • The LSTM CellA straight copper pipe of memory with three brass valves: one to forget, one to write in, one to read out.
  • The Cart-PoleA brass cart on a short steel rail with a slender pole hinged to it, leaning a few degrees: balance it or fail.
  • Value IterationA walnut 4 × 3 board, a gold exit and a garnet trap, and brass arrows showing the best move from every square.
Tell me when it’s ready

Leave your email and we’ll write the day it’s ready.

Socks mix and match across every field: three pairs for $69, and two ship free in the U.S.

Crew socks · One size

Crew socks

A field of your own, from cuff to toe.

  • The pattern
  • The previewA studio mockup; photos of the finished pair are on the way.
Shop now · Regular price $26 Sale price $23.40

Leave your email and we’ll write the day this pair is ready.

Necktie · 3.5 in wide

Subject ties

A formal introduction to your field.

  • The color
  • The previewStudio render of the tie. Zoom shows the print up close.
Shop now · Regular price $36 Sale price $32.40

Leave your email and we’ll write the day this tie is ready.

MCCULLOCH–PITTS NEURONy = H(∑iwixi − θ)·PERCEPTRON RULEw ← w + η(y − ŷ)x·PERCEPTRON CONVERGENCEmistakes ≤ (R/γ)2·XORx1 ⊕ x2 = (x1 ∨ x2) ∧ ¬(x1 ∧ x2)·LOGISTIC REGRESSIONP(y = 1 | x) = σ(wTx + b)·HOPFIELD ENERGYE = −½∑ijwijsisj·HEBBIAN STORAGEwij = (1/N)∑μξiμξjμ·HOPFIELD CAPACITYpmax ≈ 0.138N·LOGISTIC SIGMOIDσ(x) = 1/(1 + e−x), σ′ = σ(1 − σ)·RELUReLU(x) = max(0, x)·SOFTMAXsoftmax(z)i = exp(zi)/∑jexp(zj)·CROSS-ENTROPYL = −∑iyi log pi·MEAN SQUARED ERRORL = (1/n)∑i(yi − ŷi)2·GRADIENT DESCENTθ ← θ − η∇L(θ)·ROBBINS–MONRO∑ηt = ∞, ∑ηt2 < ∞·MOMENTUMv ← βv + ∇L, θ ← θ − ηv·ADAMθ ← θ − ηm̂/(√v̂ + ε)·BACKPROPAGATIONδl = (Wl+1)Tδl+1 ⊙ σ′(zl)·ROSENBROCK FUNCTIONf(x, y) = (1 − x)2 + 100(y − x2)2·DROPOUTh̃ = m ⊙ h/(1 − p), mi ~ Bernoulli(1 − p)·UNIVERSAL APPROXIMATORf(x) ≈ ∑iαiσ(wiTx + bi)·BIAS–VARIANCEE[(y − f̂)2] = Bias2 + Var + σ2·RIDGE PENALTYL(w) + λ‖w‖2·SVM MARGINmargin = 2/‖w‖·MCCULLOCH–PITTS NEURONy = H(∑iwixi − θ)·PERCEPTRON RULEw ← w + η(y − ŷ)x·PERCEPTRON CONVERGENCEmistakes ≤ (R/γ)2·XORx1 ⊕ x2 = (x1 ∨ x2) ∧ ¬(x1 ∧ x2)·LOGISTIC REGRESSIONP(y = 1 | x) = σ(wTx + b)·HOPFIELD ENERGYE = −½∑ijwijsisj·HEBBIAN STORAGEwij = (1/N)∑μξiμξjμ·HOPFIELD CAPACITYpmax ≈ 0.138N·LOGISTIC SIGMOIDσ(x) = 1/(1 + e−x), σ′ = σ(1 − σ)·RELUReLU(x) = max(0, x)·SOFTMAXsoftmax(z)i = exp(zi)/∑jexp(zj)·CROSS-ENTROPYL = −∑iyi log pi·MEAN SQUARED ERRORL = (1/n)∑i(yi − ŷi)2·GRADIENT DESCENTθ ← θ − η∇L(θ)·ROBBINS–MONRO∑ηt = ∞, ∑ηt2 < ∞·MOMENTUMv ← βv + ∇L, θ ← θ − ηv·ADAMθ ← θ − ηm̂/(√v̂ + ε)·BACKPROPAGATIONδl = (Wl+1)Tδl+1 ⊙ σ′(zl)·ROSENBROCK FUNCTIONf(x, y) = (1 − x)2 + 100(y − x2)2·DROPOUTh̃ = m ⊙ h/(1 − p), mi ~ Bernoulli(1 − p)·UNIVERSAL APPROXIMATORf(x) ≈ ∑iαiσ(wiTx + bi)·BIAS–VARIANCEE[(y − f̂)2] = Bias2 + Var + σ2·RIDGE PENALTYL(w) + λ‖w‖2·SVM MARGINmargin = 2/‖w‖·

The science on the shirt

What you’re wearing

Every object on the shirts, and the science behind it.

AI WINTER

The Mark I Perceptron

20 × 20 photocells

Rosenblatt’s perceptron (1958) learns a linear threshold rule from examples: whenever it errs, it nudges its weights toward the right answer. The Mark I built it in hardware at the Cornell Aeronautical Laboratory: a 20 × 20 eye of photocells, wired at random to 512 association units, its weights held on motor-driven potentiometers.

The XOR Problem

AI WINTER

The XOR Problem

no single line will do

Minsky and Papert proved in 1969 that a single-layer perceptron cannot compute exclusive-or: its two true cases sit at opposite corners of a square, and no straight line separates them from the other two. One hidden layer solves it with two lines, and backpropagation can learn them.

Samuel’s Checkers

AI WINTER

Samuel’s Checkers

machine learning, named 1959

Arthur Samuel’s checkers program at IBM improved by playing against itself, adjusting the weights of its board evaluation and remembering positions it had seen, until it played better than he did. His 1959 paper, “Some studies in machine learning using the game of checkers”, is usually credited with naming the field.

AI WINTER

The Hopfield Network

E = −½ ∑ wijsisj

Hopfield showed in 1982 that a network of on-off units with symmetric connections has an energy, like a magnet’s, that every update lowers. Stored patterns sit at the energy minima, so a corrupted pattern slides down to the memory nearest it. The work shared the 2024 Nobel Prize in Physics.

The Palmer Penguins

AI WINTER

The Palmer Penguins

344 birds, 3 species

From 2007 to 2009, researchers at Palmer Station, Antarctica, measured the bill length and depth, flipper length and body mass of 344 Adélie, Chinstrap and Gentoo penguins. The data were published in 2014 and released in 2020 as an alternative to the iris data for teaching classification: three species, four measurements each.

LOCAL MINIMUM

Backpropagation

the chain rule, run backward

A network’s error depends on each weight through a chain of functions, so its derivative is a product of local derivatives, the chain rule. Backpropagation computes them all in one backward pass, reusing every product it has formed. Linnainmaa described this reverse mode in 1970; Rumelhart, Hinton and Williams showed in 1986 that it teaches hidden layers useful features.

LOCAL MINIMUM

Gradient Descent

Cauchy, 1847

To minimize a function, step downhill along the negative gradient. Cauchy proposed it in 1847 for solving systems of equations. In a long, narrow bowl, steepest descent with exact line searches turns a right angle at every step and zigzags slowly to the bottom, which is why momentum and good conditioning matter.

LOCAL MINIMUM

The Rosenbrock Valley

minimum at (1, 1)

Rosenbrock’s 1960 test function, f = (1 − x)² + 100(y − x²)², has a long, curved, nearly flat valley along y = x² with its single minimum at (1, 1). Finding the valley is easy; following it to the bottom is hard. It remains the first stress test for a new optimizer, and the ground this page’s background descends.

The ReLU and the Sigmoid

LOCAL MINIMUM

The ReLU and the Sigmoid

max(0, x) and 1/(1 + e−x)

A neuron sums its inputs and passes the sum through a nonlinearity. The logistic sigmoid, smooth and S-shaped, was the classic choice, but its slope vanishes at both ends. The rectified linear unit, max(0, x), keeps a slope of one for every positive input, and made deep networks far easier to train after 2010.

Dropout

LOCAL MINIMUM

Dropout

half the units, every step

During training, dropout switches off each hidden unit at random, typically with probability one half, on every step, so no unit can rely on particular others; at test time every unit is used, its output scaled to match. Proposed by Hinton’s group in 2012 and published by Srivastava and colleagues in 2014, it trains a huge ensemble of thinned networks at once.

The Maximum Margin

NO FREE LUNCH

The Maximum Margin

margin = 2/‖w‖

Of all the lines that separate two classes, the support vector machine picks the one with the widest margin, the empty band between them. Only the points on the band’s edges, the support vectors, fix where it lies: move any other point and nothing changes. Boser, Guyon and Vapnik (1992) and Cortes and Vapnik (1995) made it practical.

NO FREE LUNCH

The Kernel Trick

z = x² + y²

Points inside a circle and points outside it cannot be split by a straight line. Lift every point to a height equal to its squared distance from the center, z = x² + y², and a flat plane splits them cleanly. Kernel methods do this without ever computing the new coordinates, using only inner products in the lifted space.

The Decision Tree

NO FREE LUNCH

The Decision Tree

two questions, three species

A decision tree asks one yes-or-no question about one feature at a time, cutting the data into rectangles. Two well-chosen questions can sort three species: the first splits one off, the second divides the other two, and only the few points near a boundary land on the wrong side. CART (1984) and ID3 (1986) grow such trees from data.

k-Means

NO FREE LUNCH

k-Means

each center at its centroid

k-means splits points into k groups by alternating two steps: assign each point to its nearest center, then move each center to the mean of its points. Lloyd devised it at Bell Labs in 1957; it was published in 1982. At convergence the cells form a centroidal Voronoi tessellation: every center sits at its own cell’s center of mass.

Overfitting

NO FREE LUNCH

Overfitting

ten points, degree nine

Fit ten noisy points taken from one smooth curve with polynomials of rising degree. A straight line misses the shape; a cubic follows it; a ninth-degree polynomial passes through every point exactly while swinging wildly between them. Its training error is zero and its error on new data is huge: the standard textbook picture.

The Convolution

LATENT SPACE

The Convolution

3 × 3 weights, shared

A convolutional layer slides one small grid of weights across an image, multiplying and summing at every position, so the same detector is used everywhere. A vertical-edge kernel such as Sobel’s lights up only where brightness changes from left to right. LeCun’s networks for handwritten zip codes (1989) learned their kernels instead.

The First Layer

LATENT SPACE

The First Layer

edges and blobs, learned

Train a deep network on millions of photographs and its first layer learns, unprompted, small filters that detect oriented edges at every angle and scale, and patches of opposing color. They resemble the Gabor-like receptive fields Hubel and Wiesel found in the visual cortex. Krizhevsky, Sutskever and Hinton showed their 96 in 2012.

The t-SNE Map

LATENT SPACE

The t-SNE Map

van der Maaten & Hinton, 2008

t-SNE lays high-dimensional data out on a plane so that points close in the original space stay close. It matches neighbor probabilities, with a heavy-tailed Student-t on the map so separate clusters pull apart. Run on handwritten digits, it draws ten islands, one per digit, with a few strays between them.

The Swiss Roll

LATENT SPACE

The Swiss Roll

Isomap and LLE, 2000

A sheet rolled into a spiral is two-dimensional, but points on neighboring layers are close in space and far apart along the sheet. Isomap, by Tenenbaum, de Silva and Langford, and locally linear embedding, by Roweis and Saul, published side by side in Science in December 2000, unroll it by measuring distances along the sheet.

LATENT SPACE

Diffusion

noise in, picture out

A diffusion model is trained by adding Gaussian noise to images step by step until only noise is left, and learning to undo each step. To generate, it runs backward: start from pure noise and remove it, step by step, until a picture appears. Sohl-Dickstein and colleagues introduced it in 2015; Ho, Jain and Abbeel’s DDPM (2020) made it work at scale.

The Attention Matrix

CONTEXT WINDOW

The Attention Matrix

softmax(QKT/√d)V

Attention lets each position in a sequence look at the earlier ones and take a weighted average of what it finds. The weights, a softmax of query–key dot products, fill a square matrix whose rows each sum to one; in a model that writes left to right everything above the diagonal is masked to zero. Bahdanau, Cho and Bengio introduced attention for translation in 2014.

CONTEXT WINDOW

The Transformer Block

Vaswani et al., 2017

A transformer block does two things in turn: multi-head self-attention, which mixes information across positions, and a small feed-forward network applied to each position, four times wider inside. Each is wrapped in a residual connection and a layer normalization. Stacking such blocks, with no recurrence at all, gave the 2017 transformer.

The LSTM Cell

CONTEXT WINDOW

The LSTM Cell

Hochreiter & Schmidhuber, 1997

A long short-term memory cell carries its memory along a straight path, the cell state, where information can travel many steps without fading. Gates, each a sigmoid between 0 and 1, decide what to forget, what to write and what to read out. Hochreiter and Schmidhuber designed it in 1997 to beat the vanishing gradient; Gers, Schmidhuber and Cummins added the forget gate in 2000.

CONTEXT WINDOW

The Cart-Pole

Barto, Sutton & Anderson, 1983

A cart on a track must keep a hinged pole upright by pushing left or right, learning only from when the pole falls or the cart hits the end. Michie and Chambers used it in 1968; Barto, Sutton and Anderson’s actor–critic learned to balance it in 1983, and it has been the first test of reinforcement learning ever since.

Value Iteration

CONTEXT WINDOW

Value Iteration

Bellman, 1957

In a gridworld the value of a square is the best immediate reward plus the discounted value of where you land: Bellman’s equation. Sweeping it until nothing changes gives every square its value and its best move. In Russell and Norvig’s 4 × 3 world, the square just below the penalty takes the long way around.

Shop AI & Machine Learning: Wear it to work

Made to order for you · free U.S. shipping on $46+ (shirts, ties and socks) · 10% off through 30 Nov

More AI & Machine Learning designs are on the way.

Leave your email and we’ll write the day they’re ready.