The Error That Traveled Backward

A network could finally learn its hidden work.

The output has a target, but a hidden unit does not. In their October 1986 Nature letter, David Rumelhart, Geoffrey Hinton, and Ronald Williams showed how the chain rule could turn output error into a training signal for every earlier connection. Their longer PDP chapter made the procedure and its learned internal representations concrete.

Editorial reconstruction of a mid-1980s research desk with a terminal, network diagrams, printouts, and loss curves — Editorial reconstruction created for this guide Small networks, consequential idea

The problem

A wrong output reveals that something upstream should change. It does not say which hidden weight was responsible, or by how much.

The method

Backpropagation reuses downstream derivatives. The chain rule converts one output loss into a signed sensitivity for each parameter.

The result

Hidden activity patterns become useful internal features. The teacher supplies the final answer, while the network discovers intermediate structure.

1986 was a landmark, not year zero. Reverse differentiation had earlier lineages in numerical methods and neural learning. This work brought it into the center of connectionist research, demonstrated its practical value, and made representation learning the main event.

Predict forward. Assign credit backward.

Backpropagation is the efficient gradient calculation between a loss and every earlier weight. An optimizer then uses those gradients to update the model.

gradient = downstream signal × local sensitivity

One chain-rule example: ∂L/∂W₁ = (∂L/∂h)(∂h/∂W₁)

Forward

Compute hidden activations and a prediction from the current weights.

Compare

Turn the difference between prediction and target into one scalar loss.

Differentiate

Trace how strongly each parameter influenced that loss.

Update

Move every parameter a small distance opposite its gradient.

A useful distinction. Backpropagation computes gradients. Gradient descent, momentum, Adam, or another optimizer decides how those gradients change the parameters.

Interactive credit lab

Teach a hidden layer XOR.

The previous guide ended where one straight boundary failed. Now freeze the learning loop between its forward guess, backward gradients, and weight update.

Choose one example

XOR · fixed order 11, 01, 10, 00

Input and target

1, 1 target 0 0, 1 target 1 1, 0 target 1 0, 0 target 0

Updates 0

Epoch 0

Accuracy 50%

weight change = -η × gradient This lab uses η = 1.5 and binary cross-entropy. Every update is computed from one frozen snapshot.

Forward value Backward derivative solid + / dashed - weight

Target0

Predictionnot run

Lossnot run

Learning record

Ready for the forward pass. The seed network guesses before it knows which hidden features XOR needs.

Mean loss 0.697

Input	Target	p(1)	Guess

Open the gradient ledger

Parameter	Gradient	Change

Visible values are rounded to three decimals. The simulation keeps full precision. Biases are trainable parameters too, even though they do not depend on an input feature.

Hidden layers redraw the problem.

XOR does not become separable in its original coordinates. The hidden layer learns a new coordinate system in which one final boundary can work.

Input space

The raw XOR corners never move. Matching targets sit on opposite diagonals.

Hidden space

The seed features still leave the cases tangled.

The hidden units are learning features, not memorizing a line. In this tiny model the two features can be plotted. Modern representations are usually distributed across hundreds or thousands of dimensions and are rarely so easy to name.

A selected path to learned representations.

Backpropagation belongs to several overlapping histories: differentiation, optimization, neural networks, cognitive science, computer hardware, and data.

1958

One boundary learns

Rosenblatt's perceptron adjusts output weights after mistakes.

1970

Reverse differentiation

Seppo Linnainmaa describes a general backward accumulation through a sequence of computations.

1974

Ordered derivatives

Paul Werbos develops backward derivatives for nonlinear systems and discusses adaptive networks.

1986

Hidden features learn

Rumelhart, Hinton, and Williams make credit assignment and internal representations widely influential.

2012→

Depth meets scale

Large datasets, accelerators, better architectures, and robust software turn gradients into modern deep learning.

This is a selected lineage, not a single-inventor story. Closely related neural-learning procedures were also described independently by David Parker and Yann LeCun in 1985.

The calculus stayed. The machinery scaled.

Today's models use richer operations and optimizers, but their training loop still evaluates a computation, measures a loss, and differentiates that loss backward.

1986 learning step W ← W - η ∂L/∂W

Modern training θ ← optimizer(θ, ∇θL)

Small sigmoid networks

Deep computation graphs with attention, normalization, residual paths, and routing

Carefully prepared toy domains

Large text, image, audio, video, scientific, and multimodal datasets

Hand-derived backward equations

Automatic-differentiation systems build the backward computation

Gradient descent with momentum

Adam-family optimizers, schedules, clipping, regularization, and distributed training

Inspectably small hidden codes

High-dimensional distributed representations learned across many layers

The breakthrough was credit assignment.

A final error became a distributed instruction for every hidden connection. Networks could learn not only a decision, but the intermediate features needed to make it.

predict compare trace influence adjust represent better

Follow the original trail.

The historical framing, equations, examples, and modern comparison in this guide are grounded in these primary publications and author-hosted copies.

1970/76 Taylor Expansion of the Accumulated Rounding Error Seppo Linnainmaa, reverse accumulation published in English in 1976 Open DOI 1974 Beyond Regression Paul J. Werbos, Harvard doctoral dissertation Open thesis 1986 Learning Representations by Back-Propagating Errors David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams Open Nature 1986 Learning Internal Representations by Error Propagation Expanded chapter in Parallel Distributed Processing, Volume 1 Open chapter 2015 Deep Learning Yann LeCun, Yoshua Bengio, and Geoffrey Hinton Open review 2017 Attention Is All You Need Ashish Vaswani and colleagues Open paper

Historical wording is intentionally careful. Backpropagation computes gradients; an optimizer changes the weights. The 1986 publications were landmark demonstrations and explanations, not the first appearance of reverse differentiation.