The synthesis
Convolutional, recurrent, and multilayer networks shared a central idea: useful representations could be learned.
Depth became a practical research program.
In their 2015 Nature review, Yann LeCun, Yoshua Bengio, and Geoffrey Hinton described deep learning as representation learning through multiple processing layers. The article connected decades of neural-network research to advances in speech, vision, and language.
The change was systemic. Larger datasets, faster accelerators, trainable architectures, regularization, and backpropagation began working together at useful scale. In 2012, AlexNet combined these ingredients to reach 15.3% top-5 ImageNet error, compared with 26.2% for the runner-up.
The ingredients
Data, accelerators, architectures, regularization, and gradient-based learning became one working system.
The evidence
Results in speech and visual recognition showed that deeper learned features could succeed at real scale.
2015 was a landmark synthesis, not an invention date. Convolutional networks, backpropagation, recurrent networks, benchmark datasets, and representation learning all had substantial earlier histories.
A deep network learns a ladder of representations.
Every layer transforms the representation before it. Depth lets simple features be reused and composed into richer ones.
raw signal
x
pixels, sound, or tokens
hidden 1
local patterns
small regularities
hidden 2
motifs
reused combinations
hidden 3
parts and context
task-relevant structure
readout
prediction
ŷ
compare
loss
L
gradients return through the full stack
h(ℓ) = f(W(ℓ)h(ℓ-1) + b(ℓ))
Representation at layer ℓ equals a learned nonlinear transform of the layer before it.
See locally
Early units respond to small regularities.
Compose
Later units combine earlier features.
Reuse
One feature can support many downstream patterns.
Decide
The final representation supports a task prediction.
Depth is capacity, not magic. Useful hierarchies still depend on data, objectives, architecture, optimization, and inductive bias. Backpropagation computes the gradients that coordinate the stack.
Interactive representation lab
Watch a network redraw the problem.
The dots do not move in the world. Their coordinates move inside the network.
Training checkpoint
Representation depth
Input coordinates
update 0
Upper moon
Lower moon
Traced point
Training accuracy
50.0%
Mean loss
0.684
Straight-line readability
87.5%
Learned parameters
21
The raw coordinates contain the pattern, but no straight line can perfectly split it.
At update 0, the hidden transforms are random and the network is guessing.
Follow one example
Upper moon, point 12
This lab2 numbers → hidden views → label
Visionpixels → visual features → prediction
Languagetokens → contextual representations → prediction
What you are seeing: a deterministic 2→2→2→2→1 tanh network trained on 48 synthetic examples. The checkpoints are computed for this guide, not extracted from the 2015 paper. All accuracy figures describe these training examples.
This toy problem does not require three hidden layers. The extra depth makes the mechanism inspectable. Real vision, speech, and language problems need much richer, high-dimensional representations.
Feature engineering moved inside the model.
Deep learning coordinates the feature extractor and the final decision with one task objective.
Earlier pattern
inputraw data
human-designeddescriptors and features
learnedclassifier
Deep learning
inputraw data
learnedfeature hierarchy
learnedclassifier
Representation and decision can be optimized together for the task.
Human choices did not disappear. Researchers still choose the data, objective, architecture, augmentation, optimization, and evaluation. Learned features are shaped by all of those choices.
Depth was a convergence, not a single moment.
Neural networks, optimization, datasets, hardware, and scientific communities all shaped the field.
1986
Error reaches hidden layers
Backpropagation makes credit assignment through multiple trainable layers practical.
1989
Convolution learns from pixels
Early convolutional networks learn features for handwritten recognition.
2006
Pretraining revives depth
Layer-wise learning offers a practical route into deeper models.
2009
ImageNet changes the scale
A large labeled image database creates a demanding shared benchmark.
2012
Depth wins ImageNet
A GPU-trained convolutional network sharply improves large-scale recognition.
2015
The field comes into focus
LeCun, Bengio, and Hinton connect the mechanisms, evidence, and research direction.
2017→
Attention scales representations
Transformers and self-supervised pretraining extend the representation-learning wager.
Modern AI still stacks representations.
Transformers changed how layers mix information. Self-supervision changed where targets come from. Scale changed what the representations can support.
Dimension
2015 field
Modern foundation models
Training scope
Task-specific supervised systems
Broad pretraining followed by adaptation
Central architecture
Convolution and recurrence
Attention often plays the central role
Targets
Labels commonly supplied by people
Targets often derived from the data itself
Domain
Strong single-domain models
Multimodal and general-purpose models
Compute
One accelerator or small clusters
Large distributed training systems
The breakthrough was hierarchy.
Learning could build its own path from raw data to useful abstraction, one reusable transformation at a time.
signal
→
feature
→
composition
→
representation
→
capability
Read the field as it formed.
The historical distinctions, examples, equations, and modern comparison in this guide are grounded in these primary publications.
1989
Backpropagation Applied to Handwritten Zip Code Recognition
Yann LeCun and colleagues
Open paper
2006
A Fast Learning Algorithm for Deep Belief Nets
Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh
Open paper
2009
ImageNet: A Large-Scale Hierarchical Image Database
Jia Deng and colleagues
Open paper
2012
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton
Open paper
2013
Representation Learning: A Review and New Perspectives
Yoshua Bengio, Aaron Courville, and Pascal Vincent
Open DOI
2015
Deep Learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton
Open Nature
2017
Attention Is All You Need
Ashish Vaswani and colleagues
Open paper
The 2015 article is a review and synthesis. This guide does not present it as the invention of deep learning. The selected lineage is intentionally compact and cannot represent every contributing researcher, institution, dataset, or hardware advance.