Every Token Learned Where to Look

Context no longer had to travel step by step.

Editorial reconstruction of a 2017 machine-learning lab with GPU workstations, translation diagrams, and attention matrices — Editorial reconstruction Translation becomes a matrix problem

Recurrent sequence models processed positions in order. Earlier attention systems improved translation by letting a recurrent decoder select different parts of the input, but recurrence still organized the computation.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin asked a sharper question: could attention itself carry the sequence model?

Their 2017 paper replaced sequence-aligned recurrence and convolution with stacked attention and feed-forward blocks. Its big model reached 28.4 BLEU on English-to-German translation after 3.5 days on eight P100 GPUs.

The old path

Recurrent state carried information through the sequence one position at a time.

The new path

Every position could compare itself directly with every available position.

The original task

The Transformer began as an encoder-decoder translation model, not a chatbot.

Attention did not begin in 2017. Bahdanau, Cho, Bengio, and others had already used learned attention with recurrent models. The Transformer made attention the central routing computation.

Attention turns relevance into a weighted mixture.

One position compares a query with every key, then gathers the corresponding values.

start token + position what and where

ask query what do I need?

match query x keys compatibility scores

normalize softmax weights sum to one

gather weighted values information is mixed

result contextual token a new representation

Attention(Q, K, V) = softmax(QK^T / √d_k)V Compare queries with keys, normalize the scores, then mix the values.

Ask

The query describes what the current position is looking for.

Match

Dot products compare that query with every available key.

Weight

Softmax turns signed scores into positive weights that sum to one.

Gather

The values are blended according to those weights.

Attention is not the entire Transformer. A complete block also needs feed-forward layers, residual paths, normalization, positional information, and learned projections.

Interactive attention lab

Let one token look around.

Choose a query, reveal its scores, and watch context become a weighted mixture.

Calculation phase

Decoder behavior

Birds cross the river

query: cross

Choose the token that will ask for context.

Query vector q_cross = [1.000, 1.000]

QK^T / √d_k softmax weighted V

Inspect every position Ask: what does cross need?

Σ weights = 1.000

Contextual output [0.456, 0.518]

For cross, river is the strongest match. Every available value still contributes to the mixture.

What you are seeing: one attention head with tiny hand-set two-dimensional vectors. They make every calculation inspectable. Real models learn much larger projections, and individual coordinates do not have guaranteed human-readable meanings.

Masking happens before softmax. A causal decoder replaces future scores with minus infinity, so their weights become zero. During generation, the model still produces one new token after another.

Attention is the mixer. The Transformer is the stack.

The original model paired attention with position, residual paths, normalization, and per-token transformations.

6 + 6encoder and decoder layers

512model dimensions

8attention heads

65Mbase parameters

Read the source

Encoder

Token embedding + position

Multi-head self-attention

Add + normalize

Position-wise feed-forward

Add + normalize

repeat across six layers

keys + values

Write the target

Decoder

Shifted output + position

Masked self-attention

Add + normalize

Encoder-decoder attention

Feed-forward + residual path

Linear projection + softmax

Position preserves order

Self-attention alone does not know which token came first.

Heads provide different views

Each head uses separate learned projections and can route different relationships.

The mask protects the future

Decoder positions cannot inspect output tokens that have not been generated.

Feed-forward layers transform

Attention mixes positions. The following network transforms each position independently.

The Transformer joined several ideas at once.

Recurrence, learned alignment, deep representations, residual learning, and accelerated matrix operations all shaped the result.

1986

Error reaches hidden layers

Backpropagation assigns credit through a trainable stack.

1997

Gates extend recurrent memory

LSTM improves learning across longer sequences.

2014

Sequences map to sequences

Encoder-decoder networks scale neural translation.

2014/15

Attention learns alignment

A recurrent decoder searches the source for relevant positions.

2015

Deep representations mature

Data, compute, and end-to-end learning converge at scale.

2017

Attention becomes the architecture

The Transformer removes sequence-aligned recurrence and convolution.

2018+

Pretraining scales the backbone

Encoder-only and decoder-only Transformers become foundation-model families.

Modern AI still routes context with attention.

The 2017 blueprint became a family of encoders, decoders, multimodal systems, and large pretrained models.

Dimension

2017 Transformer

Modern foundation models

Primary use

Supervised machine translation

Broad pretraining followed by adaptation

Stack

Encoder-decoder

Encoder-only, decoder-only, and multimodal variants

Learning signal

Translated sentence pairs

Prediction targets derived from large datasets

Attention pattern

Full encoder attention and causal decoder attention

Full, causal, local, sparse, and long-context variants

Scale

Six-layer encoder and decoder stacks

Many deeper and wider architecture families

The breakthrough was learned relevance.

Each position could build its next representation from the parts of the sequence most useful to it.

token query weights context next representation

Read the sequence as it changed.

The mechanism, historical distinctions, architecture details, and modern bridge in this guide come from these primary publications.

1997 Long Short-Term Memory Sepp Hochreiter and Jurgen Schmidhuber Open paper 2014 Sequence to Sequence Learning with Neural Networks Ilya Sutskever, Oriol Vinyals, and Quoc Le Open paper 2014 Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio Open paper 2015 Deep Learning Yann LeCun, Yoshua Bengio, and Geoffrey Hinton Open Nature 2017 Attention Is All You Need Ashish Vaswani and colleagues Open paper 2018 BERT: Pre-training of Deep Bidirectional Transformers Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova Open paper 2020 Language Models are Few-Shot Learners Tom Brown and colleagues Open paper

The selected lineage is intentionally compact. It does not represent every researcher, architecture, dataset, training method, hardware advance, or critique that shaped attention-based machine learning.