Motivation & History: From RNNs to "Attention Is All You Need"

A sequence-to-sequence task takes a sequence in and emits a sequence out. Machine translation is the canonical one, but the family is large:

The playlist frames this crisply: neural architectures are specialised by data shape. ANNs handle tabular data, CNNs handle images, RNNs handle sequences — and Transformers were built specifically for sequence-to-sequence. The name is literal: they transform one sequence into another.

A recurrent network processes tokens one at a time, carrying a hidden state forward:

h_t = f(h_{t-1}, x_t)

This is a genuinely good idea. Order is baked into the computation for free: if you feed "Nitesh killed lion" one word per timestep, the network structurally knows which came first. Nothing extra is needed to represent position.

The encoder–decoder (Sutskever et al., 2014) applied this to seq2seq: an encoder LSTM reads the whole input and compresses it into a final hidden state called the context vector; a decoder LSTM reads that vector and emits the output token by token.

Everything the encoder understood about the input must fit into one fixed-size vector. The playlist puts the practical threshold at around 30 words — beyond that, translation quality visibly degrades. You are asking a fixed set of numbers to summarise an arbitrarily long sentence, and it cannot.

The attention fix (Bahdanau 2014, Luong 2015)

Instead of one context vector shared by every decoder step, compute a different context vector for each output position:

where \(h_j\) are the encoder hidden states and \(\alpha_{ij}\) are weights saying how useful encoder position \(j\) is for producing output position \(i\). Those weights come from a softmax over alignment scores:

The two variants differ only in how \(e_{ij}\) is computed:

Remember Luong's dot product. Module 03 is going to rebuild it from scratch under a different name.

If the input has 4 words and the output has 4 words, there are 16 alphas — one per (output position, input position) pair.

Attention worked. Translation quality stopped collapsing on long sentences. But it left the second failure completely untouched.

h_t depends on h_{t-1}. You cannot compute timestep 500 until you have computed timestep 499. A 10,000-token document requires 10,000 sequential steps. GPUs are wide parallel machines; a strictly serial dependency chain wastes almost all of that width.

This is the failure that mattered more, and it is the one that motivated Transformers. As the playlist puts it: the encoder can process all words in the sentence simultaneously, and because of this the architecture is scalable — it can be trained on very large datasets. Scale, not accuracy, was the unlock.

Two failure modes, two fixes, in historical order:

Attention Is All You Need (Vaswani et al., 2017) kept the encoder-decoder skeleton and kept attention — and deleted the LSTMs. The title is a claim about what is sufficient, and the removal is the contribution.

The playlist's one-sentence summary is a good one to memorise:

That substitution buys parallelism, and it costs one thing: order information disappears. If every token is processed simultaneously with no recurrence, nothing distinguishes "Nitesh killed lion" from "Lion killed Nitesh". Module 05 is entirely about paying that bill.

The trade in one table

Note the last two rows: the Transformer trades a quadratic cost in sequence length for parallelism. In 2017 with 512-token contexts that was obviously worth it. At 256k context it is the central problem of the field — which is what modules 09–11 are about.

The authors built Transformers to win at machine translation. The playlist is blunt that they had no idea what they had:

• NLP was solved-ish, fast. Fifty years of progress — rule systems, Naive Bayes, HMMs, bag-of-words, n-grams, then LSTMs — got compressed into about five years.
• Transfer learning arrived in NLP. BERT and GPT were pretrained on huge corpora and released publicly. A startup could fine-tune on a small dataset and get state-of-the-art results — previously near-impossible. Libraries like Hugging Face made that three or four lines of code.
• Modality-independence. The architecture only requires that your data be a sequence of vectors. Get images, audio, or video into that form and the same machinery applies. Every multimodal model you use runs on this observation.

Raschka's framing is the useful corrective to hype. Comparing GPT-2 (2019) with DeepSeek V3 and Llama 4, one is "surprised at how structurally similar these models still are." The changes since 2017 are real but local:

• absolute positional embeddings → RoPE (module 05)
• Multi-Head Attention → Grouped-Query Attention (module 09)
• GELU → SwiGLU (module 07)
• LayerNorm → RMSNorm (module 06)
• dense FFN → Mixture-of-Experts (module 12)

His question — "have we truly seen groundbreaking changes, or are we simply polishing the same architectural foundations?" — is the thesis this course inherits. The skeleton from module 06 is the skeleton of everything in module 15.