Encoder, Decoder, Encoder-Decoder, and Causal Masking

The 2017 paper describes an encoder-and-decoder model. Both halves are stacks of the module-06 block, but they are wired differently, and each half turned out to be useful on its own.

Decoder-only won. Every model in module 15 is decoder-only. The reasons are practical: one stack to train, one objective, and it scales cleanly.

Nothing new here — it is module 06's block, six deep, with unrestricted attention. Following the playlist's walkthrough of "how are you":

INPUT BLOCK (before the stack)
  1. tokenization       "how are you"  ->  [how] [are] [you]
  2. embedding          each token -> 512-dim vector
  3. positional encoding  add PE(0), PE(1), PE(2)
  -> x1, x2, x3          each (512,)

ENCODER BLOCK x6
  4. multi-head attention        -> z1, z2, z3
  5. add residual (x + z), LayerNorm
  6. feed-forward network        -> y1, y2, y3
  7. add residual, LayerNorm
  -> output feeds the next block

OUTPUT: 3 vectors of 512 dims -> sent to the decoder

Shape is (3, 512) at every single checkpoint. As the playlist puts it: hold on anywhere in the encoder and you get 3 × 512.

Because there is no mask, how attends to you and you attends to how. That bidirectionality is exactly what makes encoders good at understanding and unsuited to ordinary left-to-right decoding without changing its objective or mask. Iterative masked-token generation is a different valid generation method.

The sentence to understand

Video 81 opens with a claim worth unpacking carefully:

Autoregressive means each output is conditioned on previously generated outputs. Stock prediction is autoregressive: Friday's forecast depends on Thursday's and Wednesday's. An RNN decoder is autoregressive: each timestep's input is the previous timestep's output.

Ordinary autoregressive generation is sequential. To emit word 3 you must know words 1 and 2, and you only know them because you generated them. There is no unknown sampled prefix available in advance. Speculative verification can process proposals together without changing this factorization (module 14).

But at training time, something changes.

Teacher forcing

During training you already have the correct output sentence. Teacher forcing means feeding the ground-truth previous token at each step, not the model's own (possibly wrong) prediction.

step 1:  input [START]              -> model says "tum"    (wrong; correct is "hum")
step 2:  input [START, hum]         <- feed the CORRECT word from data, not "tum"
step 3:  input [START, hum, dost]   <- again from data

And here is the consequence: if every input comes from the dataset, no step depends on any other step's output. The sequential dependency is gone. All positions can be processed in parallel.

The playlist's arithmetic: a 300-word target means running the entire decoder 301 times for one training row. Multiply by 100,000 rows. Training would be impossibly slow.

But parallelism creates data leakage

Feed all tokens at once into ordinary self-attention and something breaks.

Recall module 03: the contextual embedding of a token is a weighted mixture of every token in the sequence. So when computing the representation for position 1 (hum), the mechanism happily mixes in position 2 (dost) and position 3 (hain).

Those are future tokens. At inference they do not exist yet. The model would be learning to predict dost using information that includes dost.

The playlist's word for this is exactly right: cheating. More formally, it is data leakage — the model has information at training time that it will not have at prediction time. Training loss looks great; real-world performance is terrible.

We are now stuck between two failures:

The fix: mask before softmax

We want position i to attend only to positions ≤ i. Looking at module 03's weight matrix, that means forcing the strictly-upper-triangular entries to zero.

Setting w = 0 directly would break the softmax normalization. Instead, add -inf to the scores before softmax, exploiting softmax(-inf) = 0:

scaled scores            mask (added)              masked scores
[ 3.0  2.1  0.8 ]        [  0   -inf  -inf ]       [ 3.0  -inf  -inf ]
[ 1.2  4.0  1.5 ]   +    [  0    0    -inf ]   =   [ 1.2   4.0  -inf ]
[ 0.9  2.2  3.1 ]        [  0    0     0   ]       [ 0.9   2.2   3.1 ]

                      softmax over each row
                              |
                              v
                    [ 1.00  0.00  0.00 ]
                    [ 0.0573  0.9427  0.0000 ]  <- future weight is exactly zero
                    [ 0.0730  0.2680  0.6590 ]  <- rounded values

Each row still sums to 1 — softmax renormalizes over the surviving entries. Row 1 attends only to itself; row 2 to positions 1–2; row 3 to all three.

Best of both worlds: all positions computed in one parallel pass, and no position can see its future.

import torch

def causal_mask(T, device=None):
    """Lower-triangular boolean mask: True = allowed."""
    return torch.tril(torch.ones(T, T, dtype=torch.bool, device=device))

# inside attention, after scaling:
scores = scores.masked_fill(~causal_mask(T, scores.device), float('-inf'))
weights = torch.softmax(scores, dim=-1)

For a square causal prefill, SDPA can select an eligible fused implementation:

out = F.scaled_dot_product_attention(Q, K, V, is_causal=True)

Masking applies at inference too

Video 84 calls out a misconception explicitly, and it is worth repeating because people get this wrong:

You could argue it is unnecessary — at step 3 you only have three tokens, so there is no future to hide. But the model was trained with masked attention patterns. Removing the mask at inference changes the computation the weights were fitted to, and quality drops. Same computation at train and test time. Always.

Padding masks are a different thing

Two masks, often combined, often confused:

The decoder's second attention sublayer is different, and video 82 is dedicated to it. In the architecture diagram you can spot it immediately: two arrows come from the encoder and one from below, whereas every other attention block has all three arrows from the same place.

The motivating question

Generating the third Hindi word depends on two things:

1. What the decoder has generated so far — handled by masked self-attention.
2. What the input sentence says — needs a mechanism relating two different sequences.

Self-attention relates a sequence to itself (module 03, §3.7). For two sequences you need cross-attention.

The mechanism

Identical to self-attention with one change in wiring:

Note the rectangular score matrix: (T_dec, T_enc), not square. Row i says how much output token i should draw from each input token. That matrix is exactly the alignment table you would draw by hand:

            we    are   friends
hum        0.80  0.10    0.10
dost       0.10  0.10    0.80
hain       0.15  0.75    0.10

The number of outputs equals the number of tokens in the decoder sequence, not the encoder's — a point the playlist emphasises because it trips people up.

It is Luong attention, again

The paper calls this "encoder-decoder attention" and says so directly:

Which closes the loop from module 01. Bahdanau/Luong attention computed alignment between decoder state and encoder states; cross-attention is the same operation with learned K/V projections and multiple heads. The lineage is explicit.

Where else it shows up

Cross-attention is the general tool for relating two sequences of any kind:

• Image captioning — image patches (K,V) → text (Q)
• Text-to-image (Stable Diffusion) — text (K,V) → image latents (Q)
• Speech recognition (Whisper) — audio (K,V) → text (Q)

Every multimodal system uses it somewhere.

Three sublayers, each with residual + norm:

After the sixth block, an output head converts vectors to words:

final vectors (T, 512)
   -> Linear (512, V)        V = vocabulary size, e.g. 10,000
   -> logits (T, 10000)
   -> softmax
   -> probability distribution over the vocabulary
   -> argmax (or sample)

Each of the V output neurons corresponds to one vocabulary word. Pick the highest-probability one (or sample — module 14).

The encoder behaves identically in both. All the difference is in the decoder.

Training — one parallel pass

encoder: "We are friends"  ->  3 context vectors
decoder input (teacher forced, right-shifted):  [START] hum dost
decoder target:                                  hum    dost hain

ALL positions processed simultaneously, causal mask applied.
Loss computed over all positions at once. One backward pass.

Right-shifting is what makes the targets line up: prepend [START] so that position i of the input predicts position i of the target.

Inference — one pass per token

step 1:  decoder input [START]                  -> "hum"
step 2:  decoder input [START, hum]             -> "dost"
step 3:  decoder input [START, hum, dost]       -> "hain"
step 4:  decoder input [START, hum, dost, hain] -> [END]  -> stop

Three details from video 84 that matter:

1. The encoder runs once. Its output is fixed and reused as cross-attention K/V at every decoding step.
2. The input grows by one token each step — 1, then 2, then 3.
3. Only the last position's vector goes to the output head. At step 2 the decoder produces two 512-dim vectors, but you already emitted a word for position 1. You send only the final vector and discard the rest.

That third point is the one people find confusing, and the playlist flags it: "you are carrying all the vectors till the very end, but at the very last point you send only the last one and forget it."

The waste this reveals

Look at what happens between step 2 and step 3. Both recompute the K and V vectors for [START] and hum — identical inputs, identical weights, identical results. Every step redoes all of the previous steps' key/value work.

For a 1000-token generation that is ~500,000 redundant vector computations.

The fix is the KV cache (module 11): store K and V once and append. That one optimisation reshapes the entire inference stack — and because cache size then becomes the bottleneck, it is also what motivates MQA, GQA, and MLA in module 09.

The original design has an encoder for the input and a decoder for the output. Modern LLMs collapse this: concatenate everything into one sequence and run a single causal stack.

[prompt tokens] [generated tokens]
 <---------- one causal sequence ---------->

The prompt is attended to by everything after it — which is what cross-attention did — but with one stack, one set of weights, one objective.

The cost: the prompt is processed causally rather than bidirectionally, so early prompt tokens cannot see later ones. In practice this loses less than the simplicity gains — and simplicity is what scales.

Raschka's framing is a nice historical marker. Discussing Kimi K2's 1-trillion parameters, he notes it "may be the biggest LLM of this generation" given that "Google's 1.6 trillion Switch Transformer is an encoder-decoder architecture from a different generation." Encoder-decoder is, for LLMs, a previous era.

Encoder-decoder is still correct where input and output are genuinely different modalities — Whisper (audio→text), translation systems, diffusion text conditioning.

For a document [BOS,a,b,EOS], inputs [BOS,a,b] predict [a,b,EOS]. The likelihood remains p(a|BOS)p(b|BOS,a)p(EOS|BOS,a,b) during parallel teacher forcing. Set padded target labels to -100 for cross entropy; mask padding keys separately. Packing unrelated documents also needs a document-equality condition in the causal mask, and ignored cross-boundary targets. One mask cannot substitute for all three contracts.

Cross-attention uses decoder queries of length U and encoder keys/values of length T, giving a rectangular (U,T) score matrix. Encoder K/V can be computed once per source sequence and reused at every decoder step. Source padding is masked, but a left-to-right causal restriction is not generally applied across source columns. Decoder self-attention remains causal independently.

import torch
import torch.nn.functional as F
torch.manual_seed(8)
scores = torch.tensor([[3., 2.1, .8], [1.2, 4., 1.5], [.9, 2.2, 3.1]])
allowed = torch.ones(3, 3, dtype=torch.bool).tril()
a = scores.masked_fill(~allowed, float('-inf')).softmax(-1)
torch.testing.assert_close(a[1], torch.tensor([.05732418, .94267582, 0.]))
assert a[0, 1:].count_nonzero() == 0
q, k, v = [torch.randn(1, 1, 3, 4) for _ in range(3)]
baseline = F.scaled_dot_product_attention(q, k, v, attn_mask=allowed)
changed = v.clone()
changed[:, :, 2] += 100
torch.testing.assert_close(baseline[:, :, :2],
    F.scaled_dot_product_attention(q, k, changed, attn_mask=allowed)[:, :, :2])
decoder_q = torch.randn(1, 1, 2, 4)
source_k, source_v = torch.randn(1, 1, 5, 4), torch.randn(1, 1, 5, 4)
source_valid = torch.tensor([True, True, True, False, False])
cross = F.scaled_dot_product_attention(decoder_q, source_k, source_v, attn_mask=source_valid[None, :])
unpad = F.scaled_dot_product_attention(decoder_q, source_k[:, :, :3], source_v[:, :, :3])
torch.testing.assert_close(cross, unpad)
assert cross.shape == (1, 1, 2, 4)
print("Causal arithmetic, future isolation and rectangular source padding verified.")

SDPA boolean masks use True = allowed. nn.MultiheadAttention boolean key-padding masks instead use True = ignored. Name tensors allowed or padding rather than relying on a generic mask variable's implied polarity. Teacher forcing also creates an exposure mismatch: evaluation histories include model mistakes; this is distinct from future-token leakage.

Download train_tiny_translation.py. This is an actual optimization and evaluation experiment using torch.nn.Transformer, not randomly initialized output presented as translation. It runs on CPU with no dataset or checkpoint downloads; the tested environment uses PyTorch 2.8.

python content/courses/transformers/code/train_tiny_translation.py
python content/courses/transformers/code/train_tiny_translation.py --output-dir /tmp/course-translation
python site/test_transformer_training.py

The default run writes its reload-check checkpoint to a temporary directory and removes it afterward. --output-dir retains manifest.json, checkpoint.pt and evaluation.json. The test command also covers the tokenizer lab and therefore requires the optional tokenizers library as well as PyTorch.

A controlled word-order problem

The task translates English color/shape phrases into a deliberately restricted French vocabulary. a red circle becomes un cercle rouge: the adjective moves after the noun. The six isolated color and noun translations are also examples, so the batch contains genuinely different sequence lengths. Accents and wider French agreement are outside this ASCII toy vocabulary.

Training contains six isolated words and six three-word combinations. The three diagonal combinations are held out: a red square, a blue circle, and a green triangle. No held-out pair appears in training. Both vocabularies are built from training text only; every held-out word nevertheless occurs in another training phrase. Unknown words raise an explicit error rather than silently mapping to an unspecified unknown token.

This split asks whether the model recombines known words in unseen pairs. It is not an independent natural-language test set. The grammar, vocabulary and split were constructed by us, and three examples cannot estimate translation quality reliably. The script reports all three predictions rather than hiding them behind a large-looking aggregate percentage. It does not tune hyperparameters or stop training according to this split.

Follow one padded batch

Source sequences end in EOS. Targets contain BOS and EOS before padding. For a short isolated-word example beside a longer phrase:

source:       red EOS PAD PAD
              a blue square EOS

full target:  BOS rouge EOS PAD PAD
              BOS un carre bleu EOS

decoder in:   BOS rouge EOS PAD
              BOS un carre bleu

labels:       rouge EOS PAD PAD
              un carre bleu EOS

The decoder input is full_target[:, :-1]; labels are full_target[:, 1:]. BOS is never a target. EOS is a supervised target, so the model learns when to finish. PAD labels are excluded from cross entropy using ignore_index=PAD; this lab uses the actual PAD ID, whereas the preceding worked example used -100. Either is valid when collator and loss agree.

The objective is the sum of negative log probabilities over valid target positions divided by the number of valid target tokens. It is not the mean of per-sequence losses. Longer examples consequently contribute more terms. The test compares masked cross entropy against explicitly selecting valid positions, and checks that changing logits at ignored positions cannot change the loss.

Four independent attention constraints

These are boolean masks for nn.Transformer, where True means forbidden. That polarity is opposite to boolean masks passed directly to functional SDPA. The library also handles projection, attention and feed-forward layers; the lab concentrates on data, masks and training instead of reimplementing those kernels. See the PyTorch Transformer API.

A key-padding mask does not force every padded query output to zero. A padded target position may produce arbitrary finite logits; ignoring its label is therefore still necessary. Similarly, masking source padding in the encoder does not remove the need to mask those memory positions in cross-attention. The position embedding added to a PAD token can be nonzero even when the token embedding uses padding_idx.

The regression suite perturbs future target tokens and verifies that earlier logits do not change. It appends source padding and compares logits with the unpadded computation. It also compares a short example alone with its valid positions inside a padded mixed-length batch. These checks test information flow, not just tensor shapes. Floating-point comparisons use small tolerances because equivalent padded and unpadded library paths need not be bit-identical.

Training and evaluation are different programs

The network has one encoder layer, one decoder layer, width 32, four heads, feed-forward width 64, learned positions, and dropout zero. It trains for 250 full-batch AdamW updates at learning rate 0.015 with gradient norm clipping at 1.0. Seed 808 and one CPU thread make the experiment reproducible in the tested environment. These are tiny-data teaching settings, not a recommended schedule for a real translation system; dropout is disabled to isolate the contracts.

Training uses ground-truth target prefixes. Autoregressive evaluation starts from BOS and repeatedly supplies the model's own previous predictions. Greedy decoding excludes PAD and BOS as new outputs. A completed sequence gets PAD while other batch members continue. Decoding stops when every sequence has produced EOS or the generation limit is reached. The report records ended; an unterminated text prefix is not counted as an exact match merely because its visible words happen to agree. Exact match includes EOS.

For clarity, this small implementation recomputes the encoder and decoder prefix each generation step. It does not implement an efficient serving cache. Reusing encoder memory is an immediate optimization; decoder KV reuse requires preserving the same positions and masks described in hardware-aware attention.

One local PyTorch 2.8 CPU run produced:

Initial training cross entropy was 2.390391. The missed combination was a green triangle, decoded as un triangle rouge. The model learned a good training solution without fully learning the intended compositional rule. Backend/version differences can change this small run; tests require successful training and internally correct metric accounting, not a predetermined held-out score. Teacher-forced token accuracy is not interchangeable with free-running exact match, and a near-zero training loss is not evidence that the model understands a language.

Reload more than a weight matrix

The manifest records architecture geometry, ordered source/target vocabularies, special-token IDs through their ordered entries, update count, seed and PyTorch version. The checkpoint contains model state, AdamW state and CPU RNG state. Reload constructs the architecture from the manifest, validates vocabulary sizes and special-token ordering, loads tensors on CPU with weights_only=True, and restores optimizer state. The script then reproduces evaluation from the disk artifact. Tests compare one subsequent optimizer update with the original state, not merely two forward passes.

Changing the vocabulary order while keeping its size would silently relabel embedding rows unless the same manifest travels with the checkpoint. Treat the files as one trusted artifact. weights_only=True narrows deserialization but is not a sandbox for arbitrary hostile files; this exercise only reloads files it creates. It does not provide signed artifacts or a hardened upload endpoint.

For this full-batch, dropout-free CPU run the saved state is sufficient to reproduce the next update. A production resume also needs sampler position, data/version identifiers, scheduler and scaler state when present, distributed RNG state, and any gradient-accumulation progress. A seed alone is not a complete resume protocol. See PyTorch training workflows and the decoder training capstone.

Extend the experiment without invalidating it

1. Add a validation split separate from a final test set before choosing steps, width or learning rate. Repeatedly inspecting the three held-out phrases turns them into development data.
2. Remove one mask at a time and run the isolation tests. Source PAD masking, future masking and ignored target loss fail for different reasons; explain each failure before looking at the accuracy.
3. Add longer phrases with multiple modifiers and hold out a length regime. Report length-specific results rather than mixing easy and hard examples. Learned position tables impose a hard configured maximum.
4. Compare teacher forcing with a free-running error trace. Identify the first wrong token, then separate downstream errors caused by that prefix from errors already present under the correct prefix.
5. Move to a versioned real parallel corpus, tokenizer artifacts trained only on training text, and established corpus-level translation metrics. Include preprocessing and metric signatures, multiple seeds and uncertainty; this toy exact-match score is not a substitute for that evaluation.