The FFN / MLP Layer: GELU, SwiGLU, and Where the Parameters Live

Attention gets the attention. But the second sublayer — the feed-forward network — holds most of the parameters and is where a surprising amount of the model's knowledge lives.

Its structure is simple. Two linear layers with a nonlinearity between:

FFN(x) = activation(x · W_1 + b_1) · W_2 + b_2

The playlist walks the exact shapes from the paper:

So d_ff = 4 · d_model. That 4× ratio held for years and is still a common default.

This is the property that makes the FFN the exact complement of attention, and it is easy to miss.

The FFN sees one token vector at a time. It has no access to other positions. Feeding (T, 512) through it is just T independent applications of the same function — batched for efficiency, not because they interact.

That division of labour is the design:

Attention decides what information to gather; the FFN decides what to do with it. Alternate them enough times and you get a language model.

A fair objection: we go 512 → 2048 → 512 and end where we started. What was gained?

Nonlinearity, in a high-dimensional space. Without the ReLU the whole thing would collapse — W_2(W_1 x) is just (W_2 W_1) x, a single linear map. The activation is the point; the expansion gives it room to work.

The playlist makes an important structural observation here: self-attention is almost entirely linear. Projections are linear, the score matmul is bilinear, the weighted sum is linear. Softmax is the only nonlinearity, and it acts on weights, not on content. So if the FFN were removed, a Transformer would have very little capacity to represent nonlinear relationships in the data.

The FFN is where nonlinearity lives.

Where the parameters live

Count, for d_model = 512, d_ff = 2048, per block:

The FFN has twice the attention's parameters. Raschka's account is consistent: the FFN "typically contains a large number of the model's total parameters," and the paper the playlist cites states plainly that "feed-forward layers constitute two-thirds of the Transformer model parameters."

This single fact explains module 12 entirely. If you want to grow model capacity without proportionally growing inference cost, you attack the largest parameter block. That is the FFN. Mixture-of-Experts is the FFN, replicated and sparsely routed.

Why does the FFN work? The playlist is candid: "this is a gray area... not much has been explained in the paper and it is an area of active research."

The standard answer is "to add nonlinearity". A more interesting one comes from Transformer Feed-Forward Layers Are Key-Value Memories (Geva et al., 2021), which the playlist flags. The claim:

Read the FFN as an attention-like lookup with learned, static keys and values:

FFN(x) = sum over neurons j of  activation(x · W_1[:, j]) * W_2[j, :]
              \_______________________________/    \____________/
                   "how much does x match             "what this
                     pattern j?"  (the KEY)          neuron writes"
                                                        (the VALUE)

Each of the 2048 hidden neurons is a pattern detector; when it fires it writes its learned value vector into the residual stream. Where attention retrieves from the current sequence, the FFN retrieves from parameters — from what the model learned during pretraining.

That framing makes the FFN the model's factual memory, and it makes MoE's logic obvious: if the FFN is a lookup table, you can grow the table without reading all of it every time.

ReLU (2017)

Cheap, non-saturating for positive inputs, sparse (~50% zeros). Its flaw is the dying ReLU problem: a neuron whose pre-activation is always negative outputs zero forever, and its gradient is zero forever too. It never recovers.

GELU (BERT, GPT-2/3)

where Φ is the standard Gaussian CDF. Intuition: instead of a hard gate (keep/zero), weight the input by the probability that it exceeds a random threshold. The result is smooth, and slightly negative for small negative inputs rather than exactly zero — so gradients survive.

The common tanh approximation:

import torch.nn.functional as F
y = F.gelu(x)                      # exact
y = F.gelu(x, approximate='tanh')  # the approximation, used by GPT-2

SwiGLU (Llama, PaLM, and everything since)

The current standard, and structurally different from the two above — it changes the shape of the FFN, not just the nonlinearity.

Start with Swish (a.k.a. SiLU):

Similar to GELU, cheaper. Then add a gate. A Gated Linear Unit computes two projections and multiplies them elementwise, so one branch controls how much of the other passes through:

Three matrices, not two. To keep the parameter count comparable, d_ff is reduced from 4·d_model to roughly 8/3 · d_model ≈ 2.67 · d_model — usually rounded to a hardware-friendly multiple. That is why you see intermediate sizes like 11008 for Llama's d_model = 4096 rather than 16384.

class SwiGLU(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.w_gate = nn.Linear(d_model, d_ff, bias=False)
        self.w_up   = nn.Linear(d_model, d_ff, bias=False)
        self.w_down = nn.Linear(d_ff, d_model, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))

Note bias=False throughout. Modern models drop FFN biases — they cost parameters and contribute nothing measurable.

Why gating helps is not fully settled. Noam Shazeer's paper introducing SwiGLU ends with a famously honest line: the architectures' success is attributed "to divine benevolence." The working explanation is that multiplicative interaction adds expressiveness a single nonlinearity cannot, letting the network learn input-dependent filtering. Empirically it wins consistently, so everyone uses it.

Comparison

Raschka's one-line summary of the shift: "the more efficient SwiGLU has replaced activation functions like GELU."

One holdout

gpt-oss uses a gated activation but keeps bias units in the attention layers — something Raschka says he hasn't "seen... since the GPT-2 days," and which is "commonly regarded as redundant." He cites a paper showing this mathematically for the key projection, with empirical results finding "little difference between with and without bias units." GLM-4.5 also retains attention bias. Curiosities, not trends.

Raschka notes Olmo 3's unusually high 5.4× ratio and reads it as deliberate: the team likely "scaled up the intermediate size expansion from 5x in Qwen3 to 5.4 in Olmo 3 to have a 32B model for a direct comparison" — the ratio was tuned to hit a target parameter count, not chosen on principle. A useful reminder that not every hyperparameter encodes a deep insight.

In MoE models the picture changes completely — expert FFNs are much narrower (DeepSeek V3: 2048; Qwen3: 768) but there are hundreds of them. Module 12.