The FFN / MLP Layer: GELU, SwiGLU, and Where the Parameters Live

Attention gets the attention. But the second sublayer — the feed-forward network — holds most of the parameters and is where a surprising amount of the model's knowledge lives.

Its structure is simple. Two linear layers with a nonlinearity between:

FFN(x) = activation(x · W_1 + b_1) · W_2 + b_2

The playlist walks the exact shapes from the paper:

So d_ff = 4 · d_model. That 4× ratio held for years and is still a common default.

This is the property that makes the FFN the exact complement of attention, and it is easy to miss.

The FFN sees one token vector at a time. It has no access to other positions. Feeding (T, 512) through it is just T independent applications of the same function — batched for efficiency, not because they interact.

That division of labour is the design:

Attention decides what information to gather; the FFN decides what to do with it. Alternate them enough times and you get a language model.

A fair objection: we go 512 → 2048 → 512 and end where we started. What was gained?

Nonlinearity, in a high-dimensional space. Without the ReLU the whole thing would collapse — W_2(W_1 x) is just (W_2 W_1) x, a single linear map. The activation is the point; the expansion gives it room to work.

Attention is already nonlinear in its input: QK.T multiplies two input-dependent projections, and softmax changes the weights as the input changes. Holding weights fixed makes aggregation linear in V, not linear in X. The FFN adds a large positionwise nonlinear feature transformation; attention's projections also mix features while its aggregation mixes tokens.

Where the parameters live

Count, for d_model = 512, d_ff = 2048, per block:

The FFN has twice the attention's parameters. Raschka's account is consistent: the FFN "typically contains a large number of the model's total parameters," and the paper the playlist cites states plainly that "feed-forward layers constitute two-thirds of the Transformer model parameters."

This single fact explains module 12 entirely. If you want to grow model capacity without proportionally growing inference cost, you attack the largest parameter block. That is the FFN. Mixture-of-Experts is the FFN, replicated and sparsely routed.

Why does the FFN work? The playlist is candid: "this is a gray area... not much has been explained in the paper and it is an area of active research."

The standard answer is "to add nonlinearity". A more interesting one comes from Transformer Feed-Forward Layers Are Key-Value Memories (Geva et al., 2021), which the playlist flags. The claim:

Read the FFN as an attention-like lookup with learned, static keys and values:

FFN(x) = sum over neurons j of  activation(x · W_1[:, j]) * W_2[j, :]
              \_______________________________/    \____________/
                   "how much does x match             "what this
                     pattern j?"  (the KEY)          neuron writes"
                                                        (the VALUE)

Each of the 2048 hidden neurons is a pattern detector; when it fires it writes its learned value vector into the residual stream. Where attention retrieves from the current sequence, the FFN retrieves from parameters — from what the model learned during pretraining.

That framing makes the FFN the model's factual memory, and it makes MoE's logic obvious: if the FFN is a lookup table, you can grow the table without reading all of it every time.

ReLU (2017)

Cheap, non-saturating for positive inputs, sparse (~50% zeros). Its flaw is the dying ReLU problem: a neuron whose pre-activation is always negative outputs zero with zero local activation derivative on those inputs. Upstream features can change and reactivate it, so permanence is not guaranteed.

GELU (BERT, GPT-2/3)

where Φ is the standard Gaussian CDF. Intuition: instead of a hard gate (keep/zero), weight the input by the probability that it exceeds a random threshold. The result is smooth, and slightly negative for small negative inputs rather than exactly zero — so gradients survive.

The common tanh approximation:

import torch.nn.functional as F
y = F.gelu(x)                      # exact
y = F.gelu(x, approximate='tanh')  # the approximation, used by GPT-2

<span id="swiglu-llama-palm-and-everything-since"></span>

SwiGLU (Llama, PaLM, and many later models)

The current standard, and structurally different from the two above — it changes the shape of the FFN, not just the nonlinearity.

Start with Swish (a.k.a. SiLU):

Similar to GELU, cheaper. Then add a gate. A Gated Linear Unit computes two projections and multiplies them elementwise, so one branch controls how much of the other passes through:

Three matrices, not two. To keep the parameter count comparable, d_ff is reduced from 4·d_model to roughly 8/3 · d_model ≈ 2.67 · d_model — usually rounded to a hardware-friendly multiple. That is why you see intermediate sizes like 11008 for Llama's d_model = 4096 rather than 16384.

class SwiGLU(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.w_gate = nn.Linear(d_model, d_ff, bias=False)
        self.w_up   = nn.Linear(d_model, d_ff, bias=False)
        self.w_down = nn.Linear(d_ff, d_model, bias=False)

    def forward(self, x):
        return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))

This example chooses bias=False, as many models do. Bias removal is not universal: the gpt-oss reference MLP includes both expert projection biases and a modified gated activation.

Why gating helps is not fully settled. Noam Shazeer's paper introducing SwiGLU ends with a famously honest line: the architectures' success is attributed "to divine benevolence." The working explanation is that multiplicative interaction adds expressiveness a single nonlinearity cannot, letting the network learn input-dependent filtering. Its empirical success does not make it universal: Gemma 3 uses a GELU-tanh-based gated MLP, not SwiGLU.

Comparison

Raschka's one-line summary of the shift: "the more efficient SwiGLU has replaced activation functions like GELU."

One holdout

gpt-oss uses a gated activation but keeps bias units in the attention layers — something Raschka says he hasn't "seen... since the GPT-2 days," and which is "commonly regarded as redundant." He cites a paper showing this mathematically for the key projection, with empirical results finding "little difference between with and without bias units." GLM-4.5 also retains attention bias. Curiosities, not trends.

Raschka notes Olmo 3's unusually high 5.4× ratio and reads it as deliberate: the team likely "scaled up the intermediate size expansion from 5x in Qwen3 to 5.4 in Olmo 3 to have a 32B model for a direct comparison" — the ratio was tuned to hit a target parameter count, not chosen on principle. A useful reminder that not every hyperparameter encodes a deep insight.

In MoE models the picture changes completely — expert FFNs are much narrower (DeepSeek V3: 2048; Qwen3: 768) but there are hundreds of them. Module 12.

Let a=xW_gate, b=xW_up, and h=silu(a)*b. For an upstream gradient g at h, dL/da=g*b*silu'(a) and dL/db=g*silu(a), where silu'(a)=sigmoid(a)+a*sigmoid(a)*(1-sigmoid(a)). The gate is not restricted to [0,1]; SiLU may be negative or exceed one. GEGLU changes SiLU to GELU while retaining the three-matrix structure.

import torch
torch.manual_seed(7)
a = torch.randn(5, dtype=torch.float64, requires_grad=True)
b = torch.randn(5, dtype=torch.float64, requires_grad=True)
g = torch.randn(5, dtype=torch.float64)
loss = (torch.nn.functional.silu(a) * b * g).sum()
ga, gb = torch.autograd.grad(loss, (a, b))
s = a.sigmoid()
torch.testing.assert_close(ga, g * b * (s + a * s * (1 - s)))
torch.testing.assert_close(gb, g * torch.nn.functional.silu(a))
d, classic_width, gated_width = 12, 48, 32
assert 2 * d * classic_width == 3 * d * gated_width
print("Gate gradients agree; matched budgets require different inner widths.")

A two-matrix FFN costs 2*d*m weights, a gated FFN 3*d*g. Matched parameter and dominant projection-FLOP budgets therefore use g=2m/3, subject to kernel alignment. Equal inner width instead gives the gated model 50% more projection weights, confounding an activation comparison.

Parameter exercise: with d=4096, H=32, H_kv=8, d_head=128, and gated width 14336, bias-free GQA projections contain 41,943,040 weights and the FFN 176,160,768: about 80.8% of those per-block projection weights, not two-thirds. Add norms, embedding/output tables, layer count and biases for a whole-model fraction. MoE must separate total expert weights from active ones. The FFN memory interpretation describes learned feature/value associations; it is not evidence that all model facts reside in FFNs rather than distributed attention and residual computations.