Mixture of Experts: Sparse FFNs, Routers, and Shared Experts

Module 07 established that the FFN holds roughly two-thirds of a Transformer's parameters. Module 09 established that inference cost is what limits deployment.

MoE resolves the tension between them. Raschka's statement of the core idea:

And the crucial part:

Total parameters grow with the number of experts. Active parameters — the ones touched per token — stay roughly constant.

DeepSeek V3 makes the numbers concrete: 671B total parameters, 37B active per token. It carries the knowledge capacity of a 671B model at roughly the inference cost of a 37B one.

Note carefully: all 671B must still fit in memory. MoE saves compute and bandwidth per token, not storage. This is why MoE models are deployed on multi-GPU clusters despite modest active parameter counts.

The router is a single linear layer. That is genuinely all it is.

class Router(nn.Module):
    def __init__(self, d_model, n_experts, top_k):
        super().__init__()
        self.gate = nn.Linear(d_model, n_experts, bias=False)
        self.top_k = top_k

    def forward(self, x):
        # x: (n_tokens, d_model)
        logits = self.gate(x)                              # (n_tokens, n_experts)
        topk_logits, topk_idx = logits.topk(self.top_k, dim=-1)
        # softmax over the SELECTED experts only -> weights sum to 1
        topk_weights = F.softmax(topk_logits, dim=-1)
        return topk_idx, topk_weights, logits

Then dispatch and combine:

class MoELayer(nn.Module):
    def __init__(self, d_model, d_expert, n_experts, top_k, n_shared=0):
        super().__init__()
        self.router  = Router(d_model, n_experts, top_k)
        self.experts = nn.ModuleList(
            [SwiGLU(d_model, d_expert) for _ in range(n_experts)]
        )
        # shared experts run for EVERY token — see 12.4
        self.shared = nn.ModuleList(
            [SwiGLU(d_model, d_expert) for _ in range(n_shared)]
        )

    def forward(self, x):
        B, T, D = x.shape
        x_flat = x.view(-1, D)                             # (B*T, D)
        idx, weights, logits = self.router(x_flat)

        out = torch.zeros_like(x_flat)
        for e, expert in enumerate(self.experts):
            # which (token, slot) pairs chose expert e?
            tok, slot = (idx == e).nonzero(as_tuple=True)
            if tok.numel() == 0:
                continue
            out[tok] += weights[tok, slot, None] * expert(x_flat[tok])

        for expert in self.shared:                         # always-on
            out += expert(x_flat)

        return out.view(B, T, D), logits

Two details that matter:

• Softmax is over the selected k experts only, not all N. The combination weights sum to 1 over what was actually used.
• Routing is per token, not per sequence. Every token in a sentence may go to a different set of experts. This is why MoE routing is a load-balancing problem at all.

Why top-k and not top-1?

k = 1 (Switch Transformer) is cheapest but training is unstable — routing decisions are discrete, so a token's gradient path changes abruptly when its argmax flips. With k ≥ 2 the output is a blend, so the router receives gradient through multiple experts and transitions are smooth.

k = 8 is the current norm (DeepSeek V3, Qwen3). gpt-oss uses k = 4; Llama 4 uses k = 2.

The central failure mode of MoE, and the reason most of the machinery exists.

Nothing in the router's objective encourages spreading tokens out. Left alone it collapses: a few experts get chosen for nearly everything, get more gradient, become better, and get chosen even more. The rest are never selected, never trained, and become dead weight.

Auxiliary load-balancing loss

The standard fix (Switch Transformer). Add a term encouraging uniform usage:

The product is the trick. f_i carries the actual imbalance but is not differentiable; P_i is differentiable. Multiplying them gives a gradient that pushes probability away from experts already receiving many tokens. The sum is minimised when usage is uniform.

def load_balancing_loss(logits, top_k_idx, n_experts, alpha=0.01):
    probs = F.softmax(logits, dim=-1)                  # (n_tokens, n_experts)
    P = probs.mean(dim=0)                              # mean prob per expert

    one_hot = F.one_hot(top_k_idx, n_experts).float()  # (n_tokens, k, n_experts)
    f = one_hot.sum(dim=1).mean(dim=0)                 # fraction routed per expert

    return alpha * n_experts * torch.sum(f * P)

α is a real trade-off: too small and experts collapse; too large and the router balances at the expense of routing well.

Capacity factor

A systems constraint, distinct from the loss. For efficient batched execution each expert gets a fixed-size buffer:

A capacity factor of 1.0 means exactly even distribution. Real distributions are uneven, so values of 1.25–2.0 are typical.

Tokens arriving at a full expert are dropped — they skip that expert entirely and pass through via the residual connection only. Higher capacity factor means fewer drops but more wasted memory and compute on padding.

(Capacity factors apply to training and batched prefill. Newer implementations increasingly use dropless routing with variable-size grouped GEMMs.)

Loss-free balancing

Raschka notes DeepSeek V3 uses a bias-based approach instead: a per-expert bias term added to routing logits and adjusted during training to equalise load, with no auxiliary loss gradient. Avoids the α trade-off entirely. Arcee Trinity Large also introduces "a new MoE load-balancing strategy."

A DeepSeek design choice that Raschka highlights repeatedly.

Not new — it appeared in DeepSeek's 2024 MoE paper and the 2022 DeepSpeedMoE paper.

Why it helps, in Raschka's words:

Without a shared expert, every routed expert must independently learn the generic transformations all tokens need — wasteful duplication. Factoring that out frees the routed experts to specialise.

But not everyone agrees

Qwen3 removed shared experts, which earlier Qwen2.5-MoE had used. Raschka pressed on this, and got an answer from Junyang Lin, a Qwen3 developer:

Raschka's own guess was that with 8 routed experts (up from 2 in Qwen2.5-MoE) a shared expert became unnecessary for stability — while noting this "doesn't explain why DeepSeek V3 is still keeping their shared expert."

Then Qwen3-Next (Sept 2025) added one back, alongside 4× more experts — both changes Raschka had predicted as likely future directions.

Current state of play:

Genuinely contested. Raschka's own position: "in my opinion, shared experts are useful because they reduce redundancy among the other experts."

The other major design axis.

The DeepSeekMoE paper argues for more, smaller experts at constant total parameters. Splitting each expert into m smaller ones and activating m times as many gives the same FLOPs but far more routing combinations — C(256,8) is astronomically larger than C(16,2) — so specialisation can be much finer.

The 2026 configuration table

Several stories are visible in that table.

Llama 4 versus DeepSeek V3. Raschka: "Llama 4 Maverick uses a more classic MoE setup with fewer but larger experts (2 active experts with 8,192 hidden size each) compared to DeepSeek V3 (9 active experts with 2,048 hidden size each)." Also: DeepSeek uses MoE in every block except the first three; Llama 4 alternates MoE and dense blocks.

gpt-oss is the counter-trend. "gpt-oss has a surprisingly small number of experts (32 instead of 128), and only uses 4 instead of 8 active experts per token. However, each expert is much larger... This is interesting because the recent trends and developments point towards more, smaller models as being beneficial."

Grok 2.5 is deliberately old-fashioned. Eight large experts, "which reflects an older trend" — a rare look at a real production system from a year earlier.

Sparsity is increasing. MiniMax-M2 activates 4.37% of parameters per token versus Qwen3 235B-A22B's 9.36% — "twice as sparse as Qwen3" at comparable total size.

And sometimes it goes the other way. Mistral 3 Large adopted DeepSeek V3's architecture exactly, but "increased the size of the experts by a factor of 2 while decreasing the number of experts by the same factor." Trinity Large also made its DeepSeek-style MoE "coarser as that helps with inference throughput." Finer experts are better for quality; coarser ones are better for throughput.

A structural detail with a clear rationale.

DeepSeek V3 uses MoE in every transformer block except the first three. GLM-4.5 adopted the same choice. Raschka explains:

Early layers do generic, universal work — there is nothing to specialise on yet, and routing on unstable representations produces unstable routing.

Qwen3 ships both: seven dense models (0.6B → 32B) and two MoE models (30B-A3B, 235B-A22B). Raschka explains why:

Gemma 4 does the same — a 31B dense variant and a 26B-A4B MoE variant, with "relatively similar" benchmark performance.

The newest variant, from Nemotron 3 Super (March 2026). Experts operate in a compressed space: MoE inputs are "down-projected from 4096 to 1024 dimensions, the experts are applied, and then the outputs are up-projected back from 1024 to 4096."

Structurally this is MLA's trick (module 09) applied to the FFN rather than to attention: do the expensive work in a lower-dimensional latent space. Combined with MTP and Mamba-2 hybrid attention, Raschka reports Nemotron 3 Super hits "2x faster than Qwen3.5 122B-A10B" throughput at comparable quality.