Modern Architecture Case Studies

Raschka's opening question frames everything here:

By the end of this module you should be able to answer that yourself. The short version: the block from module 06 is unchanged. What varies is which variant fills each slot.

He also states the caveat that should accompany every comparison here:

Architecture is what we can see. It is not necessarily what makes a model good.

671B total / 37B active · 61 layers · MLA + MoE

The most influential open architecture of the period. Kimi K2, Mistral 3 Large, and GLM-5 all adopted parts of it.

The two decisions that define it

MLA over GQA. Raschka explains the reasoning via the DeepSeek-V2 ablations:

So MLA was not chosen only for the ~57× cache reduction — DeepSeek's own measurements say it is better, not merely cheaper. Raschka's summary: "MLA is a clever trick to reduce KV cache memory use while even slightly outperforming MHA in terms of modeling performance."

Fine-grained MoE with a shared expert. 256 experts of hidden size 2048, 8 routed active plus 1 always-on shared. Raschka on the shared expert:

Why it matters

37B active parameters delivering better benchmarks than the 405B dense Llama 3. That comparison is what made MoE the default for frontier open models in 2025.

The descendants

Raschka on Mistral's choice: "why change what ain't broke? A lot of the secret sauce these days is in the training pipeline as well as the inference scaling strategies."

400B total / 17B active · GQA + alternating MoE

The instructive counterpoint to DeepSeek V3: same family of ideas, different choices at nearly every knob.

Raschka's summary:

Note that DeepSeek V3 is ~68% larger in total but has more than twice the active parameters — the two models sit at very different sparsity levels.

Reading it

Every Llama 4 choice is the conservative one: GQA over MLA (simpler, well- supported kernels), few large experts (the older, better-understood MoE style), alternating rather than pervasive MoE (less routing instability). Meta optimised for deployability and reliability; DeepSeek optimised for efficiency at frontier scale.

Raschka's own conclusion is appropriately humble: "Given the many small differences between architectures, it is difficult to determine their exact impact on final model performance. The main takeaway, however, is that MoE architectures have seen a significant rise in popularity in 2025."

Dense: 0.6B → 32B · MoE: 30B-A3B, 235B-A22B

Qwen3 is the reference point Raschka compares nearly everything else against, because it ships at every size in both dense and sparse variants.

Deep and narrow

Comparing Qwen3 0.6B with Llama 3 1B, Raschka notes Qwen3 "is a deeper architecture with more layers, whereas Llama 3 is a wider architecture with more attention heads." The consequences he measured on an A100 with his own from- scratch implementations: Qwen3 has a smaller memory footprint but "a slower runtime (lower tokens/sec generation speed)" — depth cannot be parallelised.

Why both dense and MoE

The shared-expert story

Qwen3 dropped the shared expert that Qwen2.5-MoE had. Raschka asked; developer Junyang Lin replied:

Then Qwen3-Next (Sept 2025) reversed it: 4× more experts and a shared expert restored — both directions Raschka had predicted.

Qwen3-Next — where Qwen went

An 80B-A3B model, 3× smaller than 235B-A22B, and a significant departure:

The gated attention layers are GQA with three stability tweaks: a sigmoid output gate, zero-centered RMSNorm for QK-Norm, and partial RoPE. Raschka: "these are essentially just stability changes to GQA."

Qwen3-Coder-Next (Feb 2026) uses the identical architecture, trained from Qwen3-Next as a base, and reaches SWE-Bench Pro performance "roughly on par with Claude-Sonnet-4.5" — a good illustration of the module-13 point that post-training now drives most differentiation.

Gemma 3: 27B · Gemma 4: 31B dense + 26B-A4B MoE

Google's answer to long-context cost is neither MoE nor MLA. It is sliding window attention.

The sliding-window design

Gemma 2 used a 1:1 ratio with a 4096 window. Gemma 3 moved to 5:1 and shrank the window to 1024 — "this shifts the model's focus towards more efficient, localized computations."

The ablation Raschka cites shows "little to no impact on the LLM-generated output perplexity." Substantial cache reduction, essentially free.

The distinctive norm placement

Gemma is the only major family using both pre- and post-norm around each sublayer. Raschka's read:

Gemma 4's near-identical architecture

Raschka: Gemma 4 (31B) "looks pretty much unchanged compared to Gemma 3 (27B)." Two small changes:

1. In global layers, values = keys — reusing the key tensor as the value tensor, "which should result in further KV cache size reduction."
2. p-RoPE at 25% — only a quarter of frequency pairs get positional information, reducing "positional noise in long-context situations."

And the lesson he draws is the most important one in this module:

Gemma 4 (31B) ranks comparably to Qwen3.5-397B-A17B on the AI Arena leaderboard — though he immediately notes "arena scores are a bit problematic as they can be gamed and are biased towards human (style) preference," and cross-checks against standard benchmarks, where the leap holds.

Near-identical architecture, large quality jump. Architecture is not where the gains came from.

Gemma 3n — a different axis

The on-device variant uses Per-Layer Embedding (PLE): keep only a subset of parameters in GPU memory and stream modality-specific embeddings from CPU or SSD on demand. Plus MatFormer (Matryoshka Transformer) — one shared architecture sliceable into smaller independently-usable models, so you run only the part you need.

1T total / 32B active · MLA + MoE

Raschka:

At the time of writing it "may be the biggest LLM of this generation" — with the caveat that Google's 1.6T Switch Transformer "is an encoder-decoder architecture from a different generation" (module 08).

The Muon story

The genuinely novel element is the optimizer, not the architecture:

And his careful reading of the evidence — worth reproducing because it models good skepticism:

Kimi K2 Thinking (Nov 2025) has an unchanged architecture with context extended from 128k to 256k.

Kimi Linear — the other branch

A 48B model exploring linear attention: Kimi Delta Attention (channel-wise gated DeltaNet) in a 3:1 hybrid with MLA global layers, using NoPE in the MLA layers. Higher accuracy than Gated DeltaNet at the same speed, and much faster than pure MLA.

Raschka's caveat: it is "20x smaller than Kimi K2. It will be interesting to see if the Kimi team adopts this approach for their upcoming K3 model." Unproven at frontier scale.

Full table

What is universal in 2026

Every model in that table uses:

• Decoder-only architecture
• RMSNorm (never LayerNorm)
• SwiGLU or a gated variant (never plain ReLU/GELU)
• Residual connections around both sublayers
• RoPE or a deliberate variant of it
• Some form of KV reduction — GQA, MLA, sliding window, or linear

What genuinely varies

The practical payoff of this course. Open any model's config.json and answer:

Worked example — a hypothetical config:

{
  "num_attention_heads": 64,
  "num_key_value_heads": 8,
  "num_hidden_layers": 62,
  "hidden_size": 5120,
  "intermediate_size": 1536,
  "num_local_experts": 160,
  "num_experts_per_tok": 8,
  "n_shared_experts": 1,
  "first_k_dense_replace": 3,
  "sliding_window": null,
  "rope_theta": 1000000.0,
  "use_qk_norm": true
}

Reading: GQA with group size 8 · 62 layers · MoE with 160 fine-grained experts, top-8 plus one shared · first 3 layers dense · no sliding window · RoPE with a large base (long-context tuned) · QK-Norm on. This is GLM-4.5-shaped.