Modern Architecture Case Studies

Raschka's opening question frames everything here:

By the end of this module you should be able to answer that yourself. The short version: many building blocks from module 06 recur, but the complete block is not unchanged. Hybrid models can replace attention with recurrent mixers, split mixing and FFN into separate blocks, change norm placement, or add paths. The vocabulary transfers; an identical computation graph does not.

He also states the caveat that should accompany every comparison here:

Architecture is what we can see. It is not necessarily what makes a model good.

671B total / 37B active · 61 layers · MLA + MoE

The most influential open architecture of the period. Kimi K2, Mistral 3 Large, and GLM-5 all adopted parts of it.

The two decisions that define it

MLA over GQA. Raschka explains the reasoning via the DeepSeek-V2 ablations:

So MLA was not chosen only for the ~57× cache reduction — DeepSeek's own measurements say it is better, not merely cheaper. Raschka's summary: "MLA is a clever trick to reduce KV cache memory use while even slightly outperforming MHA in terms of modeling performance."

Fine-grained MoE with a shared expert. 256 experts of hidden size 2048, 8 routed active plus 1 always-on shared. Raschka on the shared expert:

Why it matters

37B active parameters delivering better benchmarks than the 405B dense Llama 3. That comparison is what made MoE the default for frontier open models in 2025.

The descendants

Raschka on Mistral's choice: "why change what ain't broke? A lot of the secret sauce these days is in the training pipeline as well as the inference scaling strategies."

400B total / 17B active · GQA + alternating MoE

The instructive counterpoint to DeepSeek V3: same family of ideas, different choices at nearly every knob.

Raschka's summary:

Note that DeepSeek V3 is ~68% larger in total but has more than twice the active parameters — the two models sit at very different sparsity levels.

Reading it

Every Llama 4 choice is the conservative one: GQA over MLA (simpler, well- supported kernels), few large experts (the older, better-understood MoE style), alternating rather than pervasive MoE (less routing instability). Meta optimised for deployability and reliability; DeepSeek optimised for efficiency at frontier scale.

Raschka's own conclusion is appropriately humble: "Given the many small differences between architectures, it is difficult to determine their exact impact on final model performance. The main takeaway, however, is that MoE architectures have seen a significant rise in popularity in 2025."

Dense: 0.6B → 32B · MoE: 30B-A3B, 235B-A22B

Qwen3 is the reference point Raschka compares nearly everything else against, because it ships at every size in both dense and sparse variants.

Deep and narrow

Comparing Qwen3 0.6B with Llama 3 1B, Raschka notes Qwen3 "is a deeper architecture with more layers, whereas Llama 3 is a wider architecture with more attention heads." The consequences he measured on an A100 with his own from- scratch implementations: Qwen3 has a smaller memory footprint but "a slower runtime (lower tokens/sec generation speed)" — depth cannot be parallelised.

Why both dense and MoE

The shared-expert story

Qwen3 dropped the shared expert that Qwen2.5-MoE had. Raschka asked; developer Junyang Lin replied:

Then Qwen3-Next (Sept 2025) reversed it: 4× more experts and a shared expert restored — both directions Raschka had predicted.

Qwen3-Next — where Qwen went

An 80B-A3B model, 3× smaller than 235B-A22B, and a significant departure:

The gated attention layers are GQA with three stability tweaks: a sigmoid output gate, zero-centered RMSNorm for QK-Norm, and partial RoPE. Raschka: "these are essentially just stability changes to GQA."

Qwen3-Coder-Next (Feb 2026) uses the identical architecture, trained from Qwen3-Next as a base, and reaches SWE-Bench Pro performance "roughly on par with Claude-Sonnet-4.5" — a good illustration of the module-13 point that post-training now drives most differentiation.

Gemma 3: 27B · Gemma 4: 31B dense + 26B-A4B MoE

Gemma 3's prominent long-context mechanism is sliding-window attention. Gemma 4 retains locality and also offers an MoE variant; these choices are not mutually exclusive.

The sliding-window design

Gemma 2 used a 1:1 ratio with a 4096 window. Gemma 3 moved to 5:1 and shrank the window to 1024 — "this shifts the model's focus towards more efficient, localized computations."

The ablation Raschka cites shows "little to no impact on the LLM-generated output perplexity." Substantial cache reduction, essentially free.

The distinctive norm placement

Gemma uses both pre- and post-norm around each sublayer. This is not unique across the full table: the pinned Trinity implementation also uses four norms. Raschka's read:

Gemma 4's near-identical architecture

Raschka: Gemma 4 (31B) "looks pretty much unchanged compared to Gemma 3 (27B)." Two small changes:

1. In global layers, values = keys — reusing the key tensor as the value tensor, "which should result in further KV cache size reduction."
2. p-RoPE at 25% — only a quarter of frequency pairs get positional information, reducing "positional noise in long-context situations."

And the lesson he draws is the most important one in this module:

Gemma 4 (31B) ranks comparably to Qwen3.5-397B-A17B on the AI Arena leaderboard — though he immediately notes "arena scores are a bit problematic as they can be gamed and are biased towards human (style) preference," and cross-checks against standard benchmarks, where the leap holds.

Similar high-level architecture, reported quality jump. This is consistent with training and post-training mattering substantially, but it does not isolate their causal contribution. Data, implementation details, evaluation conditions, and architectural changes are not controlled in this comparison.

Gemma 3n — a different axis

The on-device variant uses Per-Layer Embedding (PLE): keep only a subset of parameters in GPU memory and stream modality-specific embeddings from CPU or SSD on demand. Plus MatFormer (Matryoshka Transformer) — one shared architecture sliceable into smaller independently-usable models, so you run only the part you need.

1T total / 32B active · MLA + MoE

Raschka:

At the time of writing it "may be the biggest LLM of this generation" — with the caveat that Google's 1.6T Switch Transformer "is an encoder-decoder architecture from a different generation" (module 08).

The Muon story

The genuinely novel element is the optimizer, not the architecture:

And his careful reading of the evidence — worth reproducing because it models good skepticism:

Kimi K2 Thinking (Nov 2025) has an unchanged architecture with context extended from 128k to 256k.

Kimi Linear — the other branch

A 48B model exploring linear attention: Kimi Delta Attention (channel-wise gated DeltaNet) follows a 3:1 repeating motif with MLA global layers, using NoPE in the MLA layers. The released 27-layer config contains 20 KDA and 7 MLA layers, including a final global layer that breaks the exact overall ratio. The reported accuracy and speed comparisons are workload-dependent vendor results, not properties established by the config.

Raschka's caveat: it is "20x smaller than Kimi K2. It will be interesting to see if the Kimi team adopts this approach for their upcoming K3 model." Unproven at frontier scale.

Full table

What is universal in 2026

Common patterns in this historical snapshot, not requirements for every model:

• Decoder-only architecture
• RMSNorm in many language backbones, with placement and scope differences
• Gated FFNs are common, including SwiGLU and Gemma 3's GELU-based gate; the pinned Nemotron Nano MLP instead uses an ungated ReLU-squared path
• Residual connections around both sublayers
• RoPE or a deliberate variant of it
• Some form of KV reduction — GQA, MLA, sliding window, or linear

What genuinely varies

The practical payoff of this course. Open any model's config.json and answer:

Worked example — a hypothetical config:

{
  "num_attention_heads": 64,
  "num_key_value_heads": 8,
  "num_hidden_layers": 62,
  "hidden_size": 5120,
  "intermediate_size": 1536,
  "num_local_experts": 160,
  "num_experts_per_tok": 8,
  "n_shared_experts": 1,
  "first_k_dense_replace": 3,
  "sliding_window": null,
  "rope_theta": 1000000.0,
  "use_qk_norm": true
}

Reading: GQA with group size 8 · 62 layers · MoE with 160 fine-grained experts, top-8 plus one shared · first 3 layers dense · no sliding window · RoPE with a large base (long-context tuned) · QK-Norm on. This is GLM-4.5-shaped.

The table is an April-2026 comparison snapshot, not a live leaderboard or a claim that every row was independently reproduced. Distinguish measured vendor facts, implementation facts and explanations proposed by the author. For example, alternate dense/MoE placement is observable; "chosen to reduce routing instability" is a hypothesis unless supported by an ablation.

Two important primary-source corrections are material here: Meta's Llama 4 announcement describes 128 routed experts plus a shared expert for Maverick, and the versioned Llama 4 implementation has a separate shared path. "Two active" is not two routed experts. Gemma 3's configuration uses a GELU-based gate, so SwiGLU is not universal.

For every real checkpoint record vendor repository, exact revision, model size, text-only versus multimodal scope, config fields and implementation version. A vision tower/projector belongs in total parameter counts but not in a text-decoder KV calculation. Some fields describe defaults rather than the chosen released size; remote/custom modeling code may override generic helpers. Do not execute untrusted remote code merely to inspect a JSON configuration.

Pinned checkpoint evidence, row by row

The cited article's 23-family description is not the number of rows in this lesson: the master table contains 20 rows, combines OLMo 2 with Olmo 3, and combines Gemma 4 dense with MoE. This evidence pass accounts for every one of those 20 rows. It does not silently claim coverage of every family mentioned elsewhere in the article or every variant mentioned in this lesson.

The machine-readable ledger records 22 checkpoint config attempts, 20 public JSON snapshots, 370 exact JSON-pointer assertions, and 42 immutable implementation/model-card references. The original downloaded bytes are preserved with SHA-256 hashes. The blocked requests were Llama 4 Maverick and Gemma 3: both required authentication, so no credentials, access workarounds, gated weights, or remote model code were used. Public native vendor sources supply explicitly different evidence for those two rows.

These revisions were retrieved for this audit. They identify reproducible example checkpoints now, not an archived reconstruction of the author's April-2026 table. A current model card at an immutable revision can itself describe later family additions. The ledger therefore separates current source identity, the historical label, and the claim being supported.

Per-cell coverage

C means listed config fields support the cell's configuration components. I means a pinned implementation was inspected. V means vendor-reported, not independently measured. P means partial support with an explicit remaining claim. U means unverified or unresolved identity. N/A applies to a dense example. These are evidence kinds, not a single numerical confidence score: 370 matching fields do not prove 370 architectural or empirical conclusions.

Each C cell in the ledger names the relevant JSON pointers, not just a repository link. Each I cell links a full commit, source-file hash and inspected symbols. Some compound cells still combine kinds: head counts do not prove sink behavior, QK-normalization placement, or a linear-attention recurrence. Read the scope note and the implementation reference for those components. A field such as rms_norm_eps does not establish whether normalization happens before or after a residual branch.

What checking the actual sources changed

A RoPE field can be unused. The pinned Nemotron attention implementation projects Q/K/V and performs attention without applying rotary embeddings; the other inspected attention paths agree. Its config still serializes rope_theta. Its MLP uses relu2 and two projections, not a SwiGLU gate. This corrects the table's position claim and the idea that every family uses the same FFN block.

Layer motifs are not always total ratios. Kimi Linear's full layers are numbered 4, 8, 12, 16, 20, 24 and 27. MiMo's serialized pattern includes boundary exceptions to the advertised five-local/one-global motif. Compute layer counts from the actual list before calculating cache memory. Do not replace a concrete schedule with a rounded ratio in a memory budget.

Variants change parameter accounting. The pinned Mistral model card distinguishes 675B total / 41B active for the multimodal model from 673B / 39B for the language backbone. The Nemotron card reports 3.5B active despite the A3B name. Rounded model names are not an exact weight inventory or a uniform definition of active parameters.

Expert fraction is not parameter fraction. MiniMax selects 8 of 256 routed experts, or 3.125% of those experts, while its model card reports rounded totals of 230B and 10B active. The old 4.37% precision is not justified by those rounded numbers. Dense attention, embeddings, shared paths, and dense layers change the relationship between expert selection and total active weights.

The shared-path column needs implementation evidence. The inspected Gemma 4 decoder adds routed experts to an always-on MLP in the MoE variant, with extra norm paths. The Trinity decoder and MoE also have a shared path. Conversely, an absent shared-expert key alone would not prove that a shared path does not exist. The precise Trinity depth-scaled gain claim remains outside what the inspected forward path establishes.

Context limits have several meanings. The pinned Qwen3 model card distinguishes native 32,768-token support, a 40,960 configured allocation, and 131,072 with optional YaRN settings. These are not contradictory measurements of one quantity. Similarly, the earlier 62-layer, 64-query-head GLM-shaped example is intentionally synthetic: the real GLM-4.5 config has 92 layers and 96 query heads. A plausible-looking JSON object is not a vendor specification.

Reproduce the evidence checks

From a repository checkout, with the adjacent provenance directory preserved:

python content/courses/transformers/code/architecture_provenance.py
python site/test_transformer_provenance.py

For a standalone copy, keep architecture_provenance.py beside a provenance directory containing ledger.json and the configs subdirectory with all 20 snapshot files. Run python architecture_provenance.py from that copy. Downloading only the checker or only the ledger is insufficient: byte-integrity checks need the actual snapshots. The repository tree contains that complete hierarchy; --root /path/to/provenance selects a different local evidence directory.

The offline checker validates snapshot hashes, full revision identifiers, strict JSON values, pointer bindings, and all 120 cell-status records. It never imports downloaded modeling files or downloads weights. The repository tests also reject tampered bytes, stale expected fields, missing or misleading cell references, and regressions in the corrected table. Implementation/model-card hashes record bytes inspected during this audit; the offline checker does not fetch those files again or certify their semantics.

Precisely what remains: two gated JSON files; Maverick's exact checkpoint attention/position settings and expert-width/placement details; the Grok 2.5 identity mismatch; independent parameter inventories; Mistral Large 3 norm placement; and Trinity's exact depth-gain interpretation. The matrix leaves 11 U cells and 2 P cells explicit. The Gemma 3 and OLMo size U entries mean no independent size statement was pinned in this pass, not evidence that the 7B/27B family names are wrong. Off-table Gemma 3n, Qwen3-Coder-Next, Kimi Thinking, DeepSeek V3.2 and other descendants are not separately pinned here. Benchmark quality, training causality, optimizer comparisons, throughput and runtime backend behavior require different evidence; config inspection does not settle them.

Synthetic schema exercise

The following two synthetic schemas are a parser exercise, not vendor checkpoints. One has explicit head width different from residual width/H; the other nests its text configuration. They deliberately catch the two common accounting mistakes without downloading large models.

def projection_and_cache(config, batch=1, length=1024, bytes_per_value=2):
    c = config.get('text_config', config)
    d, h = c['hidden_size'], c['num_attention_heads']
    hk = c.get('num_key_value_heads', h)
    dh = c['head_dim'] if 'head_dim' in c else d // h
    assert h % hk == 0
    if 'head_dim' not in c:
        assert d % h == 0
    attention_weights = 2 * d * h * dh + 2 * d * hk * dh
    cache_bytes = batch * length * c['num_hidden_layers'] * hk * (2 * dh) * bytes_per_value
    return attention_weights, cache_bytes
flat = dict(hidden_size=12, num_attention_heads=3, num_key_value_heads=1,
            head_dim=6, num_hidden_layers=2)
nested = dict(text_config=dict(hidden_size=12, num_attention_heads=3,
                              num_key_value_heads=1, num_hidden_layers=2),
              vision_config=dict(hidden_size=99))
assert projection_and_cache(flat) == (576, 49152)
assert projection_and_cache(nested) == (384, 32768)
print("Explicit head width and nested text schemas produce different budgets.")

This calculation assumes equal K/V widths, all-global attention, bias-free projections and an unsharded GQA cache. MLA, local/global mixtures, KV sharing, quantization metadata and MoE routing require the corresponding formulas from 09-12. The earlier GLM-shaped JSON is also a teaching configuration, not an assertion that a vendor shipped precisely those values.