Sparse & Long-Context Attention: Sliding Windows, Linear Attention, SSMs

Module 09 shrank the KV cache by constant factors. Two costs survived:

Raschka states the reason plainly: Q, K, V are n × d matrices where n is sequence length, so QKᵀ is n × n. Double the context and you quadruple the attention compute.

Constant factors cannot fix a quadratic. To reach long context you must either compute fewer pairs (this module) or compute them more cleverly (module 11).

The simplest idea that works: let each token attend only to a window of nearby tokens.

Raschka's framing: if regular self-attention is global — every element accesses every other — sliding window attention is local, restricting "the context size around the current query position."

Full causal attention (T=8)          Sliding window, w=3
  x . . . . . . .                      x . . . . . . .
  x x . . . . . .                      x x . . . . . .
  x x x . . . . .                      x x x . . . . .
  x x x x . . . .                      . x x x . . . .
  x x x x x . . .                      . . x x x . . .
  x x x x x x . .                      . . . x x x . .
  x x x x x x x .                      . . . . x x x .
  x x x x x x x x                      . . . . . x x x

  O(T^2) pairs                         O(T * w) pairs — LINEAR in T

Cost drops from O(T²) to O(T·w). The cache is bounded by w rather than T: past a certain point, generating more tokens costs no more memory.

Introduced in Longformer (2020), used by Mistral 7B, and now standard.

Why it does not destroy the model

Two reasons. Most linguistic dependencies are local. And stacking creates an effective receptive field: with window w and L layers, information propagates up to w × L positions — the same argument that makes CNNs work.

def sliding_window_mask(T, window, device=None):
    """True = allowed. Causal AND within `window` positions."""
    i = torch.arange(T, device=device)[:, None]
    j = torch.arange(T, device=device)[None, :]
    return (j <= i) & (j > i - window)

Gemma's hybrid ratio

Raschka's Gemma 3 section is the clearest real-world account.

Gemma 2 used a 1:1 ratio — sliding window in every other layer — with a 4096-token window.

Gemma 3 changed both numbers: 5:1 (one full-attention layer per five sliding-window layers) and the window shrank from 4096 to 1024. Raschka: "This shifts the model's focus towards more efficient, localized computations."

The full-attention layers are what preserve genuine long-range capability. Local layers do the bulk of the work cheaply; a few global layers move information across the whole sequence.

The savings are substantial. For a Gemma-3-27B-shaped model at 32k context, an all-global cache of ~15.5 GB drops to ~3.0 GB under the 5:1 hybrid — 5.2× smaller.

And the quality cost? Raschka reports Gemma 3's ablation shows sliding window attention "has minimal impact on modeling performance" — "little to no impact on the LLM-generated output perplexity."

Who uses it, and how aggressively

Raschka flags Xiaomi MiMo-V2-Flash as notable: a 128-token window, "8 times smaller than Gemma 3," on a 309B model — "the largest sliding window attention model to date."

The counterexample worth knowing

Mistral Small 3.1 abandoned sliding window attention — the config file says "sliding_window": null. Raschka's speculation is instructive:

The lesson generalises: an algorithmically cheaper method can be slower in practice if it breaks the fast kernel path. MiniMax-M2 similarly has a sliding-window setting that is disabled by default.

Sliding windows are one sparsity pattern among several.

The 2025–26 direction is the last one. DeepSeek Sparse Attention (V3.2) selects which key-value pairs to attend to based on content rather than a fixed geometric pattern. Raschka describes V3.2 as "overall similar to DeepSeek V3 but they added a sparse attention mechanism to improve efficiency," and GLM-5 (Feb 2026) "adopts DeepSeek's multi-head latent attention (MLA) as well as DeepSeek Sparse Attention... to reduce the inference cost when working with long contexts."

Attention sinks

A small but important discovery: models attend heavily to the first few tokens regardless of content. These are "attention sinks" — a place to dump probability mass when a head has nothing useful to attend to (softmax must sum to 1 somewhere).

Consequence: naively evicting early tokens from the cache destroys quality. StreamingLLM keeps the first few tokens permanently.

Raschka finds explicit sinks parameters in gpt-oss: "attention sinks are special 'always-attended' tokens placed at the start of the sequence to stabilize attention." Arcee Trinity Large attacks the phenomenon from the other side — its elementwise attention gating "reduces attention sinks and improves long-sequence generalization."

A more radical option: change the mathematics so the n × n matrix never exists.

The trick

Standard attention:

The softmax is what forces you to materialise QKᵀ — you cannot reassociate across a nonlinearity. Replace it with a kernel feature map φ and the associativity is restored:

Raschka's Transformers are RNNs (2020) reference uses φ(x) = elu(x) + 1.

Look at the bracketing. On the left you compute (n×n) then multiply by V. On the right you compute φ(K)ᵀV first — a (d×d) matrix, independent of n — then multiply by φ(Q).

standard:   (Q K^T) V     ->  n x n  intermediate   ->  O(n^2 d)
linear:     Q (K^T V)     ->  d x d  intermediate   ->  O(n d^2)

O(n²) becomes O(n). And because KᵀV is a fixed-size running state, decoding becomes cache-free — you keep a d × d state instead of a growing cache. That is why the 2020 paper is titled Transformers are RNNs.

Why it failed for five years

Raschka is direct: these variants "never really gained traction as they degraded the model accuracy, and I have never really seen one of these variants applied in an open-weight state-of-the-art LLM."

The reason is capacity. A fixed d × d state cannot store arbitrary detail about an arbitrarily long sequence. Full attention keeps every token exactly; linear attention keeps a lossy summary. For precise recall — "what was the variable name on line 400?" — that loss is fatal.

Then it came back, and the story has a genuine plot twist.

The timeline

The plot twist

MiniMax released M1 with linear attention, then released M2 without it. Raschka relays the team's reasoning:

This is the most useful data point in the module. Linear attention's weakness is exactly where 2026 workloads live: long multi-turn agentic sessions requiring precise recall.

Raschka: "This could have been a turning point where linear attention may not be worth pursuing after all. However, it gets more interesting" — because Kimi Linear shipped in October and performed well.

Gated DeltaNet

The mechanism behind Qwen3-Next and Kimi Linear. Raschka's description:

Conceptually: maintain a small fast-weight memory updated by a delta rule, gated by α (how much to decay old memory) and β (how much to write new information). Read it with q.

The relationship to Mamba, in his words: "Mamba keeps a state with a learned state-space filter (essentially a dynamic convolution over time). DeltaNet keeps a tiny fast-weight memory updated with α and β and reads it with q." Gated DeltaNet is "a DeltaNet with Mamba-style gating."

The tradeoff: "DeltaNet offers less precise content-based retrieval than full attention, which is why one gated attention layer remains."

The 3:1 hybrid

Both Qwen3-Next and Kimi Linear use the same structure: three linear-attention blocks, then one full-attention block.

Same shape as Gemma's 5:1 sliding-window hybrid: cheap layers do the bulk, a few expensive layers preserve exact retrieval.

This is the dominant 2026 pattern. Nobody replaces all attention.

Gated Attention (Qwen3-Next's full layers)

Not plain GQA. Raschka lists three modifications, all stability-oriented:

• an output gate (sigmoid, usually per-channel) scaling the attention result before the residual add
• zero-centered RMSNorm for QK-Norm instead of standard RMSNorm
• partial RoPE (module 05)

"Note that these are essentially just stability changes to GQA."

Kimi Delta Attention

Kimi Linear's refinement of Gated DeltaNet, in two changes:

1. Channel-wise gating. Where Qwen3-Next "applies a scalar gate (one value per attention head) to control the memory decay rate, Kimi Linear replaces it with a channel-wise gating for each feature dimension" — finer control over memory, better long-context reasoning.
2. MLA for the full layers, replacing Qwen3-Next's gated attention, with an additional gate.

Kimi Linear also uses NoPE in the MLA layers, letting MLA "run as pure multi-query attention at inference" and avoiding RoPE retuning for long context.

Results, per Raschka: Kimi Linear "achieves higher modeling accuracy while maintaining the same token-generation speed" as Gated DeltaNet-H1, and is "much faster than an architecture with multi-head latent attention... while having a higher benchmark performance."

The caveat he attaches matters: Kimi Linear is 48B, "20x smaller than Kimi K2." It is unproven at frontier scale.

A parallel lineage that arrives at similar computational properties from control theory rather than from attention.

An SSM maintains a hidden state evolving by a learned linear recurrence:

h_t = A h_{t-1} + B x_t
y_t = C h_t

Linear in sequence length, constant memory, and (because the recurrence is linear) parallelisable over the sequence during training via a scan. Mamba (2023) made A, B, C input-dependent — "selective" — which gave it the content-awareness earlier SSMs lacked. Mamba-2 refined the formulation.

The core tradeoff is the same as linear attention:

Where SSMs actually ship

NVIDIA Nemotron 3 Nano (30B-A3B) is the most committed example Raschka covers: "a 52-layer hybrid Mamba-Transformer model that interleaves Mamba-2 sequence-modeling blocks with sparse Mixture-of-Experts (MoE) feed-forward layers, and uses self-attention only in a small subset of layers." Organised as 13 macro blocks of Mamba-2 → MoE, plus a few GQA layers.

He notes it has "really good performance compared to pure transformer architectures of similar size, while achieving much higher tokens-per-second throughput," and calls it "even more extreme than Qwen3-Next and Kimi-Linear in its use of only a few attention layers" — while flagging the open question: "one of the strengths of the transformer architecture is its performance at a (really) large scale."

Others: IBM Granite 4.0, NVIDIA Nemotron Nano 2.

A taxonomy note

Raschka draws a distinction worth borrowing:

Nemotron sits on the SSM side; Qwen3-Next and Kimi Linear on the transformer side. The line is blurry, and he says an argument could be made for one category.

The single most important takeaway: nobody removes full attention entirely. Every production long-context model keeps some full-attention layers — 1 in 6 (Gemma), 1 in 4 (Qwen3-Next, Kimi Linear, Olmo 3), or a handful (Nemotron). Exact retrieval is worth paying for somewhere.