RoPE, scaling, sliding window, attention sinks

Two symptoms, both common, both silent.

The first: you serve a model whose config says max_position_embeddings: 8192 and launch it with --max-model-len 32768. vLLM refuses:

            msg = (
                f"User-specified max_model_len ({max_model_len}) is greater "
                f"than the derived max_model_len ({max_len_key}="
                f"{derived_max_model_len} or model_max_length="
                f"{model_max_length} in model's config.json)."
            )
            warning = (
                "VLLM_ALLOW_LONG_MAX_MODEL_LEN must be used with extreme "
                "caution. If the model uses relative position encoding (RoPE), "
                "positions exceeding derived_max_model_len lead to nan. If the "
                "model uses absolute position encoding, positions exceeding "
                "derived_max_model_len will cause a CUDA array out-of-bounds "
                "error."
            )

So you set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1, the server starts, requests complete, no error appears — and the model's answers past 8k are subtly wrong. Nothing crashed, nothing logged. The positional frequencies simply left the range the weights were trained on.

The second: you serve a sliding-window model, hybrid KV caching is off, and this appears once at startup and never again:

    logger.warning(
        "Hybrid KV cache manager is disabled for this hybrid model, "
        "This means we do not enable any optimizations for saving KV cache "
        "memory (e.g., dropping the KV cache outside the sliding window). "
        "The compute of layers like sliding window is still saved."
    )

Your KV cache is now sized as if every layer were full attention: on a Llama-3-8B-shaped model at 128k context, 16 GiB per sequence instead of 4.375 GiB — on a 3:1 sliding-to-full layer pattern with a 4,096-token window, derived in §7 — a 3.7× loss of batch capacity from one warning line. Both failures come from the same place — position is not a property of the model, it is a property of the cache.

Rotary position embedding (RoPE, Su et al., arXiv:2104.09864) adds nothing to the hidden state. It cuts each head's query and key vectors into two-dimensional pairs and rotates each pair by an angle proportional to the token's absolute position. Pairs rotate at geometrically spaced rates: the first turns a full revolution roughly every 6.3 tokens, the last takes tens of thousands of tokens to turn once. The head dimension becomes a bank of clocks, and a position is the reading of all of them at once.

The reason to rotate rather than add is that the dot product of two rotated vectors depends only on the difference of the rotation angles. A query at position 5 against a key at position 3 gives exactly the inner product of a query at 105 against a key at 103. Absolute positions go in; relative distance comes out.

Two consequences fall out of that picture. The rotation is applied to Q and K only — never to V, which carries content and must not be spun. And the K written to the KV cache is the rotated K: the cache does not store keys, it stores keys-at-a-position.

Let $d_h$ be the head dimension, $r \le d_h$ the rotary dimension (the leading slice RoPE touches; the tail passes through untouched), $b$ the base frequency (`rope_theta` in the checkpoint), and $m \in \mathbb{Z}_{\ge 0}$ the token's absolute position. Index the rotary pairs by $i = 0, 1, \dots, r/2 - 1$. The frequency of pair $i$ is

That is one line of code, and it is the line every scaling scheme in this chapter modifies:

    def _compute_inv_freq(self, base: float) -> torch.Tensor:
        """Compute the inverse frequency."""
        # ...
        inv_freq = 1.0 / (
            base
            ** (
                torch.arange(0, self.rotary_dim, 2, dtype=torch.float) / self.rotary_dim
            )
        )
        return inv_freq

    def _compute_cos_sin_cache(self) -> torch.Tensor:
        """Compute the cos and sin cache."""
        inv_freq = self._compute_inv_freq(self.base)
        t = torch.arange(self.max_position_embeddings, dtype=torch.float)

        freqs = torch.einsum("i,j -> ij", t, inv_freq)
        cos = freqs.cos()
        sin = freqs.sin()
        cache = torch.cat((cos, sin), dim=-1)
        return cache

Note what that is: a table of shape $[\texttt{max\_position\_embeddings},\, r]$ built once at model construction. RoPE can reuse gathered precomputed trigonometric values, but still performs the rotation arithmetic. The rotation, writing $x^{(i)} = (x_{i}, x_{i+r/2})$ for the NeoX pairing,

is four multiplies and two adds per pair, and appears verbatim as:

        if is_neox_style:
            x1, x2 = torch.chunk(x, 2, dim=-1)
        else:
            x1 = x[..., ::2]
            x2 = x[..., 1::2]

        o1 = x1 * cos - x2 * sin
        o2 = x2 * cos + x1 * sin

        if is_neox_style:
            output = torch.cat((o1, o2), dim=-1)
        else:
            output = torch.stack((o1, o2), dim=-1).flatten(-2)

The is_neox_style branch is a real inference hazard, not a stylistic one: NeoX pairs element $j$ with element $j + r/2$, GPT-J pairs $2j$ with $2j+1$. Same mathematics, different memory layout, and checkpoints have their QK weight rows permuted to match one or the other. Get it wrong and the model produces fluent garbage with no error.

Why the dot product only sees $m - n$

Rotation matrices are orthogonal, with $R(a)^{\!\top} = R(-a)$ and $R(a)R(c) = R(a+c)$. For query pair $q^{(i)}$ at position $m$ and key pair $k^{(i)}$ at position $n$:

Summing over pairs, the full head logit is a function of $q$, $k$, and $m-n$ alone. Absolute positions vanish from the score even though they were used to compute it.

A worked number

Take pair $i = 16$ at $b = 10^4$, $r = 128$: $\omega_{16} = 10000^{-32/128} = 0.1$ rad/token, wavelength $62.8$ tokens. Let $q^{(16)} = k^{(16)} = (1, 0)$ before rotation.

Both equal $\cos(0.2) = 0.9800666$. The vectors moved to a completely different part of the circle; the score did not move at all.

Where the positions come from

Nothing in the model computes a position. The model runner materialises one flat int64 tensor per step, and it is pure scheduler bookkeeping:

        self.positions[:total_num_scheduled_tokens] = (
            self.num_computed_tokens[req_indices_gpu].to(torch.int64)
            + self.query_pos.gpu[:total_num_scheduled_tokens]
        )

Position = tokens of this request already in the cache, plus the offset within this step's chunk. The same tensor then feeds compute_slot_mapping, so positions and cache slots come from one counter — the only thing keeping rotation angle and storage location in agreement. An off-by-one in num_computed_tokens does not crash; it rotates every subsequent key by one extra tick and quietly shifts the model's sense of distance.

Read one attention block and the ordering is unambiguous:

    def forward(
        self,
        positions: torch.Tensor,
        hidden_states: torch.Tensor,
    ) -> torch.Tensor:
        qkv, _ = self.qkv_proj(hidden_states)
        q, k, v = qkv.split([self.q_size, self.kv_size, self.kv_size], dim=-1)
        q, k = self.rotary_emb(positions, q, k)
        attn_output = self.attn(q, k, v)
        output, _ = self.o_proj(attn_output)
        return output

rotary_emb runs first and mutates k in place (ops.rotary_embedding is documented as in-place at base.py:L242-L252); self.attn then writes that rotated k into the paged cache. No unrotated copy exists anywhere.

The practical cost: insert a 40-token per-user preamble before a 20,000-token shared document and every document block becomes a miss, because each token shifted by 40 positions and its stored key is the wrong rotation. Move the preamble after the document and the whole document stays cacheable. Prompt order is a throughput decision. Storing pre-RoPE keys would permit changing rotation coordinates, but would not make arbitrary substring reuse correct: earlier-layer hidden states and values also depend on preceding context. It may additionally move rotation work into decode — turning a gather into a full pass over the KV cache, exactly the traffic §3.2 exists to avoid.

Why does a model trained at 8k degrade at 32k? Look at the slowest pair. At $b = 5\times10^5$, $r = 128$, pair $i = 63$ has wavelength $2\pi (5\times10^5)^{63/64} = 2.56\times10^6$ tokens. Over the entire 8192-token training window it rotates through $0.0032$ of a revolution — an arc of about 1.15°, and every weight reading that pair was fitted on that sliver. At position 32768 the arc is four times wider. Chen et al. (arXiv:2306.15595) report that direct extrapolation blows up attention scores catastrophically, far worse than naive expectation. Fast pairs already span many rotations (pair 0 completes about 1300 in 8192 tokens), while slow pairs can leave their trained angular range. This motivates frequency-dependent extension, but does not prove all extrapolation error comes only from slow frequencies; learned multi-frequency interactions and attention statistics also matter.

Four families of fix, all edits to $\omega_i$ or to the position axis.

Each is a handful of lines. PI edits the position axis (linear_scaling_rope.py:L90-L97: build t to max_position_embeddings * scaling_factor, then t = t / scaling_factor); fixed NTK edits the base (ntk_scaling_rope.py:L31-L36); dynamic NTK recomputes the base from the ratio of served length to trained length:

        base = self.base * (
            (
                self.scaling_factor
                * self.max_position_embeddings
                / self.max_trained_positions
            )
            - (self.scaling_factor - 1)
        ) ** (self.rotary_dim / (self.rotary_dim - 2))
        inv_freq = self._compute_inv_freq(base)

Llama-3's own scheme is a third shape — a wavelength-thresholded blend, not a base shift (llama3_rope.py:L33-L54): pairs with wavelength below orig_max_position / high_freq_factor are untouched, pairs above orig_max_position / low_freq_factor are divided by the scaling factor, and the band between is linearly smoothed.

YaRN, precisely

YaRN's contribution is choosing the band by rotation count rather than by hand. Given $\beta_{\text{fast}}$ (default 32) and $\beta_{\text{slow}}$ (default 1), it inverts the wavelength formula to find the pair indices at which a pair completes exactly that many rotations within the original context length $L_0$:

def yarn_find_correction_dim(
    num_rotations: int,
    dim: int,
    base: float = 10000,
    max_position_embeddings: int = 2048,
) -> float:
    return (dim * math.log(max_position_embeddings / (num_rotations * 2 * math.pi))) / (
        2 * math.log(base)
    )
# ...
def yarn_linear_ramp_mask(
    low: float, high: float, dim: int, dtype: torch.dtype
) -> torch.Tensor:
    if low == high:
        high += 0.001  # Prevent singularity

    linear_func = (torch.arange(dim, dtype=dtype) - low) / (high - low)
    ramp_func = torch.clamp(linear_func, 0, 1)
    return ramp_func


def yarn_get_mscale(scale: float = 1) -> float:
    if scale <= 1:
        return 1.0
    return 0.1 * math.log(scale) + 1.0

and blends the two frequency sets with that ramp:

    def _compute_inv_freq(self, scaling_factor: float) -> torch.Tensor:
        pos_freqs = self.base ** (
            torch.arange(0, self.rotary_dim, 2, dtype=torch.float) / self.rotary_dim
        )
        inv_freq_extrapolation = 1.0 / pos_freqs
        inv_freq_interpolation = 1.0 / (scaling_factor * pos_freqs)

        low, high = yarn_find_correction_range(
            self.beta_fast,
            self.beta_slow,
            self.rotary_dim,
            self.base,
            self.max_position_embeddings,
            self.truncate,
        )
        # Get n-d rotational scaling corrected for extrapolation
        inv_freq_mask = (
            1
            - yarn_linear_ramp_mask(low, high, self.rotary_dim // 2, dtype=torch.float)
        ) * self.extrapolation_factor
        inv_freq = (
            inv_freq_interpolation * (1 - inv_freq_mask)
            + inv_freq_extrapolation * inv_freq_mask
        )
        return inv_freq

    def _compute_cos_sin_cache(self) -> torch.Tensor:
        inv_freq = self._compute_inv_freq(self.scaling_factor)
        t = torch.arange(
            self.max_position_embeddings * self.scaling_factor, dtype=torch.float32
        )
        freqs = torch.einsum("i,j -> ij", t, inv_freq)
        cos = freqs.cos() * self.mscale
        sin = freqs.sin() * self.mscale
        cache = torch.cat((cos, sin), dim=-1)
        return cache

Work the defaults for $r = 128$, $b = 10^4$, $L_0 = 4096$: $d(32) = 20.94$, $d(1) = 45.03$, truncated to $[20, 46]$. Pairs 0–20 keep their original frequency, pairs 46–63 are divided by $s$, pairs 21–45 ramp. Check the endpoints: pair 21 has wavelength 129 tokens — 31.75 rotations in 4096, i.e. $\beta_{\text{fast}}$. Pair 45 has wavelength 4080 — 1.00 rotations, i.e. $\beta_{\text{slow}}$. The constants mean exactly what they say.

The attention-temperature correction hides in the last two lines of _compute_cos_sin_cache. Both cos and sin are multiplied by $\mu = 0.1\ln s + 1$, so the rotation becomes $\mu R(m\omega_i)$ — applied to $q$ and $k$ from the same table, which multiplies the rotary-subspace dot-product contribution by $\mu^2$. For full RoPE this scales the full dot product; with partial RoPE the unrotated tail is unchanged. At $s = 8$, $\mu = 1.2079$ and $\mu^2 = 1.459$: the full-RoPE effective temperature becomes 1/1.459=0.685 of its previous value, about 31.5% lower, while logit scale rises 45.9%, sharpening a distribution interpolation had flattened. YaRN reports this closes most of the remaining gap (arXiv:2309.00071).

Both engines resolve the scaling type once, at startup, and memoise the module. vLLM's get_rope reads two keys and switches on the second:

    base = rope_parameters.get("rope_theta", 10000)
    scaling_type = rope_parameters.get("rope_type", "default")
# ...
        raise ValueError(f"Unknown RoPE scaling type {scaling_type}")

then caches the module in a module-level _ROPE_DICT keyed on (head_size, rotary_dim, max_position, is_neox_style, rope_parameters, ...), so every layer sharing a config shares one cos/sin table. SGLang's factory.py has the same dispatch and the same _ROPE_DICT, plus a warning for the failure mode that motivates this section:

def _get_rope_param(rope_scaling, key, default, scaling_type):
    """Get a parameter from rope_scaling dict, warn if missing.

    In transformers v5, config.rope_scaling is an alias for rope_parameters
    which may be non-None even for models with no actual scaling (rope_type=default).
    When a required key is missing, this logs a warning instead of silently
    defaulting, to make config mismatches easier to debug.
    """
    if key in rope_scaling:
        return rope_scaling[key]
    logger.warning(
        "rope_scaling (type=%s) missing key '%s', defaulting to %s. "
        "This may indicate a v5 config issue — check model accuracy.",
        scaling_type,
        key,
        default,
    )
    return default

The genuine engine difference is what happens when a position exceeds the table. vLLM builds it to max_position_embeddings and stops; an out-of-range cos_sin_cache.index_select(0, positions) is the CUDA out-of-bounds the config warning threatens. SGLang can grow the table:

    def _ensure_cos_sin_cache_length(self, needed_max_pos: int):
        """Ensure cos_sin_cache length > needed_max_pos."""
        cur_len = int(self.cos_sin_cache.shape[0])
        if needed_max_pos < cur_len:
            return

        # Align to reduce realloc frequency
        align = envs.SGLANG_ROPE_CACHE_ALIGN.get()
        new_len = ((needed_max_pos + align) // align) * align
        device = self.cos_sin_cache.device
        dtype = self.cos_sin_cache.dtype

It extends incrementally with self.base — the unscaled base for the plain class. That trades a hard crash for a soft quality loss: friendlier, and much harder to notice.

The load-bearing point for operators: the scaling factor is baked into the cos/sin table at construction. Restart with a different --rope-scaling and every KV block cached under the old table is meaningless, because the rotation that produced those keys no longer matches the rotation applied to new queries. Nothing on a KV block records this; if you persist or share prefix caches across processes (§2.6), the rope config is part of the cache identity whether or not the system says so.

Config also decides the context length. vLLM multiplies the model's own maximum by the factor:

            rope_type = rp["rope_type"]

            if rope_type not in ("su", "longrope", "llama3"):
                # NOTE: rope_type == "default" does not define factor ...
                # NOTE: This assumes all layer types have the same scaling factor.
                scaling_factor = rp.get("factor", scaling_factor)

                if rope_type == "yarn":
                    derived_max_model_len = rp["original_max_position_embeddings"]
        if scaling_factor is None:
            # Fallback the factor to 1.0 if a user assigned `null`
            logger.warning_once(
                "The model's RoPE configuration has a null scaling "
                "factor which is unexpected. This likely indicates a bug "
                "in the model's HuggingFace config.json. Please notify the "
                "model vendor. Falling back the value to 1.0. "
            )
            scaling_factor = 1.0
        # Do this outside loop since all layer types should have the same scaling
        derived_max_model_len *= scaling_factor

Note the YaRN special case: the derived length is reset to original_max_position_embeddings before being multiplied, because a YaRN checkpoint's max_position_embeddings is usually already the extended figure and would otherwise be scaled twice.

Sliding-window attention caps how far back a query may look. The kernel consequence is a mask; the serving consequence is much larger, because a key that can never be attended again is a key you can free. Full attention holds $T$ tokens of KV per sequence and grows without bound; a window of $W$ holds $\min(T, W)$ — a constant.

Using the cell size from §2.1, $c = 2 L h_{kv} d_h \cdot \texttt{bytes}$, a Llama-3-8B-shaped model ($L=32$, $h_{kv}=8$, $d_h=128$, bf16) costs 128 KiB per token:

The allocator does not think in tokens, though — it thinks in blocks, and vLLM's SlidingWindowSpec states the bound precisely:

        num_tokens = min(
            self.sliding_window - 1 + self.extra_retained_tokens + max_in_flight_tokens,
            max_model_len,
        )
        # +1 because the sliding window may not start from the beginning of
        # the block. E.g. block size 4 and num_token 4 needs two blocks
        # [XXCD][EF] to store the 6-token window [CDEF].
        return cdiv(num_tokens, self.block_size) + 1

Compare FullAttentionSpec.max_memory_usage_bytes: cdiv(max_model_len, self.block_size) * self.page_size_bytes (kv_cache_interface.py:L300-L306). Same page size, same arithmetic shape — only the numerator differs, sliding_window instead of max_model_len. That substitution is the entire memory story. At $W = 4096$, block size 16, ignoring in-flight tokens: $\lceil 4095/16 \rceil + 1 = 257$ blocks, against $\lceil 131072/16\rceil = 8192$ for full attention at 128k.

The freeing is where §2.2's null block earns its keep. A request's block table must stay index-addressable — slot $j$ is tokens $[16j, 16j+16)$ — so a freed leading block cannot be removed from the list, only replaced:

        blocks = self.req_to_blocks[request_id]
        last_block = min(last_block, len(blocks))

        freed: list[KVCacheBlock] = []
        for i in range(last_block - 1, first_block - 1, -1):
            if blocks[i] == self._null_block:
                break
            freed.append(blocks[i])
            blocks[i] = self._null_block
        if freed:
            self.block_pool.free_blocks(freed)

and the window edge is one subtraction:

        return max(
            0,
            num_computed_tokens - self.sliding_window + 1 - self.extra_retained_tokens,
        )

Hybrid models break the uniform-spec assumption

A block manager that assumed one spec per model now faces two. SGLang enumerates the pattern per architecture — for Llama-4, three of every four layers are windowed:

    if "Llama4ForConditionalGeneration" in model_architectures:
        swa_attention_layer_ids = [
            i for i in range(num_hidden_layers) if (i + 1) % 4 != 0
        ]
        full_attention_layer_ids = [
            i for i in range(num_hidden_layers) if (i + 1) % 4 == 0
        ]
    elif any(arch in SWA_SINK_ARCHS for arch in model_architectures):
        layer_types = getattr(hf_text_config, "layer_types", [])
        swa_attention_layer_ids = [
            i for i, x in enumerate(layer_types) if x == "sliding_attention"
        ]
        full_attention_layer_ids = [
            i for i, x in enumerate(layer_types) if x == "full_attention"
        ]

Derived, for a 32-layer model on that 3:1 pattern at 128k, $W = 4096$, per-layer per-token cost $2 h_{kv} d_h \cdot 2 = 4$ KiB: the 8 full layers hold $8 \times 131072 \times 4096 = 4$ GiB, the 24 windowed layers $24 \times 4096 \times 4096 = 384$ MiB. Total 4.375 GiB against 16 GiB uniform — 3.66×, and that is what the "hybrid KV cache manager is disabled" warning silently costs, because the fallback promotes every SlidingWindowSpec to FullAttentionSpec (kv_cache_utils.py:L1578-L1600).

SGLang solves the same problem in the cache rather than the allocator. Windowed and full layers have different lifetimes for the same radix-tree node, so SWARadixCache keeps two independent LRU orders over one tree (§2.4). Per the callout above it is the readable statement of the idea rather than the live class — but it is the clearest one in the tree, so it is what this section quotes:

        # LRU lists are used to maintain the order of eviction of the nodes in the tree
        self.full_lru_list = LRUList(is_swa_list=False)
        self.swa_lru_list = LRUList(is_swa_list=True)

The invariant is in the TreeNode comment at swa_radix_cache.py:L70-L74: full_lock_ref is always $\ge$ swa_lock_ref. Eviction from the full list must take a leaf; eviction from the SWA list need not, because an internal node's window-layer KV can be dropped while its children still hold live full-layer KV. Such a node becomes a tombstone — present in the tree, matchable for the full layers, its SWA indices already returned to the allocator (swa_radix_cache.py:L640-L672). An all-windowed model needs none of that, so SGLang ships a 153-line PureSWARadixCache that caches the prefill prefix and frees the window (pure_swa_radix_cache.py:L24-L30). Two caches for two model shapes, versus vLLM's one allocator parameterised by spec.

The obvious way to bound KV growth without a windowed checkpoint is to evict the oldest tokens. Xiao et al. (arXiv:2309.17453) show this destroys the model: perplexity explodes the moment the very first tokens leave the cache. Their diagnosis is that the first few positions absorb a disproportionate share of attention mass across most heads and layers — not because those tokens are informative, but because softmax must put its mass somewhere and the initial tokens are visible to every subsequent query. They are a bias term the model learned to lean on; evict them and every softmax denominator changes at once.

StreamingLLM retains a small number of each request's initial KV entries plus a rolling recent window. The StaticSinkAttention excerpts below show reserved static sink blocks, but allocation and metadata alone do not establish that they hold each request's first-token KV. Their initialization, model semantics, and ownership must be verified before identifying them with StreamingLLM. Treat static sink blocks, request-specific retained tokens, and learned sink logits as distinct mechanisms.

        sink_len = kv_cache_spec.sink_len
        assert sink_len is not None and sink_len > 0 and sink_len % self.block_size == 0
        num_sink_block = sink_len // self.block_size
        self.sink_blocks = self.block_pool.free_block_queue.popleft_n(num_sink_block)

The attention side prepends those blocks to every block table and inflates the sequence length to match, so the kernel needs no modification:

            common_attn_metadata.seq_lens[:] = (
                common_attn_metadata.seq_lens + self.sink_len
            )
            common_attn_metadata.seq_lens[
                common_attn_metadata.seq_lens == self.sink_len
            ] = 0
            common_attn_metadata.max_seq_len = (
                common_attn_metadata.max_seq_len + self.sink_len
            )
            max_num_blocks = cdiv(common_attn_metadata.max_seq_len, self.block_size)
            num_reqs = common_attn_metadata.num_reqs
            self.block_table_with_sink[
                :num_reqs, self.num_sink_blocks : self.num_sink_blocks + max_num_blocks
            ] = common_attn_metadata.block_table_tensor[:, :max_num_blocks]

A second, unrelated thing is also called an attention sink. SGLang's model config defines it:

    def _detect_attention_sinks(self) -> bool:
        """Check whether the model uses learned attention sinks.

        Attention sinks are per-head scalars added to the softmax denominator
        to compensate for evicted KV-cache entries under sliding-window
        attention.  Not every hybrid-SWA model uses them.
        """

That is a learned sink — a trained per-head logit in the denominator, costing zero cache blocks — against StreamingLLM's pinned KV sink, which costs real blocks and needs no retraining. vLLM keeps them separate deliberately: a plain Attention layer with learned sinks still yields an ordinary FullAttentionSpec or SlidingWindowSpec, and only StaticSinkAttention yields SinkFullAttentionSpec (vllm/v1/attention/selector.py:L76-L80). One is a kernel argument; the other is an allocator contract.

Follow token 4096 of a request on a windowed layer with $W = 4096$, block size 16, on vLLM.

1. Position. GPUModelRunner._prepare_inputs writes positions[i] = num_computed_tokens[req] + query_pos[i] (gpu_model_runner.py:L2248-L2251), then passes the same tensor to block_table.compute_slot_mapping (L2257-L2261).
2. Rotation. LlamaAttention.forward calls self.rotary_emb(positions, q, k) (llama.py:L228), dispatching through CustomOp to RotaryEmbedding.forward_cuda (base.py:L221-L252): a cos_sin_cache.index_select(0, positions) and an in-place mutation of q and k. No trigonometry at runtime.
3. Storage. self.attn(q, k, v) writes the rotated k to the slot from step 1. The cache now holds $R(4096\,\omega)k$, permanently.
4. Release. Before each subsequent chunk, allocate_slots calls SlidingWindowManager.remove_skipped_blocks (single_type_kv_cache_manager.py:L624-L657) → _remove_blocks_in_range. That loop runs backwards and breaks on the first null it meets — an O(newly-freed) scan rather than O(sequence) on every step.

1. See the YaRN band for your own config. The correction range is pure arithmetic on four config fields, computable without importing anything:

python3 -c '
import math
r, b, L0, bfast, bslow = 128, 10000, 4096, 32, 1
d = lambda n: (r*math.log(L0/(n*2*math.pi)))/(2*math.log(b))
lo, hi = math.floor(d(bfast)), math.ceil(d(bslow))
print("ramp band [low, high] =", (max(lo,0), min(hi, r-1)))
for i in (lo, lo+1, hi-1, hi):
    wl = 2*math.pi*b**(2*i/r)
    print(f"pair {i:2d}  wavelength {wl:10.1f}  rotations in L0 {L0/wl:7.2f}")'

Running that here prints the band (20, 46) and 36.66 / 31.75 / 1.00 / 0.87 rotations at pairs 20 / 21 / 45 / 46 — the endpoint numbers used in §5, derived, not measured. Change b to 500000 and the band moves to (14, 32): a larger base lengthens every wavelength, so more pairs fall below one rotation in $L_0$ and the interpolated tail grows. Change $L_0$ to 8192 as well — Llama-3's actual pair — and it returns to (18, 35).

2. Confirm the scaling factor actually in effect, on a machine with vLLM installed:

python3 -c '
from vllm.model_executor.layers.rotary_embedding import get_rope
plain = get_rope(128, 8192, rope_parameters={"rope_theta": 10000.0})
yarn  = get_rope(128, 8192, rope_parameters={"rope_theta": 10000.0, "rope_type": "yarn",
                 "factor": 8.0, "original_max_position_embeddings": 4096})
print(type(yarn).__name__, yarn.cos_sin_cache.shape, "mscale", yarn.mscale)
print(type(plain).__name__, plain.cos_sin_cache.shape)'

mscale should read 1.2079 for $s=8$ and the YaRN table's first dimension should be $4096 \times 8$. I did not execute this — no GPU and no vLLM install in this environment.

3. Watch a window free blocks. For an all-sliding-window checkpoint, active attention KV can plateau after the window plus alignment/in-flight allowance, although prefix-cache retention changes free-pool observations. For hybrid models, full-attention layers continue growing. Record per-group occupancy and distinguish active, cached-evictable, and reserved blocks before interpreting the trace.

• Su et al., RoFormer (arXiv:2104.09864) — the original derivation of the relative-position property worked in §3.
• Chen et al., Extending Context Window of LLMs via Positional Interpolation (arXiv:2306.15595) — PI, and the measurement that naive extrapolation blows up attention scores.
• Peng et al., YaRN (arXiv:2309.00071) — the band decomposition, $\beta_{\text{fast}}$ / $\beta_{\text{slow}}$, and the attention-temperature correction. vLLM credits github.com/jquesnelle/yarn in the class docstring.
• Xiao et al., Efficient Streaming Language Models with Attention Sinks (arXiv:2309.17453) — the attention-mass finding and the four-token fix.
• The NTK schemes originate in community posts and both repos say so in their docstrings: ntk_scaling_rope.py:L11-L12 cites kexue.fm/archives/9706; dynamic_ntk_scaling_rope.py:L33 credits "/u/bloc97 and /u/emozilla". Unusually well-attributed folklore.
• Source to read next, in order: vllm/model_executor/layers/rotary_embedding/base.py (324 lines, the whole mechanism), then vllm/v1/kv_cache_interface.py:L537-L594, then python/sglang/srt/mem_cache/swa_radix_cache.py — start at evict (L594-L700), where the dual-LRU design is most legible.