Routers, KV-aware balancing, session affinity

You run eight H100s, one Llama-3-8B replica each, behind an nginx round-robin. Your product is a B2B assistant: a thousand customers, each with their own 2,048-token system prompt — persona, tool schemas, retrieval preamble — followed by a short user turn. Traffic is 60 requests per second. Prefix caching is on in every replica. You measured a 90% hit rate in staging with one replica. In production you see 10%.

Nothing is broken. Here is the arithmetic, all of it derived from numbers this book has already established.

One replica's KV pool on an 80 GB H100 is 52.32 GiB, which at 128 KiB per token is 428,569 resident tokens (§2.1, derived). Live requests occupy roughly half of it, so call it ~214,000 tokens available to hold cached prefixes: 104 distinct 2,048-token system prompts per replica.

Under round-robin, every replica sees a uniform sample of all 1,000 prompts. Its LRU set converges to a random 104 of them, so the hit probability is $104/1000 = 10.4\%$. Under a routing policy that maps each prompt to one replica, a replica only ever sees $1000/8 = 125$ prompts and holds 104 of them: $104/125 = 83.2\%$.

A miss on a 2,048-token prefix costs a full prefill of those tokens: 34.0 TFLOP, 86 ms of H100 time at 40% of the bf16 peak (§1.1, derived). Multiply out.

You did not lose the cache to a bug. You lost it to a scheduling decision made by a component that has never heard of a KV cache. vLLM's own integration documentation states the failure in one line:

A single vLLM server is fast, but at scale the picture changes: across many replicas,
cache locality breaks under round-robin load balancing, long prompts inflate
time-to-first-token, and accelerators sit underused. llm-d adds the cluster-level layer
that vLLM does not aim to provide on its own:

- **[Prefix-aware routing](https://llm-d.ai/docs/guides/precise-prefix-cache-aware).**
  Instead of round-robin, llm-d reads vLLM's KV-cache events and routes each request to
  the replica that already holds its prefix, reusing cache instead of recomputing it.

Think of the fleet's aggregate cache as a set-associative cache where the router is the placement function. Round-robin can duplicate hot keys across replicas, but the union of caches of size $M$ can still contain up to $NM$ distinct keys. Under the uniform independent-reference model above, a request sent to a random replica hits with probability $M/W$; this is an effective hit probability, not a bound of $M$ on fleet storage. Affinity can reduce duplication and improve that probability. The routing decision, not the memory, is what multiplies.

The hard part is that the router cannot read the placement function's own output. It does not know what any replica currently holds: the engine evicts on its own schedule, under its own memory pressure, with its own block granularity. So the router keeps a model of each replica's cache, built from what it has sent where, and decays that model on a timer. Everything interesting in this chapter follows from that model being an approximation.

Let $N$ be the number of replicas, $M$ the number of distinct cached prefixes one replica's KV pool can hold, and $W$ the size of the workload's distinct-prefix working set. Assume uniform prefix popularity — the worst case for caching, and close to reality for per-tenant system prompts.

Under round-robin each replica's request stream is a uniform sample of all $W$ prefixes, so its resident set is a uniform random $M$-subset and the steady-state hit rate is

Under an affinity map $h : \text{prefix} \to \text{replica}$ that is stable and roughly balanced, a replica only ever sees $W/N$ distinct prefixes:

The fleet's effective distinct-prefix capacity is $M$ under round-robin and $MN$ under affinity. Call the gap cache dilution: round-robin replicates every hot prefix onto all $N$ replicas, so you pay $N\times$ the HBM for the same working set. It is the exact dual of the false-sharing problem in a multi-core cache.

The cost of a miss is a full prefill of the shared prefix. With arrival rate $\lambda$ and prefix length $L_p$, the wasted GPU time, in GPU-seconds per wall-second, is

Work it for the opening scenario. $N = 8$, $W = 1000$, $L_p = 2048$. Llama-3-8B ($L=32$, $h_{kv}=8$, $d_h=128$) stores $2 L h_{kv} d_h b = 131{,}072$ bytes = 128 KiB of KV per token. The 52.32 GiB pool holds 428,569 tokens; half of it is live-request KV, so $M = \lfloor 214{,}284 / 2048 \rfloor = 104$.

$H_{\mathrm{rr}} = 104/1000 = 0.104$. $H_{\mathrm{aff}} = \min(1, 832/1000) = 0.832$. A 2,048-token prefill is $2P\cdot T + 2 L h d_h T(T{-}1) = 32.9 + 1.1 = 34.0$ TFLOP, and at 40% of the H100's 989 TFLOP/s bf16 peak that is $t_{\mathrm{prefill}} = 86$ ms. So $G_{\mathrm{waste}}$ is $60 \times 0.896 \times 0.086 = 4.62$ against $60 \times 0.168 \times 0.086 = 0.87$. On a fleet with 8 GPU-seconds of capacity per wall-second, that is 58% versus 11%.

And here is the tension that makes this a genuinely hard problem rather than a hash function. A perfect affinity map sends every request for the hot prefix to one replica. If that prefix carries 40% of traffic, one replica gets 40% of the load and the other seven idle. Maximising hit rate and balancing queue depth are directly opposed objectives, and every real policy is a blend.

SGLang: a router as a separate product

SGLang ships its router as its own Rust workspace, sgl-model-gateway/, with Python and Go bindings. Reading Rust is not a prerequisite here — the design and the policy semantics are what matter, and the Python launcher under sgl-model-gateway/bindings/python/src/sglang_router/ is ordinary readable Python. Eight policies are registered at this SHA:

    /// Create a policy by name (for dynamic loading)
    pub fn create_by_name(name: &str) -> Option<Arc<dyn LoadBalancingPolicy>> {
        match name.to_lowercase().as_str() {
            "random" => Some(Arc::new(RandomPolicy::new())),
            "round_robin" | "roundrobin" => Some(Arc::new(RoundRobinPolicy::new())),
            "power_of_two" | "poweroftwo" => Some(Arc::new(PowerOfTwoPolicy::new())),
            "cache_aware" | "cacheaware" => Some(Arc::new(CacheAwarePolicy::new())),
            "bucket" => Some(Arc::new(BucketPolicy::new())),
            "manual" => Some(Arc::new(ManualPolicy::new())),
            "consistent_hashing" | "consistenthashing" => {
                Some(Arc::new(ConsistentHashingPolicy::new()))
            }
            "prefix_hash" | "prefixhash" => Some(Arc::new(PrefixHashPolicy::with_defaults())),
            _ => None,
        }
    }

cache_aware: the approximate tree, and the escape hatch

The policy's own header comment is unusually candid about what it is doing:

    This strategy maintains an approximate radix tree for each worker based on request history,
    eliminating the need for direct cache state queries. The tree stores raw text characters
    instead of token IDs to avoid tokenization overhead.

    Process:
    a. For each request, find the worker with the highest prefix match
    b. If match rate > cache_threshold:
    Route to the worker with highest match (likely has relevant data cached)
    c. If match rate ≤ cache_threshold:
    Route to the worker with smallest tree size (most available cache capacity)
    d. Background maintenance:
    Periodically evict least recently used leaf nodes to prevent memory overflow

Three design commitments are packed into that. Approximate: the router never asks a replica what it holds. Characters, not tokens: the tree is keyed on raw request text, so the router does not have to run a tokenizer on the hot path — but its notion of a shared prefix is therefore character-level, and can disagree with the engine's 16-token block hashing (§2.3) and RadixAttention's token-level tree (§2.4). History-driven: the tree records what was sent, never what was kept.

The load check runs before the tree lookup, and short-circuits it:

        let (min_load, max_load) = workers.iter().fold((usize::MAX, 0usize), |(min, max), w| {
            let load = w.load();
            (min.min(load), max.max(load))
        });
        let min_load = if min_load == usize::MAX { 0 } else { min_load };

        // Check if load is imbalanced
        let is_imbalanced = max_load.saturating_sub(min_load) > self.config.balance_abs_threshold
            && (max_load as f32) > (min_load as f32 * self.config.balance_rel_threshold);

        if is_imbalanced {
            return self.select_worker_min_load(
                workers,
                &request_text,
                &healthy_indices,
                &tree_key,
                max_load,
                min_load,
            );
        }

Both conditions must hold — an absolute gap and a ratio. This is the blend the tension demands: affinity while the fleet is level, shortest-queue the moment one replica falls behind. Note the &&. A fleet of two where one worker has 100 in flight and the other 90 is not "imbalanced" under the Rust defaults ($100-90 = 10 \not> 32$), so the router keeps concentrating on the hot replica.

When balanced, the tree decides:

            let result = tree.prefix_match_with_counts(text);
            let match_rate = if result.input_char_count == 0 {
                0.0
            } else {
                result.matched_char_count as f32 / result.input_char_count as f32
            };

            // Select worker without String allocation
            let selected_idx = if match_rate > self.config.cache_threshold {
                // Cache hit path: find worker by URL (compare &str directly, no allocation)
                let tenant_url: &str = &result.tenant;
                workers
                    .iter()
                    .position(|w| w.url() == tenant_url)
                    .filter(|&idx| workers[idx].is_healthy())

match_rate is a fraction of the whole input, not an absolute prefix length. Hold that thought; it is the sharpest edge in the chapter and §7 returns to it.

Whichever branch fires, the router then writes the request back into the tree, tagging this worker as the "tenant" of that prefix — including in the imbalanced branch, so the model stays current even while affinity is suspended:

            if let Some(idx) = selected_idx {
                // Update the tree with this request (use worker URL directly, no allocation)
                tree.insert(text, workers[idx].url());

Decay is a background LRU sweep, not a subscription to engine evictions:

        // Start background eviction thread if configured
        let eviction_task = if config.eviction_interval_secs > 0 {
            let trees_clone = Arc::clone(&trees);
            let max_tree_size = config.max_tree_size;

            Some(PeriodicTask::spawn(
                config.eviction_interval_secs,
                "Eviction",
                move || {
                    for tree_ref in trees_clone.iter() {
                        let tree_key = tree_ref.key();
                        let tree = tree_ref.value();
                        tree.evict_tenant_by_size(max_tree_size);

evict_tenant_by_size walks the tree, pushes every leaf into a min-heap ordered by tenant_last_access_time, and pops LRU leaves until each tenant's character count is at or below max_size (sgl-model-gateway/src/policies/tree.rs:L718-L753). Pure LRU on the router's own record of what it sent.

Session affinity

Cache-aware routing already gives multi-turn chat most of what it needs, because turn n shares a long prefix with turn n−1. That only works if the router keys on the whole conversation, which is a lesson the project learned the hard way:

    /// Builds the text used for cache-aware routing of a chat request.
    ///
    /// This must reflect the *full* conversation (system prompt, prior turns,
    /// the current message and tool context) so that KV-cache prefix matching
    /// routes to the worker that actually shares the most prefix. Using only the
    /// first message ignores the conversation history that drives KV reuse in
    /// multi-turn chats. See https://github.com/sgl-project/sglang/issues/26263.
    ///
    /// Returns `None` when the conversation has no text to route on, preserving
    /// the prior behavior of not feeding an empty key into prefix matching.
    fn build_chat_request_text(body: &ChatCompletionRequest) -> Option<String> {

For explicit stickiness there are two policies with deliberately different failure behaviour. consistent_hashing hashes a routing key onto a ring, so a scale event moves only ~1/N of keys — and it will infer a key from stable headers when the client sends none:

        // Priority 3: Implicit routing key from stable headers (session affinity)
        let implicit_key = info.headers.and_then(|h| {
            h.get("authorization")
                .or_else(|| h.get("x-forwarded-for"))
                .or_else(|| h.get("cookie"))
                .and_then(|v| v.to_str().ok())
                .filter(|s| !s.is_empty())
        });

        if let Some(key) = implicit_key {
            return match Self::find_by_consistent_hash(workers, info, key) {
                Some(idx) => (Some(idx), Branch::RoutingKeyHit),
                None => (None, Branch::NoHealthyWorkers),
            };
        }

manual is stronger and states its contract up front:

//! Manual routing policy based on routing key header
//!
//! This policy provides sticky session routing where each unique routing key
//! is consistently mapped to the same worker. Unlike consistent hashing,
//! this policy:
//! - Does NOT redistribute any sessions when workers are added
//! - Only remaps sessions when their assigned worker becomes unhealthy
//! - Maintains up to 2 candidate workers per routing key for fast failover
//!
//! Use this when you need stronger stickiness guarantees than consistent hashing,
//! for example with stateful chat sessions where context is stored on the worker.
//!
//! ## Header
//! - `X-SMG-Routing-Key`: The routing key for sticky session routing

The mechanism is a DashMap from routing key to a bounded candidate list, refreshed on every hit and TTL-evicted after max_idle_secs (default 4 hours). The occupied-but-unhealthy branch is the interesting one: it silently re-assigns and appends the new worker to the candidate list (manual.rs:L170-L192). What breaks affinity, then, is exactly three things: the replica dies (health check flips it, the candidate list re-rolls), scale-down removes it (registry removal makes it un-selectable), and TTL expiry. Only the first two are visible to the user, as a sudden cold turn in the middle of a long conversation.

vLLM: no router in-tree, but the ground truth exported

At a556f3f, vLLM has no first-class router in the repository. What exists is a demo proxy under examples/, and its entire policy surface is this:

class RoundRobinSchedulingPolicy(SchedulingPolicy):
    def __init__(self):
        super().__init__()

    def schedule(self, cycler: itertools.cycle) -> str:
        return next(cycler)

One policy, over two itertools.cycle objects — one for prefill instances, one for decode. The dispatch is sequential rather than concurrent: prefill first with max_tokens = 1, awaited to completion, then decode.

    async def create_completion(self, raw_request: Request):
        try:
            request = await raw_request.json()

            kv_prepare_request = request.copy()
            kv_prepare_request["max_tokens"] = 1

            prefill_instance = self.schedule(self.prefill_cycler)
            try:
                async for _ in self.forward_request(
                    f"http://{prefill_instance}/v1/completions", kv_prepare_request
                ):
                    continue
            except HTTPException as http_exc:
                self.remove_instance_endpoint("prefill", prefill_instance)
                raise http_exc

            # Perform kv recv and decoding stage
            decode_instance = self.schedule(self.decode_cycler)

That asymmetry is itself the finding, and it is a deliberate architectural split rather than an omission. vLLM's contribution to routing is not a router; it is a feed of ground truth for somebody else's router. Turn on --kv-events-config and the engine publishes every block insertion and eviction over ZMQ:

class BlockStored(KVCacheEvent):
    block_hashes: list[ExternalBlockHash]
    parent_block_hash: ExternalBlockHash | None
    token_ids: list[int]
    block_size: int
# ...
class BlockRemoved(KVCacheEvent):
    block_hashes: list[ExternalBlockHash]
    medium: str | None
    group_idx: int | None = None
    locality: str | None = None

The publisher keeps a replay buffer so a router that restarts or drops frames can resynchronise — "subscribers can request missed batches by sending the starting sequence number as an 8-byte big-endian integer" (vllm/distributed/kv_events.py:L290-L313), with replay_endpoint and buffer_steps (default 10,000) in vllm/config/kv_events.py:L27-L36.

The contrast is clean and worth stating precisely:

The published effect of closing this gap, cited from vLLM's own integration docs (Llama 3.1 70B on AMD MI300X): "3x higher output throughput and 2x faster TTFT from prefix-aware routing vs round-robin" (docs/deployment/integrations/llm-d.md:L23). That is somebody else's benchmark on hardware this book has not touched; treat the ratio, not the absolute.

The most readable artefact in either repository is mini_lb.py, SGLang's 462-line Python load balancer for prefill/decode disaggregation. It is explicitly a debugging tool:

    def _validate_router_args(self, router_args: RouterArgs):
        logger.warning(
            "\x1b[33mMiniLB is only for debugging purposes, it only supports random policy!\033[0m"
        )

        # NOTE: too many arguments unsupported, just validate some important ones
        if router_args.policy != "random":
            logger.warning("[MiniLB] Overriding policy to random")
            router_args.policy = "random"

        if not router_args.pd_disaggregation:
            raise ValueError("MiniLB only supports PD disaggregation mode")

Follow one POST /generate through it.

Step 1 — pick a pair. handle_generate_request calls lb.select_pair(), which draws one prefill URL (with its bootstrap port) and one decode URL independently at random (mini_lb.py:L103-L112). The router must therefore know two disjoint pools and their roles: prefill workers additionally carry a bootstrap port, decode workers do not. The Rust gateway keeps that distinction all the way into the cache tree, keying trees by pool::model so a prefill lookup can never evict a decode tenant (sgl-model-gateway/src/policies/cache_aware.rs:L76-L91).

Step 2 — mint a rendezvous token. The router rewrites the body, adding three fields:

@app.post("/generate")
async def handle_generate_request(request_data: dict):
    prefill_server, bootstrap_port, decode_server = lb.select_pair()

    # Parse and transform prefill_server for bootstrap data
    parsed_url = urllib.parse.urlparse(prefill_server)
    hostname = maybe_wrap_ipv6_address(parsed_url.hostname)
    modified_request = request_data.copy()

    batch_size = _get_request_batch_size(modified_request)
    if batch_size is not None:
        modified_request.update(
            {
                "bootstrap_host": [hostname] * batch_size,
                "bootstrap_port": [bootstrap_port] * batch_size,
                "bootstrap_room": [
                    _generate_bootstrap_room() for _ in range(batch_size)
                ],
            }
        )
    else:
        modified_request.update(
            {
                "bootstrap_host": hostname,
                "bootstrap_port": bootstrap_port,
                "bootstrap_room": _generate_bootstrap_room(),
            }
        )

_generate_bootstrap_room() is random.randint(0, 2**63 - 1) (mini_lb.py:L436-L437). It is a 63-bit rendezvous nonce, nothing more. bootstrap_host/bootstrap_port point at the KV bootstrap server that only prefill instances start:

    if disagg_mode == DisaggregationMode.PREFILL:
        # only start bootstrap server on prefill tm
        kv_bootstrap_server_class = get_kv_class(
            transfer_backend, KVClassType.BOOTSTRAP_SERVER
        )
        bootstrap_server = kv_bootstrap_server_class(
            host=server_args.host,
            port=server_args.disaggregation_bootstrap_port,
        )

Step 3 — dispatch the same body to both. This is the pattern worth internalising:

        async with aiohttp.ClientSession(
            timeout=aiohttp.ClientTimeout(
                total=self.timeout
            )  # Add timeout for request reliability
        ) as session:

            tasks = [
                session.post(f"{prefill_server}/{endpoint}", json=prefill_req),
                session.post(f"{decode_server}/{endpoint}", json=decode_req),
            ]

            # Wait for both responses to complete. Prefill should end first.
            prefill_response, decode_response = await asyncio.gather(*tasks)

Why the same body to both? Because the decode instance needs the full request to allocate its own KV pages, set up sampling state and know the output length — it is running a real request, not replaying a transcript. What it does not do is recompute the prompt: it registers bootstrap_room with the prefill side's bootstrap server, blocks in WaitingForInput, and receives the KV pages over the transport that §1.6 owns. The room number is the join key between two independent HTTP requests to two independent processes.

The comment "Prefill should end first" is the whole invariant in four words. If it does not, something is wrong: the decode instance is blocked waiting for KV that will never arrive. The production Rust router encodes that suspicion explicitly, polling both futures with biased so prefill is checked first, and treating an early decode completion as a rejection:

        // Poll both until prefill resolves; decode normally resolves later, but
        // may resolve first if it rejects the request outright.
        let prefill_result;
        let mut decode_early: Option<Result<reqwest::Response, reqwest::Error>> = None;
        loop {
            tokio::select! {
                biased;
                pr = &mut prefill_fut => {
                    prefill_result = pr;
                    break;
                }
                dr = &mut decode_fut, if decode_early.is_none() => {
                    decode_early = Some(dr);
                }
            }
        }

        // Decode can't generate without prefill's KV, so any prefill failure
        // (non-2xx / transport error) dooms the paired decode request, which would
        // otherwise block in WaitingForInput until the 300s disaggregation
        // timeout. Drop the decode future to close its connection; the decode
        // engine then detects the disconnect and aborts the request in ~4-8s.

Step 4 — return decode's response, not prefill's. The client's tokens come from the decode server. The prefill response is discarded, except when return_logprob is set, in which case the router splices prefill's input_token_logprobs onto decode's output (mini_lb.py:L143-L156) — the prompt logprobs physically only exist on the machine that computed the prompt.

Autoscaling a stateless web service is a solved problem because a new replica is useful within a second of starting. An LLM replica is not. Here is what a scale-up event actually costs, taking the weight-load times §8.4 derived for Llama-3-70B at TP=8 (141.1 GB of bf16 weights, read once per node).

And then add the part nobody budgets for: the new replica's prefix cache is empty. It is not merely late, it is slower than its peers for as long as it takes to refill. Derive that too. A 70B TP=8 node's KV pool holds 1,332,954 tokens (§2.1); filling it with 2,048-token prefixes takes 651 prefills. One 2,048-token prefill of Llama-3-70B is $2P T + 2 L h d_h T(T{-}1) = 289.3 + 5.5 = 294.8$ TFLOP, and eight H100s at 40% MFU deliver 3,165 TFLOP/s, so 93 ms each — 61 s of aggregate prefill work in this illustrative full-cache fill model. It is not mandatory dead time: these prefills serve requests, partial caches are useful, and arrivals, repetition, routing, and eviction determine wall-clock convergence.

The router's own defaults tell you the maintainers know this:

            max_payload_size: 536_870_912,     // 512MB
            request_timeout_secs: 1800,        // 30 minutes
            worker_startup_timeout_secs: 1800, // 30 minutes for large model loading
            worker_startup_check_interval_secs: 30,

A thirty-minute default startup budget is not defensive programming, it is a statement about the domain. Three consequences follow, and none of them is an implementation detail:

Ready, not merely up

"Up" means the process accepted a TCP connection. "Ready" means it has loaded weights, allocated the KV pool, captured CUDA graphs and can produce a token. The gateway distinguishes them at two levels. In Kubernetes discovery, a pod counts only when the kubelet's own readiness gate has passed:

        let is_ready = if let Some(conditions) = &status.conditions {
            conditions
                .iter()
                .any(|condition| condition.type_ == "Ready" && condition.status == "True")
        } else {
            false
        };
# ...
    pub fn is_healthy(&self) -> bool {
        self.is_ready && self.status == "Running"
    }

At runtime, a hysteretic probe keeps flapping replicas out of the pool:

impl Default for HealthConfig {
    fn default() -> Self {
        Self {
            timeout_secs: 5,
            check_interval_secs: 30,
            endpoint: "/health".to_string(),
            failure_threshold: 3,
            success_threshold: 2,
            disable_health_check: false,
        }
    }
}

Do the arithmetic on those defaults: three consecutive failures at 30-second intervals is up to 90 seconds of routing requests into a dead replica before set_healthy(false) fires (worker.rs:L733-L782). The per-worker circuit breaker is what actually catches a fast failure; the health probe is the slow, hysteretic backstop. Note also that /health is a liveness ping, whereas /health_generate runs a real generation — mini_lb.py:L248-L257 fans it out to every prefill and decode server, which is why it is a startup gate and not a 30-second probe.

Draining

Draining a replica without killing in-flight streams is two separate acts. First, remove it from the registry — DELETE /workers/{worker_id} ends in WorkerRegistry::remove_by_url (sgl-model-gateway/src/core/worker_registry.rs:L364-L370), after which get_healthy_worker_indices can no longer select it, so no new request is routed there. Second, let the existing streams finish: the router's own shutdown path hands axum_server a grace period (sgl-model-gateway/src/server.rs:L1061-L1065) whose CLI default is 180 seconds (sgl-model-gateway/src/main.rs:L290-L292). Kill the pod inside that window and every open SSE stream terminates mid-token; §9.3 covers what the client sees.

One structural constraint on co-location while you are here: an embedding model and a chat model cannot share an engine process, because pooling and generation cannot share a batch (§7.5). A RAG deployment is therefore always at least two pools with two independent caches, and the router must model them separately — which is exactly why the cache tree is keyed by pool::model rather than by model alone.

1. cache_threshold is a ratio, and long unique suffixes suppress it. match_rate = matched_char_count / input_char_count. Send a 2,048-token system prompt followed by an 8,000-token retrieved document and the match rate is roughly $2048/10048 = 0.20$ — below both defaults, so the router takes the min-load branch and throws away a hit worth 86 ms. The threshold suppresses cache-aware routing exactly when the shared prefix is most valuable in absolute terms. If your workload is long-context RAG with a shared preamble, lower --cache-threshold deliberately and watch queue depth.

2. The Rust defaults and the Python launcher's defaults are different. Both are in-tree, both are current at this SHA:

Sources: sgl-model-gateway/src/policies/mod.rs:L96-L116 and sgl-model-gateway/bindings/python/src/sglang_router/router_args.py:L53-L64. The max_tree_size gap is four orders of magnitude and it matters. 10,000 characters per tenant is roughly 2,500 tokens — one system prompt — so the Rust default forgets almost everything every 30 seconds. 2²⁶ characters is roughly 16.8M tokens, about 39× the 428,569 tokens a Llama-3-8B replica can actually hold (derived, at ~4 characters per token), so the Python default believes in prefixes the engine evicted long ago. Neither default is calibrated to your pool; compute $M \times L_p \times 4$ and set it.

3. The balance guard almost never fires at moderate load. With the Python defaults, "imbalanced" needs $(\max-\min) > 64$ and $\max > 1.5\min$. At 60 req/s across 8 replicas with ~1-second requests, mean in-flight is 7.5 per replica; a 64-request gap is a fleet already in trouble. Concentration on a hot prefix therefore goes uncorrected until it is severe. If you want the guard to engage earlier, --balance-abs-threshold is the knob, and the cost is hit rate.

4. load() is the router's in-flight counter, not the engine's queue depth. The polling LoadMonitor exists, but it feeds only power-of-two:

            let power_of_two_policies = policy_registry.get_all_power_of_two_policies();

            if power_of_two_policies.is_empty() {
                debug!("No PowerOfTwo policies found, skipping load fetch");
                continue;
            }

            let result = WorkerManager::get_all_worker_loads(&worker_registry, &client).await;

So cache_aware's balance decision is made on requests the router has dispatched and not yet seen complete. A replica that is thrashing its KV pool but holding few concurrent requests reads as lightly loaded.

5. A stale tenancy silently disables affinity. If the tree's tenant for a prefix is gone or unhealthy, the router removes the tenancy and falls back to healthy_indices.first() — the same worker for every such request until the tree re-learns (cache_aware.rs:L492-L512). And if the pool tree was never seeded at all, the log is explicit about the consequence:

            warn!(
                "cache_aware: no tree found for key '{}', falling back to random \
                 worker selection — pool tree was not seeded \
                 (init_pd_cache_aware_policies missed or a race during worker \
                 registration); cache affinity is effectively disabled until this \
                 clears",
                tree_key
            );

Grep for that string first when hit rate is inexplicably at random-routing levels.

6. In P/D mode, a prefill failure strands the decode request for 300 seconds. The decode instance is blocked in WaitingForInput on a bootstrap_room that will never be filled. The router mitigates by dropping the decode future so the engine sees a disconnect and aborts in ~4–8 s (pd_router.rs:L728-L732) — but a router that merely fires and forgets, as a naive proxy would, leaves decode slots pinned for five minutes each.

Two replicas, two policies, one workload with a shared prefix. No cluster required — two GPUs, or two CPU-mode servers if you only want to watch the routing decisions.

# two separate GPUs, each large enough for one replica; enable engine metrics
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30001 --enable-metrics &
CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30002 --enable-metrics &

# router A: the wrong answer
python3 -m sglang_router.launch_router \
  --worker-urls http://127.0.0.1:30001 http://127.0.0.1:30002 \
  --policy round_robin --port 30000

# router B: the right one, with the tree budget sized to the pool
python3 -m sglang_router.launch_router \
  --worker-urls http://127.0.0.1:30001 http://127.0.0.1:30002 \
  --policy cache_aware \
  --cache-threshold 0.3 --max-tree-size 8388608 \
  --eviction-interval 60 --log-level debug --port 30000

# hit rate is reported by the engines, not the router:
curl -s http://127.0.0.1:30001/metrics | grep -i cache
curl -s http://127.0.0.1:30002/metrics | grep -i cache

# and the router's own view:
curl -s http://127.0.0.1:30000/workers

Drive it with a generator that emits, say, 40 distinct 2,000-token system prompts and short user turns, and compare cached-token counters between the two runs. Then flip one knob at a time: raise --cache-threshold to 0.9 and watch affinity collapse; drop --balance-abs-threshold to 2 and watch it collapse for the opposite reason.

For the vLLM side, watch the ground-truth feed a router would consume:

vllm serve "$MODEL_NAME" \
    --port 8100 \
    --max-model-len 100 \
    --enforce-eager \
    --gpu-memory-utilization 0.8 \
    --trust-remote-code \
    --kv-events-config \
    '{"enable_kv_cache_events": true, "publisher": "zmq", "topic": "kv-events"}' &

Run examples/features/kv_events/kv_events_subscriber.py against it and send two prompts that share a prefix. You will see one BlockStored batch for the first and a shorter one for the second — that difference is precisely the signal SGLang's router is guessing at.

• SGLang sgl-model-gateway — the router workspace. Its README is the only prose documentation of the policy set; §"Load Balancing Policies" names the flags each policy reads.
• SGLang issue #26263 — cache-aware routing keyed on only the first chat message, which broke multi-turn affinity. The fix comment is quoted in §4 and is the clearest short statement of why the routing key must be the whole conversation.
• SGLang PR #19524 — cited in pd_router.rs:L697-L699 as the origin of the upstream-cancel behaviour that makes a client disconnect cancel the paired decode request.
• SGLang: RadixAttention — the per-engine cache the router is trying to keep hot. §2.4 covers it properly; note that the engine's tree is over tokens and the router's is over characters.
• llm-d: precise prefix-cache-aware routing and vLLM production-stack — the two out-of-tree routers vLLM's docs point at. Both consume the KV event stream described in vllm/distributed/kv_events.py.
• vLLM disagg_proxy_multiturn.py — a proxy that keys cached KV-transfer parameters by a non-standard conversation_id so turn n+1's prefill can read turn n's decode blocks. A different answer to the same multi-turn problem: move the cache instead of the request.
• Preble / prefix-aware scheduling literature and the Mooncake paper — the academic framing of the affinity-versus-balance tradeoff this chapter derives from first principles.
• Neighbours: §1.6 for the KV transport behind bootstrap_room; §2.3 and §2.4 for the caches routing is trying to hit; §5.3 for DP-rank routing inside one engine; §9.3 for the single-request lifecycle the router wraps; §8.4 for the cold-start arithmetic reused in Figure 4; §13.1 for where this asymmetry sits in the wider comparison.