Workload characterization

Two teams run the same command against the same server. Team A summarises support tickets: 8,000 input tokens, 200 output tokens. Team B runs a chat assistant: 500 in, 500 out. Both saturate the GPU. Both report "Llama-3-8B on an H100."

Team A would see 447 output tokens/s. Team B, 12,796. Neither is wrong. Neither is comparable to the other. And neither number tells you anything about what the third team — the one running a coding agent at 1,000 in and 8,000 out — is going to see.

The derivation uses only constants this book has already established. Prefill of $S$ tokens costs $t_{\text{pre}}(S) = 40.6\,\mu\text{s} \cdot S + 6.63{\times}10^{-10}\,\text{s} \cdot S^2$ — reproducing §1.1's 86 ms at 2,048 tokens and 2,041 ms at 32,768 to within 0.2%. The KV pool holds 428,569 resident tokens at 128 KiB/token (§2.1), and filling it makes a decode step cost $4.48\,\text{ms} + 56.2\,\text{GB}/3.35\,\text{TB/s} = 21.25\,\text{ms}$, regardless of workload.

Same silicon, same weights, same scheduler. The spread comes entirely from two numbers per request. That is the thesis of this chapter and of Part 10: workload shape is not a footnote on a benchmark, it is the dominant term. Before you tune a flag, before you compare two engines, before you believe a blog post, you characterise the workload.

A workload is not "chat" or "RAG". Those are product categories. A workload, for the purpose of predicting what a GPU will do, is four distributions:

Axes 1 and 2 place you on the roofline. Axis 3 decides whether the queue in front of the roofline is empty or 300 deep. Axis 4 decides how much of axis 1's cost you actually pay.

The table in §1 came out of a two-constant model. Deriving it is worth five minutes because the model then predicts any workload you care to name.

Prefill constant. From FORMULAS, $F_{\text{prefill}}(S) = 2PS + 2Lh d_h S^2$. Calibrating against the book's 86 ms for a 2,048-token Llama-3-8B prefill gives an effective throughput of 395 TFLOP/s — 39.9% MFU — and splits into a per-token term and a quadratic term:

The quadratic term is 3.2% of prefill at $S = 2{,}048$, 11.6% at 8,000, and 34.9% at 32,768. Below about 4k tokens you can pretend prefill is linear; above it you cannot.

Decode constant. A saturated engine runs at $B = N_{\max}/\bar{s}$ concurrent requests, where $N_{\max} = 428{,}569$ is the pool in tokens and $\bar{s}$ is the mean resident sequence length. Each step costs $t_{\text{step}} = 4.48\,\text{ms} + kN_{\max}/\beta = 21.25$ ms — a machine constant, not a workload one, because the pool is full either way. The engine time one request's decode phase consumes is its share of those steps:

Read that carefully. Decode engine time is bilinear in output length and resident length. Doubling the input length doubles the cost of every decode step the request will ever take, because it halves the batch that shares the weight stream. This is the mechanism by which axis 1 controls decode performance, and it is invisible if you only look at prefill.

The ratio that predicts everything

For a symmetric workload with $S_{\text{in}} = S_{\text{out}} = x$ and $\bar{s} = 1.5x$, prefill and decode cost exactly the same engine time at:

That is the balance point for Llama-3-8B on an H100, and it is why "chat" feels like the neutral workload: a 500-in/500-out request lands within 10% of a 50/50 split, by coincidence of model shape and hardware. Move either number and the split collapses fast — $t_{\text{pre}}/t_{\text{dec}}$ is 4.58 for summarisation and 0.021 for agentic, a factor of 220 between them.

Where each lands on the roofline

Prefill intensity is $I(T) = 2Td/(b(2T+d))$, and every prefill in this chapter — 402 at $T=500$, 1,631 at $T=8{,}000$ — sits far right of the 295 ridge (§0.4). Prefill is compute-bound in all three workloads. The interesting variation is in decode, where the whole-step intensity is $B(2P + 4Lhd_hs)$ FLOPs over 71.2 GB of traffic — 15.01 GB of streamed weights (the 7.505 B non-embedding parameters; the embedding table is resident but never read, §0.4) plus the 56.17 GB pool:

A chat decode step does 8.9× the arithmetic of a summarisation decode step in the same 21.25 ms, because 571 rows amortise the weight stream instead of 52. If someone shows you a "5% MFU" decode measurement and calls it an engine problem, ask what the input lengths were.

Axis 1 — input length: the mean is the wrong statistic

Take a mixed production workload: 80% short chat turns at 600 tokens, 15% short documents at 4,000, 5% long documents at 16,000. Mean input length: 1,880 tokens. Now take a synthetic workload where every request is exactly 1,880 tokens. Identical means, identical token volume, identical arrival rate.

The mean prefill time differs by 10%. The mean queueing delay differs by 4.4×, because §1.2's Kingman correction scales the wait by $(C_a^2 + C_s^2)/2$, and service-time variability comes straight off the input-length distribution. The table sets $C_s^2$ equal to the length $C^2$, which is exact only if prefill were linear in $S$; carrying the $S^2$ term through raises $C_s^2$ to 4.14 and the gap to 5.1×, so 4.4× is the conservative reading, not the generous one. Five percent of requests carry 47% of the prefill work and set the tail. Without chunked prefill, each of those 16k prefills stalls every decoding request in the batch for 820 ms — the head-of-line blocking derived in §1.1.

Axis 2 — output length: unknown at admission

Input length is known when the request arrives. Output length is not. The scheduler has, at best, max_tokens — an upper bound the client picked, often the model's context limit. This asymmetry is why admission control has to guess, and both engines guess differently.

SGLang guesses explicitly, with a scalar it tunes online. new_token_ratio scales the KV reservation for each running request (§1.4), and retraction snaps it up towards the observed generation fraction before decaying back — additive increase, additive decrease, on KV admission. Read that as a feedback loop that learns your output-length distribution at runtime. If your workload's real generation fraction is 5% of max_tokens, the controller settles low and admits aggressively. If it is 90%, it settles high and your effective batch shrinks. Two workloads with identical input lengths and identical arrival rates will therefore run at different concurrency, purely from axis 2.

Axis 3 — the arrival process changes the physics

§1.2 owns the queueing result: at $\rho \in [0.5, 0.95]$, 89–99% of p99 TTFT is queueing. It states Kingman's correction and leaves the arrival-variability treatment here. So:

with $C_a^2$ the squared CV of inter-arrival times and $C_s^2$ that of service times. Both harnesses let you set $C_a^2$, and in vLLM's case the mapping is exact. vLLM draws inter-arrival gaps from a Gamma distribution:

            if current_request_rate == float("inf"):
                delay_ts.append(0)
            elif burstiness == float("inf"):
                # when burstiness tends to infinity, the delay time becomes constant
                # and tends to the inverse of the request rate
                delay_ts.append(1.0 / current_request_rate)
            else:
                theta = 1.0 / (current_request_rate * burstiness)

                # Sample the request interval from the gamma distribution.
                # If burstiness is 1, it follows exponential distribution.
                delay_ts.append(np.random.gamma(shape=burstiness, scale=theta))

Gamma with shape $\kappa = \texttt{burstiness}$ and scale $\theta = 1/(\lambda\kappa)$ has mean $\kappa\theta = 1/\lambda$ and variance $\kappa\theta^2 = 1/(\lambda^2\kappa)$. So:

The flag is the queueing parameter, inverted. At $\rho = 0.8$ with the mixed workload above ($C_s^2 = 3.38$), the same offered rate gives $\mathbb{E}[W] = 7.26\,S$ at --burstiness 4, $8.76\,S$ at the Poisson default, and $14.76\,S$ at --burstiness 0.25. Two engineers benchmarking "at 10 req/s" can differ by 2× in mean queueing delay and 0% in throughput. SGLang has no equivalent knob — it samples exponential only, i.e. $C_a^2 = 1$, permanently:

            yield request

            if request_rate == float("inf"):
                # If the request rate is infinity, then we don't need to wait.
                continue

            # Sample the request interval from the exponential distribution.
            interval = np.random.exponential(1.0 / request_rate)
            # The next request will be sent after the interval.
            await asyncio.sleep(interval)

Open loop versus closed loop is not a methodology preference; it is a different experiment. Both harnesses implement an open-loop/semaphore hybrid, not a true closed loop, around the request coroutine:

    pbar = None if disable_tqdm else tqdm(total=len(input_requests))

    semaphore = (
        asyncio.Semaphore(max_concurrency)
        if max_concurrency
        else contextlib.nullcontext()
    )

    async def limited_request_func(request_func_input, session, pbar):
        async with semaphore:
            return await request_func(
                request_func_input=request_func_input, session=session, pbar=pbar

With --max-concurrency N and no rate limit, Little's Law runs backwards: you have fixed $\bar{L} = N$, so $\lambda = N/\bar{W}$ is an output. If the engine slows, arrivals slow with it. The queue never grows, TTFT never diverges, and you cannot observe overload — the experiment has no $\rho$ to exceed 1. Resident tokens are bounded by $N\bar{s}$, so preemption may never fire and that entire code path goes unmeasured. In open loop you fix $\lambda$, $\rho = \lambda/\mu$ is yours to choose, and at $\rho \to 1$ the waiting queue grows without bound — which is precisely when admission control, preemption, and starvation handling become load-bearing. The two modes execute different code. §10.3 owns which to run and how; the physics is that they measure different systems.

Axis 4 — prefix sharing: position matters more than volume

Both engines match prefixes and only prefixes: vLLM by chaining block hashes from token 0 (§2.3), SGLang by walking a radix tree from the root (§2.4). A 2,000-token preamble at the front of every request is worth $2000 \times 40.6\,\mu\text{s} = 81$ ms of prefill per request. The same 2,000 tokens appended after per-request content is worth exactly zero. And §2.3's alignment result makes this brittle in one direction: prepending a single token — a user id, a timestamp — changes $h_0$, and the chain makes every subsequent hash miss. You do not lose a block; you lose all of them.

That last line is not decorative. SGLang's cache-aware router gates on match_rate = matched_char_count / input_char_count and falls back to load balancing below cache_threshold (§9.4). A ratio penalises exactly the workload where the absolute shared prefix is largest: long-context RAG, where a valuable 2,048-token preamble is diluted by an 8,000-token retrieved document to a match rate of 0.20 — under both defaults. Characterise your sharing in tokens matched per request and separately as a ratio, because the router only sees the ratio and the GPU only feels the tokens.

§10.2 covers the harnesses; what matters here is the distribution each loader emits, because that is the workload you are silently choosing. In vLLM the real code is vllm/benchmarks/datasets/datasets.py — 4,814 lines — not the deprecated root-level shim.

The same flag, two different distributions

--random-range-ratio exists in both harnesses and means different things. vLLM samples symmetrically around the target length:

    output_low = math.floor(output_len * (1 - output_range_ratio))
    output_high = math.ceil(output_len * (1 + output_range_ratio))
    # Ensure the lower bound for output length is at least 1 to
    # prevent sampling 0 tokens.
    output_low = max(output_low, 1)
    output_high = max(output_high, 1)
# ...
    input_lens = rng.integers(input_low, input_high + 1, size=num_requests)
    output_lens = rng.integers(output_low, output_high + 1, size=num_requests)

SGLang samples downward from the target, treating the ratio as a floor:

def compute_random_lens(full_len: int, range_ratio: float, num: int) -> List[int]:
    # full_len=0 is valid for embedding benchmarks where no output tokens are generated
    if full_len <= 0:
        return [0] * num
    return np.random.randint(
        max(int(full_len * range_ratio), 1),
        full_len + 1,
        size=num,
    ).tolist()

At the shipped default, the "same" synthetic workload has a 2× different mean input length and a variance that is zero in one engine and $C^2 = 1/3$ in the other. The ratio's meaning is inverted: 1.0 is maximum variance in vLLM and zero variance in SGLang, where vLLM rejects it outright (input_range_ratio must be in [0, 1)). Any cross-engine comparison run on random with a non-zero range ratio is comparing two workloads.

ShareGPT is not one dataset either

Both loaders keep only the first two turns — SGLang's comment says so in as many words at python/sglang/benchmark/datasets/sharegpt.py:L88, and vLLM indexes entry["conversations"][0] and [1] directly at vllm/benchmarks/datasets/datasets.py:L1393-L1396. That alone deletes the multi-turn prefix growth that makes real chat cacheable. But vLLM also prunes:

def is_valid_sequence(
    prompt_len: int,
    output_len: int,
    min_len: int = 4,
    max_prompt_len: int = 1024,
    max_total_len: int = 2048,
    skip_min_output_len_check: bool = False,
) -> bool:
    """
    Validate a sequence based on prompt and output lengths.

    Default pruning criteria are copied from the original `sample_hf_requests`
    and `sample_sharegpt_requests` functions in benchmark_serving.py, as well as
    from `sample_requests` in benchmark_throughput.py.
    """

Every ShareGPT conversation with a prompt over 1,024 tokens, or a prompt-plus-completion over 2,048, is dropped. SGLang applies no such bound by default — --sharegpt-context-len defaults to None. vLLM's ShareGPT has had its tail amputated; SGLang's has not. The two harnesses, pointed at the same JSON file, produce different length distributions, and the difference is exactly the part of the distribution that dominates queueing.

You cannot tune against a dataset. Here is the procedure, using only what the engines already export.

vLLM. Every finished request carries a full timing and token breakdown:

@dataclass
class FinishedRequestStats:
    """Stats associated with a finished request."""

    finish_reason: "FinishReason"
    request_id: str | None = None
    e2e_latency: float = 0.0
    num_prompt_tokens: int = 0
    num_generation_tokens: int = 0
    max_tokens_param: int | None = None
    queued_time: float = 0.0
    prefill_time: float = 0.0
    inference_time: float = 0.0
    decode_time: float = 0.0
    mean_time_per_output_token: float = 0.0
    is_corrupted: bool = False
    num_cached_tokens: int = 0

That single dataclass gives you axes 1, 2 and 4 directly, plus queued_time, which is Kingman's $W$ measured rather than predicted. PrefillStats splits the cache term further — local prefix cache versus external KV transfer versus tokens newly written to the cache (vllm/v1/metrics/stats.py:L258-L276) — which is what you need to tell "my workload shares prefixes" from "my workload re-reads its own recent prefixes".

Without touching the code, the Prometheus endpoint carries the distributions as histograms with 1–2–5 decade buckets up to max_model_len:

        histogram_num_prompt_tokens_request = self._histogram_cls(
            name="vllm:request_prompt_tokens",
            documentation="Number of prefill tokens processed.",
            buckets=build_1_2_5_buckets(max_model_len),
            labelnames=labelnames,
        )
        self.histogram_num_prompt_tokens_request = create_metric_per_engine(
            histogram_num_prompt_tokens_request, per_engine_labelvalues
        )

        histogram_num_generation_tokens_request = self._histogram_cls(
            name="vllm:request_generation_tokens",
            documentation="Number of generation tokens processed.",
            buckets=build_1_2_5_buckets(max_model_len),
            labelnames=labelnames,
        )

Twelve buckets is coarse — [1, 2, 5, 10, … 5000] at max_model_len = 8192 — so a p99 read off this histogram lands somewhere in [2000, 5000). Good enough to classify the workload, not good enough to size a KV pool. For that, log per-request.

SGLang. Start the server with --export-metrics-to-file plus the mandatory --export-metrics-to-file-dir, and FileRequestMetricsExporter writes one JSON record per request combining the request parameters with meta_info (python/sglang/srt/observability/request_metrics_exporter.py:L33-L62). The cache term lives there:

            if not isinstance(recv_obj, BatchEmbeddingOutput):
                meta_info.update(
                    {
                        "reasoning_tokens": recv_obj.reasoning_tokens[i],
                        "completion_tokens": recv_obj.completion_tokens[i],
                        "cached_tokens": recv_obj.cached_tokens[i],
                    }
                )

The six statistics. From that log, compute — and publish alongside every benchmark number you ever quote:

Note the cap: §2.3 established that a fully cached prompt reports a hit of at most $B\lfloor(S-1)/B\rfloor$ tokens, so a sharing rate of 1.0 is unreachable by construction. Do not chase the last percent.

Isolate one axis at a time. Fix everything else, including the seed.

for B in 0.25 1.0 4.0; do
  vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct \
    --dataset-name random --random-input-len 1024 --random-output-len 256 \
    --random-range-ratio 0.0 --num-prompts 1000 --seed 0 \
    --request-rate 12 --burstiness $B \
    --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,99 \
    --result-filename burst-$B.json --save-result
done

Throughput should be flat across the three (you fixed $\lambda$); TTFT p99 should rise as burstiness falls, by roughly the $(C_a^2+C_s^2)/2$ factor with $C_a^2 = 1/B$. That is Kingman, visible, on one GPU.

vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset-name prefix_repetition --prefix-repetition-prefix-len 2048 \
  --prefix-repetition-suffix-len 256 --prefix-repetition-num-prefixes 8 \
  --num-prompts 512 --request-rate 8 --seed 0
# then scrape the hit rate the engine itself reports:
curl -s localhost:8000/metrics | grep -E 'vllm:(prefix_cache|request_prompt_tokens)'

# server: pin the policy so a concurrency change cannot silently switch it,
# and dump one JSON record per request
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --schedule-policy fcfs \
  --export-metrics-to-file --export-metrics-to-file-dir ./reqlog

# client
python3 -m sglang.benchmark.serving --backend sglang \
  --dataset-name generated-shared-prefix --gsp-num-groups 64 \
  --gsp-prompts-per-group 32 --gsp-system-prompt-len 2048 --gsp-question-len 256 \
  --gsp-output-len 256 --gsp-group-distribution zipf --gsp-zipf-alpha 1.1 \
  --request-rate 8 --num-prompts 2048

Then re-run with --gsp-group-distribution uniform. Identical sharing rate, different sharing concentration: the Zipf run keeps its hot prefixes resident and the uniform run thrashes them out. This is the experiment that tells you whether your cache is big enough for your workload's popularity distribution rather than for its average.

• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (FAST'25). The source of the four traces SGLang replays via --dataset-name mooncake; the paper's workload analysis is the best published account of real arrival and sharing structure.
• BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems. The trace behind vLLM's burstgpt loader — note the loader keeps the lengths and discards the timestamps.
• LongBench v2, whose SGLang loader defaults to a 10-token output and is therefore the sharpest long-in/short-out probe either repo ships.
• vLLM PR #10105 — "Add Gamma-Distribution Request Generation Support for Serving Benchmark", which is where --burstiness and therefore a controllable $C_a^2$ entered the harness; #26941 later added the inf (deterministic, $C_a^2 = 0$) case.
• vLLM PR #39795 — timed trace replay, the only vLLM path that reproduces arrivals and prefix structure together; #20638 for PrefixRepetitionRandomDataset.
• SGLang PR #29215 — the agentic-trace multi-turn dataset; #3211 for the hierarchical-cache serving benchmarks in benchmark/hicache/, and #19077 for the refactor that split the dataset loaders out of bench_serving.
• John Kingman, The single server queue in heavy traffic (1961) — the diffusion approximation this chapter leans on. The modern statement is in Hopp & Spearman, Factory Physics, ch. 8.
• §10.2 for what the harnesses do with these datasets, and §10.3 for warmup, saturation, and the reporting discipline that makes two runs comparable.