Running a benchmark that is not a lie

Run this twice against the same server, same model, same GPU, back to back:

vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset-name random --random-input-len 2048 --random-output-len 256 \
  --num-prompts 500

Three of the defaults in that invocation decide the answer before the server does any work.

    parser.add_argument(
        "--request-rate",
        type=float,
        default=float("inf"),
        help="Number of requests per second. If this is inf, "
        "then all the requests are sent at time 0. "
        "Otherwise, we use Poisson process or gamma distribution "
        "to synthesize the request arrival times.",
    )

    parser.add_argument(
        "--num-warmups",
        type=int,
        default=0,
        help="Number of warmup requests.",
    )

So: all 500 requests are dispatched at $t=0$, there is no warmup, and — because vLLM's prefix-cache flush lives behind a development-mode flag we will get to in §5 — whatever the previous run left in the KV cache is still there. The first run pays cold Triton autotuning and a cold prefix cache; the second run finds a warm cache holding the tail of the first run's prompts. On a fixed-seed random dataset the two runs use the same prompts, so the second run's prefills are largely free.

Neither number is wrong. They measure two different systems. And the delta between them is comfortably larger than most of the improvements people publish.

SGLang's harness defaults differently but not better: --warmup-requests defaults to 1, and the flush that would undo that one request's cache pollution only fires in CI or when you ask for it explicitly.

    # Flush cache after warmup so the measured run does not benefit from
    # request-local prefix reuse. vLLM exposes a different, development-mode
    # endpoint for the same purpose.
    should_flush_cache = (
        "sglang" in backend and _get_bool_env_var("SGLANG_IS_IN_CI")
    ) or flush_cache
    if should_flush_cache:
        flush_server_cache(base_url, backend, flush_cache_timeout)

    time.sleep(1.0)

The thesis of this chapter: the reasons benchmarks are wrong are systematic, mechanical, and enumerable. Every one of them has a mechanism you can read in the source and a control you can apply. What follows is the enumeration.

A throughput number without the latency it was achieved at is not a measurement, it is a coordinate with one axis missing. The object a benchmark actually produces is a curve: for each offered load, one point of (throughput, latency-percentile). The shape of that curve is fixed by queueing theory and does not depend on which engine you run. What the engine changes is where the curve sits.

Three consequences drop straight out of the picture and they organise the rest of the chapter. First, below saturation the harness's reported request throughput is not an output at all — it is your --request-rate echoed back, because request_throughput=completed / dur_s (vllm/benchmarks/serve.py:L725-L734) and dur_s spans the arrival schedule you specified. Second, above saturation the throughput number is the same for every engine that can saturate, and the latency number is a function of how long you ran. Third, there is exactly one interesting point per configuration — the load at which the SLO binds — and finding it requires sweeping.

How long is warmup, really

"We discarded the first 10 requests" is not a method, because at least four separate things converge on separate timescales and none of them is 10 requests.

Take the last two quantitatively. Model the replica as a single server with mean service time $S$ and utilisation $\rho = \lambda S$, exactly as §1.2 does. Started empty, an M/M/1 queue approaches its stationary distribution with relaxation time

where $N_{\mathrm{rel}}$ is the expected number of arrivals during one spectral-relaxation timescale. This assumes stationary Poisson arrivals, exponential service and $\rho<1$; it is a planning scale, not a finite-time convergence guarantee for a batched inference engine. Pipeline fill has a different scale: one mean residence time corresponds to $\lambda\bar{W}=\bar{L}$ expected arrivals by Little's Law. That identity does not prove that discarding exactly that many requests removes a transient.

Anchor $S$ with real arithmetic. Llama-3-8B bf16 on an H100 SXM: 15.01 GB of weights, 3.35 TB/s of HBM, so the batch-1 decode floor is 4.48 ms (Formula sheet). At batch 64 with 2,048-token prompts and 256 output tokens, the mid-run context is 2,176 tokens and KV is 128 KiB per token (§2.1):

Batch 64 is well under the H100 ridge $I^{*} = 295$, so the bandwidth floor is the right bound. Now charge each request the GPU time it actually consumes, keeping the two kinds of work separate. The 255 decode steps are shared — one step advances all 64 requests by one token — so a request's share of them is $255 \times 9.93/64 = 39.6$ ms. Its prefill is not shared: 2,048 tokens of prefill is serial GPU work no other request in the batch benefits from, and at the 989.4 TFLOP/s peak it costs $2 \times 7.505{\times}10^{9} \times 2048 / 9.894{\times}10^{14} = 31$ ms.

Little's Law now reads out the residence time instead of manufacturing capacity: $\bar{W} = N/\mu = 64/14.2 = 4.5$ s at 64 in flight. The shortcut is worth naming because it is the natural mistake — "a request lives for its own 2.53 s of decode plus its own 31 ms of prefill, so $\bar{W} = 2.56$ s and $\mu = 64/2.56 = 25$ req/s". That charges the batch one prefill where it owes 64, and 64 prefills are 1.98 s of GPU time the decode steps must make room for. It is also circular: in a closed loop at $N = 64$, $\bar{W}$ is set by capacity ($\bar{W} = N/\mu$), so you cannot recover $\mu$ from a $\bar{W}$ computed as though the system were otherwise empty. Chunked prefill does not rescue it — interleaving prefill chunks into decode steps redistributes that 1.98 s, it does not delete it, and the steps it lands in get slower.

Both constants are hardware floors — 100% of HBM bandwidth for decode, 100% of peak FLOP/s for prefill — so 14.2 req/s is an upper bound on capacity, not a prediction. Substituting §1.1's realistic 86 ms for the same 2,048-token prefill puts $S$ at 126 ms and $\mu$ near 8 req/s. All derived arithmetic, no measurement.

Near $\rho\approx0.9$, the illustrative three-timescale warmup is about 1,000 arrivals and 80 seconds, around one hundred times ten requests. It grows rapidly near saturation. An initially empty, short run can understate stationary tail latency, but actual convergence depends on the workload and system. Check rolling occupancy, throughput and latency rather than treating this estimate as certification.

Why an overloaded run measures your patience

Past saturation the queue is not in steady state and never will be. Offered $\lambda > \mu$, the backlog after $T$ seconds of running is $(\lambda - \mu)T$, and a request arriving at time $t$ waits $(\lambda-\mu)t/\mu$. Offer 20% over capacity — with the $\mu = 14.2$ req/s above, $\lambda = 17$ req/s:

Notice what cancelled: $\mu$ itself. At a fixed fraction over capacity the wait depends only on the overshoot and on how long you ran, so the three numbers below are properties of your experiment, not of the engine, and hold for any model and any GPU.

Same server, same engine, same offered load — three different p99s, differing by a factor of ten, entirely because of run duration. An overload number is only interpretable alongside $T$, and even then it tells you about your load generator rather than about the engine. This is why run duration is a mandatory field in the reporting template of §8, and why comparing two published overload numbers is meaningless unless both ran for the same wall time.

This is the most consequential methodological choice in the chapter, and it is usually made by accident.

Closed loop fixes the number of requests in flight: a client holds $N$ slots and issues a new request only when one returns. Open loop fixes the arrival process: requests are emitted on a schedule that does not know or care what the server is doing. Production is open loop — your users do not wait for the previous user's response before typing.

Why closed loop flatters

Under a closed loop with $N$ slots, the offered rate is $\lambda = N/\bar{W}(\lambda)$ — the system settles at the intersection of the Little's Law line $L = N$ with the latency curve. As $N$ grows, $\lambda \to \mu$ and $\bar{W} \to N/\mu$: latency grows linearly in $N$ and never diverges. There is no configuration of a closed-loop client that produces a queue growing without bound, because the client is the admission control.

Work the difference on a 20% decode regression. Engine A: $\mu_A = 14.2$ req/s from §3. Only decode slows 20%, so $S_B=31.1+1.2(39.6)=78.62$ ms and $\mu_B\approx12.72$ req/s, versus $S_A=70.7$ ms and $\mu_A\approx14.14$ req/s.

A roughly 10.1% capacity regression can turn a stable open-loop load into a growing queue. The closed loop cannot see the second because it never offers a load the server cannot absorb; it converts every capacity deficit into a proportional latency increase, which is the single most flattering transform available.

There is a worse hybrid, and both harnesses ship it: an open arrival schedule plus a client-side semaphore. vLLM creates a task per request at its scheduled arrival time and the semaphore blocks inside the task:

    semaphore = (
        asyncio.Semaphore(max_concurrency)
        if max_concurrency
        else contextlib.nullcontext()
    )

    async def limited_request_func(request_func_input, session, pbar):
        async with semaphore:
            return await request_func(
                request_func_input=request_func_input, session=session, pbar=pbar
            )

The per-request clock starts inside request_func, after the semaphore is acquired. §1.2 works this through: time spent in the client-side queue is invisible to every reported latency. Setting --max-concurrency below the server's capacity therefore drives reported TTFT p99 down while real user wait rises without bound. If you report a latency number, you must report the concurrency limit next to it or the number means nothing.

What each harness can express

vLLM precomputes the whole arrival schedule as absolute offsets from one start_ts and sleeps to each in turn, so dispatch overhead cannot accumulate:

    start_ts = time.time()
    for request_index, request in enumerate(input_requests):
        if delay_ts[request_index] > 0:
            current_ts = time.time()
            sleep_interval_s = start_ts + delay_ts[request_index] - current_ts
            if sleep_interval_s > 0:
                await asyncio.sleep(sleep_interval_s)
        yield request, request_rates[request_index]

SGLang sleeps a fresh exponential sample after each yield, so every interval is measured from the moment the previous request was dispatched:

        input_requests_iter = iter(input_requests)
        for request in input_requests_iter:
            yield request

            if request_rate == float("inf"):
                # If the request rate is infinity, then we don't need to wait.
                continue

            # Sample the request interval from the exponential distribution.
            interval = np.random.exponential(1.0 / request_rate)
            # The next request will be sent after the interval.
            await asyncio.sleep(interval)

The achieved rate is therefore slightly below the target by the per-request dispatch cost, and the shortfall grows with client load — a bias that goes the wrong way exactly when the client is busiest. Structurally, vLLM's generator supports a tunable burstiness (Gamma shape; 1.0 is Poisson) and linear or exponential ramp-up; SGLang's is exponential only, plus timestamped trace replay. §10.2 owns the full comparison.

vLLM does the heavy warmup at startup, before the server is reachable. compile_or_warm_up_model runs a dummy forward at every size the compiler needs, warms JIT kernels, then captures the graphs:

        # We skip EPLB here since we don't want to record dummy metrics
        for size in sorted(warmup_sizes, reverse=True):
            logger.info("Compile and warming up model for size %d", size)
            self.model_runner._dummy_run(size, skip_eplb=True, remove_lora=False)
        self.model_runner.maybe_remove_all_loras(self.model_runner.lora_config)

        # Warmup and tune the kernels used during model execution before
        # cuda graph capture.
        kernel_warmup(self)

        cuda_graph_memory_bytes = 0
        if not self.model_config.enforce_eager:
            cuda_graph_memory_bytes = self.model_runner.capture_model()

kernel_warmup is a long list of model-specific JIT pre-compilations, and the comments say exactly why:

    # DSv4 mHC TileLang kernels (hc_pre/hc_post/hc_head_op) run every decoder
    # layer per token; warm them across token sizes first so the first real
    # request doesn't pay JIT cost. No-op for non-DSv4 models (gated inside).
    deepseek_v4_mhc_warmup(
        worker.get_model(),
        max_tokens=worker.scheduler_config.max_num_batched_tokens,
        cudagraph_capture_sizes=cudagraph_capture_sizes,
    )

At the end of that function the worker draws a line and starts policing it:

        # All warmup is done — start monitoring for unexpected JIT
        # compilations that would cause latency spikes during inference.
        from vllm.utils.jit_monitor import activate as activate_jit_monitor

        activate_jit_monitor(
            mode=self.observability_config.jit_monitor_mode,
            verbose=self.observability_config.jit_monitor_verbose,
        )

        # Freeze the worker heap so the GC won't scan static objects
        # (model weights, KV caches, CUDA graphs) during inference.
        freeze_gc_heap()

Note what is not on the left of that boundary: the prefix cache, the running batch, and the queue-depth distribution. Startup warmup solves the compilation problem completely and the benchmarking problem not at all.

Flushing the prefix cache

You almost always want the measured run to start with a cold prefix cache, because otherwise the result depends on what the previous run did — an unwritten, unreported input. Both engines expose a flush, asymmetrically.

SGLang's is a first-class route, and it refuses unless the engine is idle:

@app.api_route("/flush_cache", methods=["GET", "POST"])
@auth_level(AuthLevel.ADMIN_OPTIONAL)
async def flush_cache(timeout: float = Query(0.0, ge=0.0)):
    """Flush the radix cache."""
    ret = await _global_state.tokenizer_manager.flush_cache(timeout_s=timeout)
    if ret.success:
        content = (
            "Cache flushed.\nPlease check backend logs for more details. "
            "(When there are running or waiting requests, the operation will not be performed.)\n"
        )
    else:
        content = ret.message or "Flush cache failed.\n"
    return Response(
        content=content,
        status_code=200 if ret.success else HTTPStatus.BAD_REQUEST,
    )

    def flush_cache(self, empty_cache: bool = True):
        """Flush memory pools (e.g., KV cache, Mamba cache) and optionally empty device allocator cache."""
        if self.is_fully_idle():
            self.cur_batch_for_debug = None
            self.last_batch = None
            self.tree_cache.reset()
            self.req_to_token_pool.clear()
            self.token_to_kv_pool_allocator.clear()
            self.grammar_manager.clear()
            self.metrics_reporter.reset_metrics()
# ...
        else:
            logging.warning(
                f"Cache not flushed because there are pending requests. "
                f"#queue-req: {len(self.waiting_queue)}, "
                f"#running-req: {len(self.running_batch.reqs)}"
            )
            success = False
        return success

vLLM's equivalent exists only in development mode. The route is registered behind VLLM_SERVER_DEV_MODE (vllm/envs.py:L1419-L1421, vllm/entrypoints/launchers/api_server/routers.py:L34-L37):

@router.post("/reset_prefix_cache")
async def reset_prefix_cache(
    raw_request: Request,
    reset_running_requests: bool = Query(default=False),
    reset_external: bool = Query(default=False),
):
    """
    Reset the local prefix cache.
# ...
    Returns `{"success": bool}`. The reset fails (`success=false`) while
    blocks are still held, e.g. by running requests or in-flight async KV
    offload transfers; callers may retry.
    """

SGLang's harness knows about both and calls whichever matches the backend, which is the one place in either repo where the cross-engine methodology is written down:

def flush_server_cache(
    base_url: str,
    backend: str,
    flush_cache_timeout: float = _DEFAULT_SGLANG_FLUSH_CACHE_TIMEOUT,
) -> None:
    """Flush an engine's prefix cache after benchmark warmup."""
    if backend.startswith("vllm"):
        response = requests.post(
            base_url + "/reset_prefix_cache", headers=get_auth_headers()
        )
    elif backend.startswith("sglang"):
        response = requests.post(
            base_url + "/flush_cache",
            headers=get_auth_headers(),
            params={"timeout": flush_cache_timeout},
        )
    else:
        response = requests.post(base_url + "/flush_cache", headers=get_auth_headers())
    response.raise_for_status()

The sweep driver

vLLM ships the load-curve machinery as vllm bench sweep: it restarts the server per server-side configuration, runs --num-runs repetitions per point, and resets caches between them.

    VLLM_RESET_CACHE_ENDPOINTS = [
        "/reset_prefix_cache",
        "/reset_mm_cache",
        "/reset_encoder_cache",
    ]
# ...
            for endpoint in self.VLLM_RESET_CACHE_ENDPOINTS:
                res = requests.post(server_address + endpoint)
                res.raise_for_status()
# ...
        else:
            raise NotImplementedError(
                f"No implementation of `reset_caches` for `{server_cmd[0]}` server. "
                "Please specify a custom command via `--after-bench-cmd`."
            )

Those three POSTs 404 unless the server was started with VLLM_SERVER_DEV_MODE=1, and raise_for_status() turns that into a hard failure mid-sweep. That is the correct behaviour — a sweep that silently skipped the flush would produce a curve whose points are not comparable to each other — but it is a trap the first time.

Follow one default invocation, --num-prompts 500 --request-rate inf, against a freshly started vLLM server, in code order.

1. The readiness probe does not fire. --ready-check-timeout-sec defaults to 0 (vllm/benchmarks/serve.py:L1899-L1905) and wait_for_endpoint sits behind if ready_check_timeout_sec > 0, so the default path prints "Skipping endpoint ready check." and sends nothing (:L858-L874). Pass a positive timeout and it does send one full-length test_input — built from input_requests[0] (:L826-L832) — whose blocks then sit in the prefix cache for the entire measured run.
2. num_warmups is 0, so the warmup block at L876-L899 is skipped too. On --dataset-name random the only pre-run traffic is _align_prompts_to_server_tokenizer POSTing to /tokenize (:L2125-L2128), which allocates no KV. State at $t=0$ on a freshly started server: prefix cache empty, running batch empty, queue empty — so request 1 pays anything the startup sweep missed.
3. benchmark_start_time = time.perf_counter() at L999, then the async for at L1013 drains get_request. With request_rate == inf, every delay_ts entry is 0, so all 500 asyncio.create_task calls fire in one event-loop pass.
4. The server receives a 500-request thundering herd. The scheduler admits what fits, the rest queue. Queue depth goes from 0 to ~440 in one step — which, on SGLang, would cross the 128-request LPM boundary immediately and for the entire run.
5. Requests complete. As they do, the batch shrinks; the last ~64 requests run at a batch size that falls monotonically toward 1, so their decode steps are the fastest in the run (fewer KV bytes per step) and their TPOTs pull the mean down.
6. benchmark_duration = time.perf_counter() - benchmark_start_time at L1077. Throughput is 500 divided by that makespan.

So the reported number is the makespan of a 500-request burst through a transient that starts at batch 0, peaks at whatever max_num_seqs allows, and decays back to batch 0. The only regime it never visits is steady state. It is a perfectly good capacity probe — it answers "what is $\mu$?" — and it is useless as a latency measurement, because every latency percentile it reports is dominated by the queueing of the initial burst, whose magnitude is set by --num-prompts.

A number is checkable when a stranger with the same hardware can reproduce it. That requires the following, and no harness records all of it for you — vLLM's result JSON stores client-side arguments and a free-form --metadata KEY=VALUE (vllm/benchmarks/serve.py:L2214-L2240), and nothing about the server. SGLang does better: it GETs /server_info and embeds the whole server configuration in the result (python/sglang/benchmark/serving.py:L1779-L1790). Either way, the fields below are your responsibility.

IDENTITY
  engine + commit SHA          vllm a556f3f   (not "vLLM 0.x")
  model + revision             meta-llama/Meta-Llama-3-8B-Instruct @ <hf sha>
  harness + commit SHA         which repo's serve.py, at what SHA

HARDWARE
  GPU / SKU / count            1x H100 SXM 80GB
  driver + CUDA + torch        e.g. 550.x / 12.4 / 2.x
  host CPU, RAM, NUMA pinning
  client host                  same box or separate? NIC speed?

SERVER CONFIGURATION
  full `vllm serve` / `python -m sglang.launch_server` command line
  every non-default flag, especially: --max-num-seqs, --max-num-batched-tokens,
  --enable-prefix-caching / --schedule-policy, --enforce-eager, quantization,
  attention backend, --gpu-memory-utilization
  attention backend selection: --attention-backend  (the old VLLM_ATTENTION_BACKEND
    env var no longer exists at a556f3f -- see 03-04)
  environment: VLLM_SERVER_DEV_MODE, TORCH_CUDA_ARCH_LIST

WORKLOAD
  dataset name + version, or the input/output length DISTRIBUTIONS (not means)
  prefix-sharing rate of the dataset            (see 10-01)
  number of distinct prompts, and the seed

LOAD
  arrival process              open loop, Poisson, lambda = 12 req/s
                               (or: closed loop, N = 64 -- and say which)
  burstiness parameter         vLLM only; 1.0 = Poisson
  --max-concurrency            if set, the reported latency EXCLUDES client queueing
  ramp-up strategy             if any

PROTOCOL
  warmup: how many requests, at what load, for how long
  cache flushed after warmup?  which endpoint, did it return success
  measured window: start and end, in requests AND seconds
  number of independent runs, and whether the server was restarted between them
  order of sweep points (randomised?)

RESULTS -- a curve, not a number
  for each offered load: throughput, TTFT p50/p90/p99, ITL p50/p99/max,
                         E2EL p99, goodput under the stated SLO
  the SLO used to define goodput
  run-to-run standard deviation for at least the headline metric
  GPU clock/temperature range over the run

When is a difference real

Let $\sigma$ be the run-to-run standard deviation of the metric and $\mu$ its mean, with $n$ independent runs per arm. The minimum difference detectable at $\alpha = 0.05$ two-sided with 80% power is

Read the CV = 5% column, which is what you get when the prefix cache is not flushed between runs or the GPU is thermally drifting: three runs give an approximate 11.4% planning MDE at the stated significance and power. That is not the observed significance of a particular experiment. Lower run variance and more independent runs both improve sensitivity; paired or blocked designs require the variance of their differences. Measure $\sigma$ first, then decide $n$: $n = (3.96\,\mathrm{CV}/\mathrm{MDE})^{2}$. To resolve 5% at CV = 5% needs $n = 16$; at CV = 2% it needs $n = 3$.

Percentiles need their own sample count. §1.2 works this through: p99 from 1,800 completed requests sits on the 18th-worst, which is noisy but usable; p999 from the same sample is an anecdote. Do not quote a p99 computed from fewer than ~1,000 completed requests, and never quote a p999 from fewer than ~10,000.

Comparing two engines

And report the curve. Two engines swap places along a load curve routinely: one wins at low load because its per-step overhead is lower, the other wins near saturation because its scheduler packs better. A single point picks a winner by picking a load.

Produce one defensible point, then one curve. Start the server in a mode that lets you flush:

# server: dev mode so /reset_prefix_cache exists, everything else pinned
VLLM_SERVER_DEV_MODE=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --max-num-seqs 256 --max-num-batched-tokens 8192 --enable-prefix-caching

# warm at the load you will measure, then flush, then measure
vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset-name random --random-input-len 2048 --random-output-len 256 \
  --request-rate 12 --num-prompts 1200 --num-warmups 0 --seed 1
curl -X POST localhost:8000/reset_prefix_cache   # expect {"success":true}

vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset-name random --random-input-len 2048 --random-output-len 256 \
  --request-rate 12 --num-prompts 2000 --burstiness 1.0 --seed 2 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 \
  --metadata gpu=H100-SXM driver=550 sha=a556f3f \
  --save-result --result-filename rate12-run1.json

Then sweep. vllm bench sweep restarts the server per configuration and resets caches between runs, which is the whole protocol above in one command:

VLLM_SERVER_DEV_MODE=1 vllm bench sweep \
  --serve-cmd "vllm serve meta-llama/Meta-Llama-3-8B-Instruct --max-num-seqs 256" \
  --bench-cmd "vllm bench serve --dataset-name random --random-input-len 2048 --random-output-len 256 --num-prompts 2000" \
  --bench-params bench_params.json \
  --num-runs 5 --output-dir sweep-results

with bench_params.json listing {"request-rate": [4, 6, 8, 10, 12, 13, 14, 16]}. Five runs per point gives you $\sigma$; the table in §8 turns $\sigma$ into what you are allowed to claim. Two things to verify before trusting the output: that the reset POSTs returned 200 (they will crash the sweep if not), and that client CPU stayed below saturation.

Three cheap diagnostics worth running once. (1) Run the same configuration twice back to back without flushing, and once with. The delta is your cache-carryover confounder, in your units. (2) Add --probe-request-rate 1 to a saturated run; it sends single-token probes that bypass --max-concurrency and reports their latency separately (vllm/benchmarks/serve.py:L1678-L1686) — a direct measurement of how much your main workload stalls an unrelated user. (3) Sweep concurrency across 8, 9, 12, 13, 16 and confirm the plateau predicted by §8.1. The lab 11-end-to-end-benchmark in Labs walks the full protocol.

• §10.1 for the workload distributions this chapter assumes you already measured, and the bias baked into each shipped dataset; §10.2 for what each harness computes and why the two cannot be mixed; §10.4 for why identical inputs give different tokens across runs; §10.5 for turning a defensible curve into cost per million tokens.
• §1.2 — the metric definitions, the M/M/1 and Kingman results this chapter builds on, Little's Law, and goodput. Do not re-derive them; cite them.
• Mor Harchol-Balter, Performance Modeling and Design of Computer Systems — chapters on open vs closed systems are the canonical treatment of why the two answer different questions, and where the relaxation-time result comes from.
• Schroeder, Wierman and Harchol-Balter, "Open versus Closed: A Cautionary Tale" (NSDI 2006) — the paper that put this distinction on the map for systems benchmarking; the conclusions transfer to LLM serving unchanged.
• vllm/benchmarks/sweep/ in the vLLM tree — param_sweep.py, server.py, plot_pareto.py. The Pareto plotting is the closest thing either project has to an opinion about how results should be presented.
• Gray, The Benchmark Handbook (1993), chapter 1 — relevance, portability, scalability, simplicity. Thirty years old and every criterion still fails on a typical LLM serving benchmark.