An end-to-end benchmark you can defend

One curve per engine, and one number off each curve.

The curve is goodput against offered load. For each offered rate $\lambda$ you get a point: requests per second that completed within your SLO. The number is the curve's maximum — the highest load at which the replica still keeps its promise. That is the capacity figure a capacity plan is made of, and it is not the same as the throughput number either harness prints by default.

Goodput is not a synonym for throughput with a footnote. It is defined per request, against SLOs you name, and vLLM's harness implements exactly that:

    parser.add_argument(
        "--goodput",
        nargs="+",
        required=False,
        help='Specify service level objectives for goodput as "KEY:VALUE" '
        "pairs, where the key is a metric name, and the value is in "
        'milliseconds. Multiple "KEY:VALUE" pairs can be provided, '
        "separated by spaces. Allowed request level metric names are "
        '"ttft", "tpot", "e2el". For more context on the definition of '
        "goodput, refer to DistServe paper: https://arxiv.org/pdf/2401.09670 "
        "and the blog: https://hao-ai-lab.github.io/blogs/distserve",
    )

    if goodput_config_dict:
        valid_metrics = []
        slo_values = []

        if "ttft" in goodput_config_dict:
            valid_metrics.append(ttfts)
            slo_values.append(
                goodput_config_dict["ttft"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )
        if "tpot" in goodput_config_dict:
            valid_metrics.append(all_tpots)
            slo_values.append(
                goodput_config_dict["tpot"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )
        if "e2el" in goodput_config_dict:
            valid_metrics.append(e2els)
            slo_values.append(
                goodput_config_dict["e2el"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )

        for req_metric in zip(*valid_metrics):
            is_good_req = all([s >= r for s, r in zip(slo_values, req_metric)])
            if is_good_req:

Read the predicate: all([s >= r for s, r in zip(slo_values, req_metric)]). A request counts only if it met every stated SLO, and the counter is divided by the same dur_s that throughput uses (vllm/benchmarks/serve.py:L730-L732). So goodput and throughput are directly comparable, and their ratio is the fraction of your traffic that was served acceptably — which is the number that decides whether a config ships.

Only three metric names are accepted, and the harness refuses anything else rather than silently ignoring it:

def check_goodput_args(args):
    # Check and parse goodput arguments
    goodput_config_dict = {}
    VALID_NAMES = ["ttft", "tpot", "e2el"]
    if args.goodput:
        goodput_config_dict = parse_goodput(args.goodput)
        for slo_name, slo_val in goodput_config_dict.items():
            if slo_name not in VALID_NAMES:
                raise ValueError(
                    f"Invalid metric name found, {slo_name}: {slo_val}. "
                    "The service level objective name should be one of "
                    f"{str(VALID_NAMES)}. "
                )
            if slo_val < 0:
                raise ValueError(
                    f"Invalid value found, {slo_name}: {slo_val}. "
                    "The service level objective value should be "
                    "non-negative."
                )
    return goodput_config_dict

§10.3 establishes that cross-harness comparison is invalid — the two clients do not compute the same ITL and do not count output tokens the same way, so their numbers cannot appear in one table. That leaves one choice: pick a client and point it at both servers. Which client is settled by the asymmetry above rather than by preference, and the choice costs one flag.

vLLM's harness dispatches on a backend name, and openai is the generic OpenAI-Completions path:

ASYNC_REQUEST_FUNCS: dict[str, RequestFunc] = {
    "vllm": async_request_openai_completions,
    "openai": async_request_openai_completions,
    "openai-chat": async_request_openai_chat_completions,
    "openai-audio": async_request_openai_audio,

and SGLang serves that route, with the same default path the harness asks for (--endpoint defaults to /v1/completions, vllm/benchmarks/serve.py:L1574-L1578):

@app.post("/v1/completions", dependencies=[Depends(validate_json_request)])
async def openai_v1_completions(request: CompletionRequest, raw_request: Request):
    """OpenAI-compatible text completion endpoint."""
    return await raw_request.app.state.openai_serving_completion.handle_request(
        request, raw_request
    )

The extra fields the payload carries — ignore_eos, repetition_penalty, stream_options — are all declared on SGLang's CompletionRequest (python/sglang/srt/entrypoints/openai/protocol.py:L340-L375), so nothing is dropped and nothing 422s. The readiness probe uses /v1/models, which SGLang also serves (python/sglang/srt/entrypoints/http_server.py:L1843).

The load must be identical, not merely equal in rate

Both arms must be offered the same requests at the same instants. vLLM's generator precomputes the whole arrival schedule before the run and seeds it globally (random.seed(args.seed) / np.random.seed(args.seed), vllm/benchmarks/serve.py:L1968-L1969), so the same --seed, --num-prompts and --request-rate produce a byte-identical workload for both engines. It then rescales the schedule so the achieved rate hits the target exactly:

            delay_ts[i] += delay_ts[i - 1]
        if ramp_up_strategy is None and delay_ts[-1] != 0:
            # When ramp_up_strategy is not set, we assume the request rate is fixed
            # and all requests should be sent in target_total_delay_s, the following
            # logic would re-scale delay time to ensure the final delay_ts
            # align with target_total_delay_s.
            #
            # NOTE: If we simply accumulate the random delta values
            # from the gamma distribution, their sum would have 1-2% gap
            # from target_total_delay_s. The purpose of the following logic is to
            # close the gap for stabilizing the throughput data
            # from different random seeds.
            target_total_delay_s = total_requests / request_rate
            normalize_factor = target_total_delay_s / delay_ts[-1]

That rescaling matters more than it looks. Without it the realised rate wanders 1–2% with the seed, which is the same order as the differences people publish.

The arrival process itself is a knob, and its default is the one you want:

    parser.add_argument(
        "--burstiness",
        type=float,
        default=1.0,
        help="Burstiness factor of the request generation. "
        "Only take effect when request_rate is not inf. "
        "Default value is 1, which follows Poisson process. "
        "Otherwise, the request intervals follow a gamma distribution. "
        "A lower burstiness value (0 < burstiness < 1) results in more "
        "bursty requests. A higher burstiness value (burstiness > 1) "
        "results in a more uniform arrival of requests.",
    )

--burstiness 1.0 is Poisson. Below 1 is burstier, above 1 is smoother, and if your production traffic is neither, measure its burstiness first (§10.1) rather than accepting Poisson because it is the default.

Six steps, in this order. Steps 2 and 3 conflict — warming the queue also warms the prefix cache — which is why the flush comes between them and why step 4 exists.

Sizing it, before you burn an afternoon

$ python3 run.py plan --mu 25 --rho 0.9 --slo-ttft 500 --slo-tpot 50
$ python3 run.py plan --mu 25 --cv 0.03 --mde 0.04     # how many runs per arm?
$ python3 run.py --help

The two questions plan answers are the two that decide whether the experiment can work at all. How long might warmup take? — an M/M/1 planning estimate that must be checked empirically. How small a difference can I defend? — from

at $\alpha = 0.05$ two-sided with 80% power (§10.3). With three runs and a 5% coefficient of variation you cannot defend anything smaller than 11%, and most published deltas are smaller than that. Measure $\sigma$ first, then choose $n$; if $n$ comes out large, drive the CV down with the flush and the randomised order instead of buying more runs.

The commands

# vLLM. Dev mode so /reset_prefix_cache exists at all.
VLLM_SERVER_DEV_MODE=1 vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
    --port 8000 --max-num-seqs 256 --max-num-batched-tokens 8192 --enable-prefix-caching

# SGLang. Match the semantics, not the spelling. --schedule-policy is stated
# explicitly because its default is fcfs, not the cache-aware policy people
# assume they are measuring (10.3).
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
    --port 30000 --max-running-requests 256 --chunked-prefill-size 8192 \
    --schedule-policy fcfs

URL=http://127.0.0.1:8000        # or :30000 for SGLang, with --backend openai

# 2. warm AT the load you will measure
vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct --base-url $URL \
    --dataset-name random --random-input-len 2048 --random-output-len 256 \
    --request-rate 22 --burstiness 1.0 --num-prompts 1200 --seed 1

# 3. flush, and CHECK THE BODY, not just that curl exited 0
curl -sS -X POST $URL/reset_prefix_cache          # vLLM  -> {"success":true}
curl -sS -X POST $URL/flush_cache                 # SGLang -> 200, or 400 if busy

# 4+5. measure. Goodput needs an SLO stated up front.
vllm bench serve --model meta-llama/Meta-Llama-3-8B-Instruct --base-url $URL \
    --dataset-name random --random-input-len 2048 --random-output-len 256 \
    --request-rate 22 --burstiness 1.0 --num-prompts 2000 --seed 2 \
    --goodput ttft:500 tpot:50 \
    --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 \
    --metadata gpu=H100-SXM engine=vllm sha=a556f3f \
    --save-result --result-filename vllm-rate22-run1.json

Then sweep $\lambda$ and repeat, three times per point, in randomised order. run.py sweep generates exactly those invocations — use --dry-run and read them before you let it run anything.

    parser.add_argument(
        "--num-warmups",
        type=int,
        default=0,
        help="Number of warmup requests.",
    )

    parser.add_argument(
        "--warmup-requests",
        type=int,
        default=1,
        help="Number of warmup requests to run before the benchmark",
    )

SGLang's harness at least knows the warmup polluted the cache and offers to fix it — --flush-cache, off by default (python/sglang/benchmark/serving.py:L2587-L2591). vLLM's harness has no flush at all, which is why step 3 is a separate curl.

Goodput will collapse before throughput does

That is the whole reason to measure it. Throughput rises with offered load until capacity $\mu$ and is then flat forever; goodput rises, peaks before $\mu$, and falls, because past the knee the queue is adding latency to requests that still complete. A throughput curve tells you the machine is fine right up to the point where every user has left. Plot both on one axis and the gap between the two peaks is your operating headroom.

The two engines will swap places along the curve

Routinely. One wins at low load because its per-step overhead is lower; the other wins near saturation because its scheduler packs better. A single load point picks a winner by picking a load, which is why the deliverable is a curve and why a comparison that reports one number is not wrong so much as unfalsifiable.

Things that make the comparison invalid, in the order they bite

Two diagnostics worth running once

First, run the same configuration twice back to back without flushing, then once with. The delta is your cache-carryover confounder measured in your own units, and it is the single most useful number in this lab because it bounds how much of any difference you later measure could be an artefact.

Second, add --probe-request-rate 1 to a saturated run. It sends single-token probes that bypass the concurrency limit and reports their latency separately (vllm/benchmarks/serve.py:L1678-L1687) — a direct measurement of how much your main workload stalls an unrelated user, which is the thing an SLO is actually protecting.

The deliverable is a table with one row per offered load per engine, and a provenance block above it. Below is the shape, with every cell left as a dash. It is a template, not results — nothing in this book was measured. The good/completed fraction uses matched completed-request counts, weighted by run duration; it excludes failed arrivals. Report failures and good/offered separately when evaluating offered-load SLO attainment. run.py report emits this table filled in from your result JSON.

Above it, the provenance block from §10.3's reporting template, in full. Neither harness records all of it: vLLM's result JSON stores the client-side arguments plus a free-form --metadata KEY=VALUE and nothing about the server; SGLang's embeds the whole /server_info payload but is not the harness you are running. The fields are your responsibility either way, and a result table without them is a screenshot.