TTFT, TPOT, ITL, E2E, goodput

A dashboard says mean TTFT = 180 ms. Support says the product feels broken. Both are true.

The mean is being computed over a distribution whose right tail is generated by a completely different mechanism than its body. In the body, TTFT is prefill compute: a 1,800-token prompt through Llama-3-70B, a fixed number of FLOPs, a stable number. In the tail, TTFT is queueing — the request sat in the waiting queue while other prefills ran ahead of it. Those two mechanisms have different scaling laws, and averaging across them produces a number that describes neither.

Worse, the four metrics people quote are not four measurements of one thing. They are four different clocks with four different start and stop points, and at least three of them have more than one definition in common use. Consider a single concrete ambiguity that will recur through this chapter: is the first token included in TPOT?

            tpot = 0.0
            if output_len > 1:
                latency_minus_ttft = outputs[i].latency - outputs[i].ttft
                tpot = latency_minus_ttft / (output_len - 1)
                tpots.append(tpot)
            # Note: if output_len <= 1, we regard tpot as 0 for goodput
            all_tpots.append(tpot)
            itls += outputs[i].itl
            ttfts.append(outputs[i].ttft)
            e2els.append(outputs[i].latency)

vLLM's harness excludes it: it divides by output_len - 1 after subtracting TTFT. The other definition in the wild — E2E / output_len — folds a 400 ms prefill into a 200-token generation and inflates the reported per-token cost by 2 ms. On a 20 ms baseline that is a 10% swing produced entirely by a choice of denominator. If you cannot say which one a number came from, the number is not comparable to anything.

Draw one request as a line in time and mark seven instants on it: the HTTP request arrives; it is tokenized and enters the waiting queue; the scheduler admits it; prefill finishes and the first token is emitted; tokens 2…N are emitted with gaps between them; the last token is emitted; the stream is closed. Every metric in this chapter is a span between two of those instants. That is the whole mental model — the difficulty is entirely in which two, and in whose clock.

Two structural facts follow immediately from the picture, and they are the reason the rest of the chapter exists. First, TTFT contains queueing, so it is a function of load, not just of prompt length. Second, TPOT is one number per request while ITL is many, so a single 1.5-second stall inside a 500-token generation shows up as a 75× outlier in ITL and as a 15% bump in TPOT. Choosing between them is choosing what you want to be able to see.

The definition table

Let a request arrive at $t_a$, be admitted by the scheduler at $t_s$, emit its $k$-th output token at $t_k$ for $k = 1 \dots N$, and have its stream closed at $t_c$. Then:

Note that $\mathrm{TPOT}$ is exactly the arithmetic mean of that request's $\mathrm{ITL}$ samples when each streamed chunk carries exactly one token, because the $N-1$ gaps telescope to $t_N - t_1$. Speculative decoding — one forward pass emitting several accepted tokens at once, §6.2 — along with chunk bundling and any proxy that coalesces SSE frames, breaks that identity, and both engines have code that exists solely to repair it.

Throughput needs its own disambiguation because there are four of them and they are not interchangeable: request throughput (req/s), output token throughput (generated tok/s), total token throughput (input + output tok/s), and input token throughput. Prefill dominates the input number and decode dominates the output number, so a system tuned for one can look terrible on the other. vLLM computes all but the input-only variant in one place:

    metrics = BenchmarkMetrics(
        completed=completed,
        failed=len(failed_outputs),
        total_input=total_input,
        total_output=sum(actual_output_lens),
        request_throughput=completed / dur_s,
        request_goodput=good_completed / dur_s,
        output_throughput=sum(actual_output_lens) / dur_s,
        total_token_throughput=(total_input + sum(actual_output_lens)) / dur_s,

Why TTFT p99 is a queueing measurement, not a compute measurement

Model one replica's prefill capacity as a single server with mean service time $S$ and utilisation $\rho = \lambda S$, where $\lambda$ is the arrival rate. For an M/M/1 queue the waiting time before service has mean $\mathbb{E}[W] = S\rho/(1-\rho)$ and tail $\Pr[W > t] = \rho\, e^{-(1-\rho)t/S}$, so the 99th percentile of waiting is

Take S=100 ms as an illustrative mean service time. In M/M/1, exact total residence p99 is S*ln(100)/(1-rho); for rho=(0.5,0.7,0.8,0.9,0.95) this gives approximately (921,1535,2303,4605,9210) ms. The following historical table instead shows mean service plus waiting p99 as a labeled approximation. Real serving is not M/M/1: batching, correlated arrivals, and variable prompt work require measured service and arrival distributions. Kingman's formula is a mean-wait approximation, not a universal tail bound.

Read the last column again. At any load you would actually run a fleet at, 90–99% of p99 TTFT is time the request spent doing nothing. But do not read that as “kernels do not matter”, because $\rho = \lambda S$ — the queueing term is itself a function of service time, and the $1/(1-\rho)$ makes the coupling violent in both directions. At $\lambda = 9$ req/s, a prefill kernel 20% faster takes $S$ from 100 ms to 80 ms and therefore $\rho$ from 0.90 to 0.72, and p99 TTFT from 4,600 ms to $80\ln(72)/0.28 + 80 \approx 1{,}300$ ms: a 20% kernel win bought a 3.5× latency win. The same lever run backwards is what actually hurts — a 20% regression puts $\rho$ at 1.08 and the queue never drains at all. Adding a replica, which drops $\rho$ from 0.9 to 0.45, moves p99 TTFT to roughly 800 ms by exactly the same nonlinearity. The lesson is not that compute is irrelevant; it is that near the knee, every quantity is leveraged by $1/(1-\rho)$ and none of it is visible in the mean — which at $\rho = 0.9$ reads 1,000 ms while hiding a 4.6-second tail. All derived from the two formulas above.

Why ITL p99 is a scheduler measurement

Decode step time is remarkably stable: the same matrix shapes every step, no data-dependent branching. Left alone, a request's ITL samples would be a tight cluster. They are not tight in production, and the reason is interference — most commonly a long prefill admitted into the middle of an ongoing generation. Every decoding request in the batch waits for that prefill's forward pass.

Work the arithmetic. Illustrative assumption: baseline ITL 20 ms, a 32k-token prefill admitted unchunked costs 1,500 ms, a generation of $N = 500$ tokens — so 499 inter-token gaps — eats exactly one such stall.

Derived: $\mathrm{TPOT} = (498 \times 20 + 1500)/499 = 11{,}460/499 = 22.97$ ms. So a 75× stall that a user perceives as the text freezing appears as +15% on TPOT and as nothing at all on ITL p99, because one bad gap in 500 lives at the 99.8th percentile. The stall is only recoverable from max(ITL), or from p99.9, or from noticing that TPOT's median moved. This is the single most useful thing to know about these metrics: ITL p99 and TPOT p99 fail in opposite directions. A rare-but-severe stall hides in ITL p99 and shows in TPOT p50; a universal mild slowdown shows in both. Track ITL max explicitly. §1.5 is the chapter that removes this failure mode by slicing prefills.

Percentile hygiene: which one, over what window, aggregated how

Three rules, each of which is violated constantly.

Use p99, not p95, and not p999 — unless you can defend the choice. The argument is about sample count, not about taste. A percentile estimated from $n$ samples has a standard error roughly $\sqrt{p(1-p)/n}$ in probability space; converting that to a latency error depends on how steep the distribution is at that point, and the tail of a queueing distribution is very steep. At 6 req/s over a 5-minute window you have $n = 1800$ requests, so p99 sits on the 18th-worst request — noisy but usable — and p999 sits between the 1st and 2nd worst, which is not an estimate, it is an anecdote. Widen the window to get p999 and you lose the ability to detect a two-minute incident. p95 is stable but, per the queueing table above, sits in a region where the curve is still flat and therefore fails to move until the system is already in trouble. p99 is the compromise, and the compromise is a function of your traffic rate; at 60 req/s, p999 becomes defensible.

Never average percentiles. Two replicas reporting p99 TTFT of 800 ms and 1,200 ms do not give the fleet a p99 of 1,000 ms — the fleet p99 could be between 800 ms and 1,200 ms for exact component quantiles using the same inverse-CDF convention (histogram approximation can add estimation error) depending on the request split. This is exactly why both engines export histograms rather than pre-computed quantiles: histogram_quantile(0.99, sum by (le) (rate(vllm:time_to_first_token_seconds_bucket[5m]))) sums the bucket counters across replicas first and takes the percentile of the pooled distribution. Summing then quantiling is correct; quantiling then averaging is not. Any dashboard that stores a per-replica p99 as a gauge and averages it has thrown the fleet number away permanently.

A one-hour window can respond more slowly and dilute a short incident, but a 10-minute brownout comprising 17% of requests can certainly move p99. Detection time depends on traffic, the incident distribution, histogram buckets, evaluation cadence, and alert thresholds; window length alone does not determine it.

Goodput, throughput, and Little's Law

Throughput and latency are not independent quantities you can tune separately; they are tied by Little's Law. For a system in steady state with mean concurrency $\bar{L}$, arrival rate $\lambda$, and mean end-to-end latency $\bar{W}$:

SGLang's harness computes exactly this and reports it as a first-class metric, using the sum of per-request E2E latencies divided by the wall-clock duration:

        concurrency=np.sum(e2e_latencies) / dur_s,

Little's Law uses mean residence time, not an SLO quantile or cap. If mean E2EL is 12 s at 240 req/s in steady state, mean system concurrency is 2880. Only admitted requests occupying GPU state contribute to resident KV; queued requests can have no allocated KV. Split the system into waiting and active stages before converting concurrency into cache bytes. An upper bound on latency does not imply that the mean equals that bound.

Goodput counts requests satisfying the chosen per-request predicate. For M/M/1 with mean service 0.1 s and arrival rate 9.5/s, total residence time is exponential with rate 0.5/s, so P(T<=0.9)=1-exp(-0.45)=0.3624 and goodput is about 3.44 req/s. This is not 96%. If a different measured workload had a 96% pass rate at 9.5 req/s, its goodput would be 9.12 req/s. A fleet-level p99 SLO verdict is a separate binary compliance decision.

The trade surface, stated plainly

Writing an SLO a capacity plan can be built from

An SLO with no workload attached is not an SLO. "p99 TTFT under 1 second" is satisfiable by any GPU ever made if the prompts are 10 tokens and the arrival rate is 0.1 req/s, and satisfiable by none of them at 100k-token prompts and 50 req/s. A well-formed SLO names five things: metric, percentile, window, workload distribution, and offered load.

The goodput restatement is the operationally useful one, because it is a single scalar you can divide total demand by. If peak demand is 240 req/s and one replica delivers 5.94 req/s of goodput at that workload, you need 41 replicas. Sizing on raw throughput instead — a replica with $S = 100$ ms tops out at $1/S = 10$ req/s — would have told you 24 replicas, and every one of those 24 would be sitting at $\rho = 1.0$, off the right-hand edge of the queueing table, where nothing meets a 900 ms TTFT bound and the waiting queue grows without limit. The 17-replica gap between 24 and 41 is the price of the SLO, and it is invisible to any throughput number. The cost-per-million-tokens arithmetic that turns replica counts into money is in §10.5.

Both harnesses have moved. Check where you are pointing.

At the pinned SHAs, both projects' canonical serving benchmarks have been relocated behind deprecation shims. vLLM's is a hard error:

if __name__ == "__main__":
    print("""DEPRECATED: This script has been moved to the vLLM CLI.

Please use the following command instead:
    vllm bench serve

For help with the new command, run:
    vllm bench serve --help

Alternatively, you can run the new command directly with:
    python -m vllm.entrypoints.cli.main bench serve --help
""")
    sys.exit(1)

SGLang's still works but warns. The live implementations are vllm/benchmarks/serve.py (2,368 lines) and python/sglang/benchmark/serving.py (2,752 lines); python/sglang/bench_serving.py is now a 22-line re-export that emits a FutureWarning. Any blog post or CI job citing the old paths is describing code that no longer runs.

Where the clocks are actually read

In vLLM's client harness the request clock starts immediately before the HTTP POST, and — critically — output.latency stops at the last content timestamp, not at stream close:

    st = time.perf_counter()
    output.start_time = st
    most_recent_timestamp = st
# ...
                                text = choices[0].get("text")
                                timestamp = time.perf_counter()
                                # First token
                                if not first_chunk_received:
                                    first_chunk_received = True
                                    ttft = time.perf_counter() - st
                                    output.ttft = ttft

                                # Decoding phase
                                else:
                                    output.itl.append(timestamp - most_recent_timestamp)

                                most_recent_timestamp = timestamp
# ...
                output.generated_text = generated_text
                output.latency = most_recent_timestamp - st

SGLang's OpenAI-compatible backend recomputes latency at the top of the chunk loop, before the [DONE] check, and assigns that value after the loop ends:

                        chunk = remove_prefix(chunk_bytes.decode("utf-8"), "data: ")
                        latency = time.perf_counter() - st
                        if chunk == "[DONE]":
                            pass
                        else:
# ...
                                if ttft == 0.0:
                                    ttft = time.perf_counter() - st
                                    output.ttft = ttft
# ...
                    output.generated_text = generated_text
                    output.success = True
                    output.latency = latency

The ITL definition difference that actually matters

vLLM appends one ITL sample per streamed chunk. SGLang's native backend appends one per token, dividing the observed chunk gap by the number of new tokens the chunk carried:

                                if ttft == 0.0:
                                    ttft = time.perf_counter() - st
                                    output.ttft = ttft

                                # Decoding phase
                                else:
                                    num_new_tokens = output_len - last_output_len
                                    if num_new_tokens == 0:
                                        continue
                                    chunk_gap = timestamp - most_recent_timestamp
                                    adjust_itl = chunk_gap / num_new_tokens
                                    output.itl.extend([adjust_itl] * num_new_tokens)

The consequence is sharp. Under speculative decoding, where one forward pass can emit 3–4 accepted tokens in one chunk, vLLM's ITL distribution is a distribution of chunk gaps — it will look 3–4× worse than SGLang's per-token amortised distribution on the same engine behaviour. SGLang applies the same correction a second time in calculate_metrics, re-tokenizing each chunk to get an exact token count when spec-decode is active (python/sglang/benchmark/serving.py:L1136-L1147). SGLang also reports a statistic vLLM does not: max_itl_ms=np.max(itls or 0) * 1000 at python/sglang/benchmark/serving.py:L1261 — precisely the statistic the stall arithmetic above said you need. And it reports mean concurrency via Little's Law, concurrency=np.sum(e2e_latencies) / dur_s at python/sglang/benchmark/serving.py:L1268.

Goodput exists in one harness only

    if goodput_config_dict:
        valid_metrics = []
        slo_values = []

        if "ttft" in goodput_config_dict:
            valid_metrics.append(ttfts)
            slo_values.append(
                goodput_config_dict["ttft"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )
        if "tpot" in goodput_config_dict:
            valid_metrics.append(all_tpots)
            slo_values.append(
                goodput_config_dict["tpot"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )
        if "e2el" in goodput_config_dict:
            valid_metrics.append(e2els)
            slo_values.append(
                goodput_config_dict["e2el"] / MILLISECONDS_TO_SECONDS_CONVERSION
            )

        for req_metric in zip(*valid_metrics):
            is_good_req = all([s >= r for s, r in zip(slo_values, req_metric)])
            if is_good_req:
                good_completed += 1

Three observations. The predicate is a conjunction over per-request scalars, so a request must satisfy every declared SLO to count. The allowed metric names are exactly ["ttft", "tpot", "e2el"] (vllm/benchmarks/serve.py:L1432-L1445) — ITL is deliberately absent, because goodput is defined per request and ITL is per gap; you would have to choose an aggregation, and the harness declines to choose for you. And requests with output_len <= 1 are given tpot = 0.0 via all_tpots, which means they always pass a TPOT SLO. Grepping SGLang's harness for goodput at this SHA returns nothing: SGLang's serving benchmark has no goodput support. If you need goodput on SGLang you compute it from the per-request dump yourself.

The Prometheus surface, and why bucket edges decide what you can see

Both engines export latency histograms. Because histogram_quantile() interpolates linearly within a bucket, a percentile is only as precise as the bucket that contains it. That makes the hard-coded bucket edges a load-bearing design decision, and the two projects made opposite choices.

        histogram_inter_token_latency = self._histogram_cls(
            name="vllm:inter_token_latency_seconds",
            documentation="Histogram of inter-token latency in seconds.",
            buckets=[
                0.01,
                0.025,
                0.05,
                0.075,
                0.1,
# ... 0.15, 0.2, 0.3, 0.4, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0, 20.0, 40.0, 80.0
            ],
            labelnames=labelnames,
        )

        if bucket_inter_token_latency is None:
            bucket_inter_token_latency = [
                0.002,
                0.004,
                0.006,
                0.008,
                0.010,
                0.015,
                0.020,
                0.025,
                0.030,
                0.035,
                0.040,

vLLM additionally exports the phase decomposition drawn under the axis in Figure 1 — vllm:request_queue_time_seconds, vllm:request_prefill_time_seconds, vllm:request_decode_time_seconds, vllm:request_inference_time_seconds and vllm:e2e_request_latency_seconds, all sharing one bucket list whose lowest edge is 0.3 s (vllm/v1/metrics/loggers.py:L889-L926). It also exports a per-request TPOT histogram, vllm:request_time_per_output_token_seconds (vllm/v1/metrics/loggers.py:L859-L886), which SGLang has no equivalent of — SGLang exports sglang:time_to_first_token_seconds, sglang:inter_token_latency_seconds, sglang:e2e_request_latency_seconds and sglang:queue_time_seconds only. If you want TPOT percentiles from SGLang you must derive them.

Two entries deserve comment. SGLang's sglang:queue_time_seconds opens with a 0.000 edge and reaches 3,000 s across 40 buckets (python/sglang/srt/observability/metrics_collector.py:L686-L732) — a deliberately enormous range, because queue time is the one quantity that can grow without bound under overload and you want the histogram to keep resolving instead of pinning at +Inf. And SGLang's per-stage histogram uses generated exponential edges, exponential_buckets(start=0.001, width=1.62, length=30) at python/sglang/srt/observability/metrics_collector.py:L733-L739, which is the principled choice for a quantity spanning many orders of magnitude: constant relative resolution everywhere, so histogram_quantile has bounded percentage error at every latency instead of excellent error in one decade and useless error in the next.

Arrival processes differ too, which matters because $\rho$ in the queueing argument is an arrival-rate concept. vLLM samples inter-arrival gaps from a Gamma distribution with a tunable burstiness shape parameter that reduces to Poisson at 1.0 (vllm/benchmarks/serve.py:L455-L467); SGLang samples exponential only (python/sglang/benchmark/serving.py:L1090). Burstiness below 1 raises $C_a^2$ and therefore raises queueing at the same mean rate. Open- versus closed-loop methodology is §10.3's subject.

Follow one streaming request through vLLM's server-side instrumentation. The clock starts in the frontend, not the engine core.

1. OutputProcessor.add_request constructs a RequestState carrying RequestStateStats(arrival_time=arrival_time) — vllm/v1/engine/output_processor.py:L183, fed from request.arrival_time at vllm/v1/engine/output_processor.py:L276. That is $t_a$.
2. Each engine-core step produces an IterationStats, whose iteration_timestamp = time.time() is stamped at construction (vllm/v1/metrics/stats.py:L429) and reused as "now" for the whole batch via _time_since:
   vllm/v1/metrics/stats.py:L449-L451vLLM

       def _time_since(self, start: float) -> float:
           """Calculate an interval relative to this iteration's timestamp."""
           return self.iteration_timestamp - start

3. IterationStats.update_from_output fires TTFT on the first output batch for the request, and ITL on every later one — but note that the two use different clocks: TTFT against the frontend's iteration_timestamp, ITL against the engine core's own timestamp.
   vllm/v1/metrics/stats.py:L465-L502, L494-L502vLLM

           if is_prefilling:
               if output.prefill_stats is not None:
                   self.prompt_token_stats.update_from_output(output.prefill_stats)
   
               first_token_latency = self._time_since(req_stats.arrival_time)
               self.time_to_first_tokens_iter.append(first_token_latency)
   # ...
           # Process the batch-level "new tokens" engine core event
           if is_prefilling:
               req_stats.first_token_ts = engine_core_timestamp
           else:
               itl = engine_core_timestamp - req_stats.last_token_ts
               self.inter_token_latencies_iter.append(itl)
   
           req_stats.last_token_ts = engine_core_timestamp

   So vllm:time_to_first_token_seconds includes tokenization, IPC to the engine core, and the return hop; vllm:inter_token_latency_seconds includes none of that. They are not on the same ruler, which is exactly right for their purposes but means you cannot add them.
4. update_from_events records queued_ts on the QUEUED event and scheduled_ts on the first SCHEDULED event, explicitly ignoring re-scheduling after preemption (vllm/v1/metrics/stats.py:L518-L522).
5. On finish, update_from_finished_request computes the whole decomposition, with the comments spelling out that preemptions land inside whichever phase they occurred in:
   vllm/v1/metrics/stats.py:L537-L560vLLM

           e2e_latency = self._time_since(req_stats.arrival_time)
   
           # Queued interval is from first QUEUED event to first SCHEDULED
           queued_time = req_stats.scheduled_ts - req_stats.queued_ts
   
           # Prefill interval is from first SCHEDULED to first NEW_TOKEN
           # Any preemptions during prefill is included in the interval
           prefill_time = req_stats.first_token_ts - req_stats.scheduled_ts
   
           # Decode interval is from first NEW_TOKEN to last NEW_TOKEN
           # Any preemptions during decode are included
           decode_time = req_stats.last_token_ts - req_stats.first_token_ts
   
           # Inference interval is from first SCHEDULED to last NEW_TOKEN
           # Any preemptions during prefill or decode are included
           inference_time = req_stats.last_token_ts - req_stats.scheduled_ts
   
           # Do not count the token generated by the prefill phase
           mean_time_per_output_token = (
               decode_time / (req_stats.num_generation_tokens - 1)
               if req_stats.num_generation_tokens - 1 > 0
               else 0
           )

6. PrometheusStatLogger.record drains the per-iteration lists into the histograms — vllm/v1/metrics/loggers.py:L1216-L1219 for TTFT and ITL, vllm/v1/metrics/loggers.py:L1253-L1255 for per-request TPOT.

SGLang's equivalent path runs entirely in TokenizerManager. set_created_time is called when the request state is registered (python/sglang/srt/managers/tokenizer_manager.py:L3369), set_first_token_time on the first output batch (python/sglang/srt/managers/tokenizer_manager.py:L2402-L2403), and collect_metrics observes TTFT once then ITL thereafter (python/sglang/srt/managers/tokenizer_manager.py:L2823-L2841). The definitions are one-liners:

    def get_interval(self):
        return time.perf_counter() - self.last_time

    def get_first_token_latency(self):
        return self.first_token_time - self.created_time

    def get_e2e_latency(self):
        return self.finished_time - self.created_time

    def get_decode_latency(self):
        return self.finished_time - self.first_token_time

And the per-token amortisation seen in the harness is mirrored in the exporter, which increments the bucket by num_new_tokens while adding the un-divided interval to _sum — so rate(sum)/rate(count) still yields a correct mean per-token ITL:

    def observe_inter_token_latency(
        self, labels: Dict[str, str], internval: float, num_new_tokens: int
    ):
        adjusted_interval = internval / num_new_tokens

        # A faster version of the Histogram::observe which observes multiple values at the same time.
        # reference: https://github.com/prometheus/client_python/blob/v0.21.1/prometheus_client/metrics.py#L639
        his = self.histogram_inter_token_latency.labels(**labels)
        his._sum.inc(internval)

        for i, bound in enumerate(his._upper_bounds):
            if adjusted_interval <= bound:
                his._buckets[i].inc(num_new_tokens)
                break

Both engines therefore measure server-side TTFT from frontend arrival, not from scheduler admission. That is the honest choice — it is what the user waits — and it is why vllm:request_queue_time_seconds exists as a separate series: it is the term you subtract to isolate compute.

Run the same workload through both harnesses and diff the definitions rather than the numbers. Start a server, then:

vllm bench serve --backend vllm --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset-name random --random-input-len 1800 --random-output-len 220 \
  --num-prompts 500 --request-rate 6.0 --burstiness 1.0 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 \
  --goodput ttft:900 e2el:14100 \
  --save-result --result-filename run-vllm.json

python -m sglang.benchmark.serving --backend sglang-oai \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset-name random --random-input-len 1800 --random-output-len 220 \
  --num-prompts 500 --request-rate 6.0 --output-file run-sgl.jsonl

Three things to do with the output. (1) Compare mean_itl_ms against mean_tpot_ms within a single run — a difference can arise even with one token per chunk because pooled ITL weights requests by their number of gaps, whereas mean TPOT weights requests equally. Inspect lengths and chunk token counts before diagnosing buffering. (2) Scrape /metrics and run histogram_quantile(0.99, rate(vllm:inter_token_latency_seconds_bucket[5m])); if the answer is suspiciously round and under 10 ms, you have hit the bucket floor described above. (3) Re-run with --burstiness 0.3 at the same --request-rate and watch TTFT p99 move while throughput does not — that is $C_a^2$ in Kingman's formula, visible.

• DistServe, Zhong et al., arXiv:2401.09670 — the goodput framing this book uses, and the paper vLLM's own --goodput help text points at (vllm/benchmarks/serve.py:L1786-L1788). The companion post is at hao-ai-lab.github.io/blogs/distserve.
• Orca: A Distributed Serving System for Transformer-Based Generative Models, Yu et al., OSDI '22 — the origin of iteration-level scheduling, which is what makes ITL a scheduler-dependent quantity in the first place. Covered in §1.3.
• vLLM production metrics documentation, docs.vllm.ai/en/latest/usage/metrics.html — the exported series list; cross-check it against vllm/v1/metrics/loggers.py, which is authoritative and moves faster than the docs.
• vLLM RFC: V1 metrics, vllm-project/vllm#10582 — the design discussion behind the V1 stat-logger architecture, including why TTFT and ITL end up on different clocks.
• Kingman's formula — for the general-arrival, general-service correction $\mathbb{E}[W] \approx \frac{\rho}{1-\rho}\cdot\frac{C_a^2+C_s^2}{2}\cdot S$. Lognormal prompt lengths push $C_s^2$ well above 1, so the M/M/1 table in this chapter understates real queueing.
• Continue to §1.3 for why batching moves these numbers, §1.5 for the fix to ITL stalls, §10.3 for warmup, saturation, and open- versus closed-loop, and §9.5 for turning these series into alerts.