Observability, failure modes, recovery

Three things are different about operating an inference server, and all three break habits carried over from stateless HTTP services.

The unit of work is not a request, it is a step. A request occupies GPU memory for its whole lifetime and is re-batched every iteration (§1.3). So a metric like "requests per second" tells you almost nothing; the quantities that predict failure are occupancy quantities — how many sequences are resident, how much KV the pool holds, how deep the waiting queue is.

The healthy latency scale is smaller than the instrumentation's resolution. The book's decode floor for Llama-3-8B on an H100 SXM is 4.48 ms per token at batch 1 (15.0 GB of bf16 weights over 3.35 TB/s of HBM — the full checkpoint is 16 GB, but §0.4 excludes the gathered embedding table, which a decode step does not stream; derived, see §1.1). vLLM's inter-token-latency histogram has its lowest bucket edge at 10 ms. Every healthy sample lands in bucket one.

There is no such thing as losing one worker. A tensor-parallel rank that dies leaves its peers blocked forever inside an NCCL collective. There is no quorum, no re-shard, no degraded mode. The engine dies, and restart is the recovery mechanism — a fact §5.5 established from the barrier structure and which we verify here from the code that claims otherwise.

Think of the serving system as three concentric shells, each with its own failure signal and its own recovery story. The outer shell is the HTTP process: it can be up while everything inside it is dead. The middle shell is the engine core — the scheduler, the KV cache manager, the block table. The inner shell is the set of GPU worker processes bound in a collective. A fault in the inner shell propagates outward and kills everything; a fault in the outer shell kills only itself.

Which is to say: the blast radius of a failure is determined by where it happens in the process topology, not by how bad it looks in the log.

A Prometheus classic histogram is a set of cumulative counters, one per bucket upper bound $b_i$, plus a _sum and a _count. An observation $x$ increments every bucket with $b_i \ge x$. All information about where inside a bucket the observations fell is discarded at observation time and cannot be recovered downstream.

histogram_quantile(q, ...) finds the bucket containing the $q$-th observation and then interpolates linearly between that bucket's lower and upper bound. For the lowest bucket the lower bound is taken to be 0. So if a fraction $f = 1$ of the mass lands in the first bucket with upper bound $b_0$, the reported quantile is exactly

— a function of $q$ alone. It contains zero bits about the data.

Now the numbers. vLLM's ITL buckets, read at a556f3f:

        histogram_inter_token_latency = self._histogram_cls(
            name="vllm:inter_token_latency_seconds",
            documentation="Histogram of inter-token latency in seconds.",
            buckets=[
                0.01,
                0.025,
                0.05,
                0.075,
                0.1,
# ... 0.15 through 40.0 elided
                80.0,
            ],
            labelnames=labelnames,
        )

$b_0 = 10$ ms. A healthy Llama-3-8B on one H100 decodes at 4.48 ms/token at batch 1 (derived), and batching raises throughput without raising per-token latency much until the batch becomes compute-bound at the ridge point $I^* = 295$ — call it 5 ms of realised ITL with sampling and Python overhead. Then $f = 1$ and the dashboard reports:

The p99 line is flat at 9.9 ms whether your true ITL is 1 ms or 9 ms. It moves only when real ITL crosses 10 ms — at which point it does not drift, it jumps, because the mass is now split across two buckets whose edges are 10 and 25 ms. You get a metric that is simultaneously blind in the healthy regime and coarse in the degraded one.

SGLang's defaults start an order of magnitude lower:

        if bucket_inter_token_latency is None:
            bucket_inter_token_latency = [
                0.002,
                0.004,
                0.006,
                0.008,
                0.010,
                0.015,
                0.020,
                0.025,

Neither engine is simply right. The mirror image holds for TTFT: vLLM's lowest TTFT edge is 0.001 s (loggers.py:L796-L804), while SGLang's is 0.1 s (metrics_collector.py:L1623-L1644). An 8B model prefilling a 300-token prompt returns its first token in a few tens of milliseconds — so on SGLang every healthy TTFT lands in bucket one, and the exact same blindness argument applies with the engines swapped. §1.2 catalogues the bucket edges for both engines; the operational consequence is this section's.

Both engines register their Prometheus series in one place, so the honest way to learn what exists is to read that file rather than a docs page. In vLLM it is vllm/v1/metrics/loggers.py (1402 lines at this SHA); in SGLang it is python/sglang/srt/observability/metrics_collector.py (2430 lines).

Two differences are worth the paragraph. First, SGLang exports absolute token counts for the KV pool alongside the ratio, and separates kv_used_tokens from kv_evictable_tokens — the latter being radix-cached prefixes that are reclaimable under pressure. vLLM exports only kv_cache_usage_perc, so "90% used" does not tell you how much of that 90% is live versus cached. Second, SGLang exports its startup capacity constants as gauges, which means a dashboard can compute headroom in tokens without you hard-coding the pool size. vLLM logs the equivalent ("Available KV cache memory: %s GiB", vllm/v1/worker/gpu_worker.py:L577-L580) and you have to scrape it out of the log.

The periodic INFO log is the other primary surface, and on vLLM it is deliberately terse:

    def log(self):
        self._update_stats()
        self.aggregate_scheduler_stats()
        # Avoid log noise on an idle production system
        log_fn = logger.debug if self.engine_is_idle else logger.info
        # Format and print output.
        log_parts = [
            "Avg prompt throughput: %.1f tokens/s",
            "Avg generation throughput: %.1f tokens/s",
            "Running: %d reqs",
            "Waiting: %d reqs",
        ]
# ...
        if self.num_preemptions > 0:
            log_parts.append("Preemptions: %d")
            log_args.append(self.num_preemptions)

        log_parts.extend(
            [
                "GPU KV cache usage: %.1f%%",
                "Prefix cache hit rate: %.1f%%",
            ]
        )

Read the conditional at L283: the word "Preemptions" only appears in the log at all when the counter is non-zero. Its absence is the healthy state. Its appearance is the single most useful free signal either engine gives you, and it is why grepping for it is the first thing to do on a latency page. The interval is VLLM_LOG_STATS_INTERVAL, default 10.0 s (vllm/envs.py:L47). SGLang's equivalent line is built in python/sglang/srt/managers/scheduler_components/metrics_reporter.py:L811 and reads Decode batch, #running-req: N, ... gen throughput (token/s): X, #queue-req: Y.

A GPU at 100% utilisation is doing its job. The distinction you need is between busy (the batch is full, latency is at its designed value, the queue drains as fast as it fills) and overloaded (arrival rate exceeds service rate, so the queue grows without bound and every additional request makes every existing request slower). The metrics separate them cleanly if you look at the right three.

Why preemption is the signal rather than an error: when the scheduler cannot allocate blocks for the running set it evicts a victim back to the waiting queue. vLLM records it as an event, and the frontend counts it:

        for event in events:
            if event.type == EngineCoreEventType.QUEUED:
                req_stats.queued_ts = event.timestamp
                lora_states.request_waiting(req_id, lora_name)
            elif event.type == EngineCoreEventType.SCHEDULED:
                if req_stats.scheduled_ts == 0.0:  # ignore preemptions
                    req_stats.scheduled_ts = event.timestamp
                lora_states.request_running(req_id, lora_name)
            elif event.type == EngineCoreEventType.PREEMPTED:
                self.num_preempted_reqs += 1
                lora_states.request_waiting(req_id, lora_name)

SGLang calls the same thing retraction and, unlike vLLM, logs it loudly:

            msg_prefix = (
                "KV cache pool is full. Retract requests. "
                if kv_full_retract_flag
                else "Testing retraction. "
            )
            msg_details = f"#retracted_reqs: {len(retracted_reqs)}, #new_tokens_gained: {new_token_gained}"
# ...
            logger.warning(msg_prefix + msg_details)

Neither is an error. Both mean the same thing: you admitted more concurrent sequences than the KV pool can hold, and the engine is paying for it by throwing away computed KV. On vLLM the victim's prefix is recomputed from scratch on re-admission (§1.4), so one preemption of a request with 4k tokens of context costs a full 4k-token prefill. That is a capacity signal with a price tag, which is exactly what you want to alert on.

Deriving a starting threshold rather than inventing one. Llama-3-8B stores 128 KiB of KV per token. Take the book's reference budget, which §2.1 owns and which is careful to stay in one unit throughout — an "80 GB" H100 is 79.65 GiB, neither 80 GiB nor $80\times10^{9}$ bytes:

At 128 KiB per token that pool is 428,569 token slots (derived; the exact figure depends on your profiling run, and §2.6 owns the subtraction). One request at max_model_len = 8192 occupies 1 GiB, i.e. 1.9% of the pool. So a KV-utilisation alarm at 90% leaves headroom for about five more max-length admissions before the scheduler must start preempting. That is a defensible starting point to tune — not a measured SLO, and it moves the moment your prompt-length distribution moves. Re-derive it for your model and your max_model_len; do not copy the number.

Walk an overload through the actual counters, on our Llama-3-8B H100 with 428,569 KV token slots (derived above). Traffic doubles at $t_0$.

1. $t_0$. Arrivals rise. Scheduler.schedule() admits from the waiting queue until the token budget or the block budget binds. num_requests_running climbs; kv_cache_usage_perc climbs with it. Nothing is wrong.
2. $t_0 + 30$ s. The batch stops growing — admission now fails on blocks. num_requests_waiting starts a monotone climb, and vllm:num_requests_waiting_by_reason{reason="capacity"} is the label carrying it (that split exists precisely so you can distinguish "no room" from "deferred by a LoRA or KV-transfer constraint"; loggers.py:L513-L523). This is the first honest alert.
3. $t_0 + 90$ s. A running request needs one more block and the pool has none. The scheduler preempts a victim; IterationStats.update_from_events increments num_preempted_reqs; the periodic log line grows a Preemptions: N field for the first time. The victim's KV is discarded and it re-enters the waiting queue.
4. $t_0 + 120$ s. The victim is re-admitted and its 4k-token prefix is recomputed — which consumes prefill budget that would otherwise have gone to new arrivals. Prefill work per unit of useful output has risen. vllm:prompt_tokens rises faster than vllm:generation_tokens. This is the feedback loop: preemption creates work that causes more preemption.
5. $t_0 + 300$ s. request_queue_time_seconds now dominates e2e_request_latency_seconds. Meanwhile inter_token_latency_seconds has barely moved, because the decode steps themselves are still fine — and even if they had moved from 5 to 9 ms, the first bucket would have swallowed it. The ITL panel is green through the entire incident.

The lesson: on this workload, queue depth leads preemption by about a minute and preemption leads user-visible e2e latency by a few more. Alert on the leading indicator, not the lagging one, and certainly not on the one whose buckets cannot see.

OOM at startup versus OOM ten minutes in

These are different bugs with the same word. A startup OOM is a sizing error and it is loud:

    if available_memory <= 0:
        raise ValueError(
            "No available memory for the cache blocks. "
            "Try increasing `gpu_memory_utilization` when initializing the engine "
            "(this flag also controls CPU memory reservation on the CPU "
            "backend, despite its name). "
            "See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ "
            "for more details."
        )
# ...
        raise ValueError(
            f"To serve at least one request with the model's max seq len "
            f"({max_model_len}), ({format_gib(needed_memory)} GiB KV "
            f"cache is needed, which is larger than the available KV cache "
            f"memory ({format_gib(available_memory)} GiB). {estimated_msg}"
            f"Try increasing `gpu_memory_utilization` (which also controls "
            f"CPU memory on the CPU backend) or decreasing `max_model_len` "
            f"when initializing the engine. "

An OOM ten minutes into serving is the gpu_memory_utilization trap: the profiler measured peak activation memory during a one-shot profile run and reserved the rest for KV, so the moment a real batch exceeds the profiled peak there is nothing left to take. §2.6 owns that derivation and the reason 0.98 is a trap; do not raise the utilisation flag on a page without reading it. The tell-tale is a CUDA OOM raised from inside execute_model with a healthy uptime and a normal-looking KV utilisation.

vLLM has a third, subtler startup failure worth knowing because the message names the cause exactly:

        assert self.init_snapshot.free_memory >= free_gpu_memory, (
            "Error in memory profiling. "
            f"Initial free memory {format_gib(self.init_snapshot.free_memory)} GiB, "
            f"current free memory {format_gib(free_gpu_memory)} GiB. "
            "This happens when other processes sharing the same container "
            "release GPU memory while vLLM is profiling during initialization. "
            "To fix this, ensure consistent GPU memory allocation or "
            "isolate vLLM in its own container."
        )

NCCL hang

The signature is an absence: the step counter stops advancing, no exception is raised, the process stays resident, GPU utilisation reads high (the ranks are spinning in the collective) while SM occupancy is effectively idle, and /health still returns 200 on vLLM, because AsyncLLM.check_health only tests an errored flag (vllm/v1/engine/async_llm.py:L940-L943) and a hang sets no flag.

It is usually not a network fault. A collective hangs when the ranks disagree about what collective to call — different shapes, different order, or one rank taking a branch the others did not. The network is fine; the ranks diverged. That is why SGLang ships an explicit divergence detector: every rank hashes the sequence of decisions it made and all-reduces min and max of the hash; if they differ, the rank kills its own process rather than hanging.

def _check_for_consensus(events: list[str]) -> None:
    # Compute sha1 of concatenation of all msgs.
    hasher = hashlib.sha1()
    for msg in events:
        hasher.update(msg.encode("utf-8"))
# ...
    if not torch.equal(min_value, max_value):
        # When divergence, all rank should output the following log.
        logger.critical(
            f"Found rank divergence for {len(events)} events(s)! local hash: {value_bytes.hex()}, events = {events}"
        )
        for handler in logger.handlers:
            handler.flush()

        # os._exit instead of sys.exit: this runs in a background thread, where
        # SystemExit would only kill the thread, not the process. os._exit tears
        # down the whole scheduler process so a TP/PP mismatch can never
        # silently keep serving.
        os._exit(1)

It is off by default (SGLANG_ENABLE_RANK_CONSENSUS_CHECKER, and configure() logs "the server will suicide if rank divergence detected"), it costs a gloo all-reduce per batch of decisions, and at this SHA the only annotated call sites are three methods in python/sglang/srt/mem_cache/unified_radix_cache.py (L495, L1753, L1971) — i.e. it catches cache-eviction divergence, which is exactly the class of bug where one rank frees a block the others keep. It is a debugging tool to turn on when reproducing a hang, not a production default.

The first-party diagnostic set is small and worth memorising, from docs/usage/troubleshooting.md:L33-L41: VLLM_LOGGING_LEVEL=DEBUG, VLLM_LOG_STATS_INTERVAL=1., CUDA_LAUNCH_BLOCKING=1, NCCL_DEBUG=TRACE, and VLLM_TRACE_FUNCTION=1 — the last of which the doc itself warns slows generation by "over 100x". The same file (L83-L175) ships a standalone two-node NCCL test script; if that hangs, the problem is below vLLM and you should stop debugging vLLM. VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS defaults to 300 (vllm/envs.py:L243) and bounds how long a step may take before the RPC gives up.

A dead worker — and why the engine dies with it

vLLM has a directory named vllm/v1/fault_tolerance/. It is 222 lines across three files. §5.5 flagged that it does not do what the name suggests, and reading it confirms that exactly. Here is the fault handler:

    def on_fault(self, exc: Exception):
        """Called by the wrapper when the busy loop raises an exception."""
        self.resumed.clear()
        logger.warning(
            "[FT] Busy loop raised %s. Waiting for recovery.", type(exc).__name__
        )

        engine = self.engine
        aborted = engine.scheduler.finish_requests(None, RequestStatus.FINISHED_ABORTED)
        engine._send_abort_outputs(aborted)
        if engine.batch_queue is not None:
            engine.batch_queue.clear()
        if (
            hasattr(engine.model_executor, "is_failed")
            and engine.model_executor.is_failed
        ):
            self.status_type = EngineStatusType.DEAD
        else:
            self.status_type = EngineStatusType.UNHEALTHY

And here is the entire recovery action:

    def retry(self, ft_request: FaultToleranceRequest) -> FaultToleranceResult:
        engine = self.engine
        executor = engine.model_executor

        with set_current_vllm_config(engine.vllm_config):
            ft_request.params.update(self._reinit_dp_group())
        if hasattr(engine, "step_counter"):
            engine.step_counter = 0

        executor.collective_rpc("handle_ft_command", args=(ft_request,))

        self.status_type = EngineStatusType.HEALTHY
        logger.info("[FT] Engine %d status -> HEALTHY", self.engine_index)

_reinit_dp_group() tears down and rebuilds the stateless data-parallel gloo group on fresh ports. That is the whole mechanism. It restores a DP/EP replica set after one replica dropped out; it does nothing whatsoever for a dead tensor-parallel rank. And the branch at L92-L96 is explicit about it: if the executor reports is_failed, the status is DEAD, not UNHEALTHY, and retry refuses to run (handle_command rejects any instruction whose status is not UNHEALTHY, L54-L67).

is_failed is set in exactly one place — the worker liveness monitor:

        def monitor_workers():
            sentinels = [h.proc.sentinel for h in workers]
            died = multiprocessing.connection.wait(sentinels)
            _self = self_ref()
            if not _self or getattr(_self, "shutting_down", False):
                logger.debug("MultiprocWorkerMonitor: shutdown already initiated")
                return
            _self.is_failed = True
            proc = next(h.proc for h in workers if h.proc.sentinel == died[0])
            logger.error(
                "Worker proc %s died unexpectedly (exit code: %s), "
                "shutting down executor.",
                proc.name,
                proc.exitcode,
            )
            _self.shutdown()

SGLang's answer to the same problem is more direct: since a stuck collective produces no Python exception, it watches the forward counter from a daemon thread and shoots the process group.

    def _watchdog_once(self):
        watchdog_last_counter = 0
        watchdog_last_time = time.perf_counter()

        while True:
            current = time.perf_counter()
            if self.is_active():
                current_counter = self.get_counter()
                if watchdog_last_counter == current_counter:
                    if current > watchdog_last_time + self.watchdog_timeout:
                        break
                else:
                    watchdog_last_counter = current_counter
                    watchdog_last_time = current
            time.sleep(self.watchdog_timeout / 2)

        if self.dump_info is not None and (info_msg := self.dump_info()):
            logger.error(f"{self.debug_name} debug info:\n{info_msg}")

        pyspy_dump_schedulers()
        logger.error(
            f"{self.debug_name} watchdog timeout "
            f"({self.watchdog_timeout=}, {self.soft=})"
        )
# ...
        if not self.soft:
            # Wait for some time so that the parent process can print the error.
            time.sleep(5)
            self.parent_process.send_signal(signal.SIGQUIT)

The counter is scheduler.forward_ct, the timeout defaults to 300 s (--watchdog-timeout, python/sglang/srt/server_args.py:L1261-L1264, whose help text is candid: "the server will crash to prevent hanging"), and before it fires it calls pyspy_dump_schedulers() so the stack of every stuck rank lands in the log. There is also a --soft-watchdog-timeout that dumps without killing — the right thing to set below the hard timeout in production so you get a stack trace of the near-miss. And separately, SubprocessWatchdog handles the case where a child dies without Python noticing:

class SubprocessWatchdog:
    """Monitors subprocess liveness and triggers SIGQUIT when a crash is detected.

    When a subprocess crashes (e.g., NCCL timeout causing C++ std::terminate()),
    Python exception handlers never run, leaving the main process as a zombie
    service. This watchdog polls subprocess liveness in a daemon thread and
    sends SIGQUIT to trigger proper cleanup.

    See: https://github.com/sgl-project/sglang/issues/18421
    """

"Leaving the main process as a zombie service" is the failure mode you should fear most: an HTTP server that accepts requests and never answers. Both engines have converged on the same answer — detect the stall, kill the whole tree, let the orchestrator restart it.

Model load failure, and "loads but outputs nonsense"

A load failure is easy: the process exits during startup with a shape or key mismatch. The dangerous sibling is a model that loads cleanly and emits garbage — a silently mismatched weight permutation, a quantisation scale applied to the wrong axis, a rotary base read from the wrong config key. Nothing in /metrics will tell you. §8.4 owns the loader and the mapping bugs that cause this; the operational point is that output quality is not observable from the serving system and needs an independent canary — a fixed prompt with a known-good completion, checked on every deploy. vLLM offers one narrow numerical tripwire: VLLM_COMPUTE_NANS_IN_LOGITS=1 (default off, vllm/envs.py:L247) enables vllm:corrupted_requests, a counter of requests that saw NaNs in logits (loggers.py:L572-L583). It catches numerical collapse, not wrong weights.

Slow or disconnected clients

A slow reader can accumulate frontend output without keeping completed engine KV live. Live generation retains its required KV until completion or abort; buffered completed text has a separate lifetime. A hundred slow clients can therefore cause byte-buffer pressure even after engine slots are released. This is the backpressure path that §9.3 traces end to end; from an observability standpoint the signature is num_requests_running high, gen_throughput low, and e2e_request_latency far exceeding request_inference_time. SGLang gives you two blunt instruments for it — SGLANG_REQ_WAITING_TIMEOUT and SGLANG_REQ_RUNNING_TIMEOUT, which abort with "Request waiting timeout reached." / "Request running timeout reached." and HTTP 503 (python/sglang/srt/managers/scheduler.py:L1655-L1669 and L2878-L2903). Both default to disabled.

vLLM's OpenTelemetry integration is real but thin. Set --otlp-traces-endpoint and the output processor emits one span per request, named llm_request, at finish time:

        instrument_manual(
            span_name="llm_request",
            start_time=arrival_time_ns,
            attributes=attributes,
            context=trace_context,
            kind=SpanKind.SERVER,
        )

The span carries the phase breakdown as attributes rather than as child spans: gen_ai.latency.time_to_first_token, .e2e, .time_in_queue, .time_in_model_prefill, .time_in_model_decode, .time_in_model_inference, plus token counts and sampling params (vllm/tracing/utils.py:L24-L45). W3C traceparent/tracestate headers on the incoming HTTP request are propagated, so the span nests under your caller's trace (vllm/tracing/utils.py:L11-L12). What you do not get is a span per scheduler step or per forward pass; --collect-detailed-traces exists in the config (vllm/config/observability.py) and is documented as "possibly costly and or blocking".

SGLang goes the other way: a hierarchy of named stages, each of which can become a span and each of which is also a Prometheus histogram label.

class RequestStage:
    # Tokenizer/gRPC Server
    TOKENIZE = RequestStageConfig(
        "tokenize",
        level=1,
    )
    API_SERVER_DISPATCH = RequestStageConfig(
        "api_server_dispatch",
        level=2,
    )
# ...
    REQUEST_PROCESS = RequestStageConfig(
        "request_process",
        level=2,
        metrics_is_observed=True,
    )
    PREFILL_WAITING = RequestStageConfig(
        "prefill_waiting",
        level=1,

The stage set continues through prefill_forward, chunked_prefill, decode_loop, decode_forward and the PD-disaggregation stages (prefill_bootstrap, prefill_transfer_kv_cache, decode_prepare). Stages flagged metrics_is_observed land in sglang:per_stage_req_latency_seconds{stage=...}, whose buckets are exponential_buckets(start=0.001, width=1.62, length=30) (metrics_collector.py:L733-L739) — note that this histogram, unlike the ITL one on either engine, has resolution starting at 1 ms. Tracing is gated by SGLANG_TRACE_LEVEL (default 3) and a --trace-modules filter (python/sglang/srt/observability/trace.py:L36-L37 and L77-L83), with an async exporter process behind SGLANG_TRACE_ASYNC=1 so span construction stays off the scheduler thread (python/sglang/srt/observability/trace.py:L133-L142).

The tradeoff is the usual one. vLLM's single span is nearly free and answers "where did this request spend its time" at the phase level. SGLang's stage tree answers "which of eleven pipeline stages regressed" but costs a span per stage per request and needs a filter to stay affordable. If you run PD disaggregation, the stage tree is worth it — the KV-transfer stages are where the failures live.

Prompts are user data. Logging them is a privacy and compliance decision that belongs to whoever owns your data policy, not a debugging default you flip on during an incident and forget. Say it out loud in your runbook, because the flags make it a one-liner.

Both engines default to not logging request content. vLLM's enable_log_requests defaults to False (vllm/engine/arg_utils.py:L2879). SGLang's --log-requests defaults to False too — but note what happens when you turn it on:

    log_requests: A[
        bool,
        "Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level",
        NS("observability"),
    ] = False
    log_requests_level: A[
        int,
        Arg(
            help="0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output.",
            choices=[0, 1, 2, 3],
        ),
        NS("observability"),
    ] = 2

The default level is 2 — partial input and output. So --log-requests with no further argument writes user prompts to your log pipeline. If you need request-level logs for debugging without the content, that is --log-requests --log-requests-level 1, and it is the setting almost everyone actually wants. HTTP headers are allow-listed rather than dumped (python/sglang/srt/utils/request_logger.py:L30-L41).

The second trap is crash dumps. --crash-dump-folder is unset by default, and when set it writes the last five minutes of requests — full request objects, prompts included — to disk on SIGTERM or SIGQUIT (python/sglang/srt/managers/tokenizer_manager.py:L2956-L2995). It is an excellent debugging tool and a data-retention obligation the moment you enable it. Give the folder a retention policy before you give it a path.

What you should log unconditionally: request IDs, token counts, finish reasons, timings, the scheduler's periodic line, and every WARNING and above from the engine. That set is enough to reconstruct an incident and contains no user text.

Liveness is not readiness. A replica that has started its process, mapped its weights, and is 40 seconds into CUDA-graph capture is alive and must not receive traffic. Both engines make the distinction available, in different shapes.

vLLM's /health is a pure liveness probe on the engine:

@router.get("/health", response_class=Response)
async def health(raw_request: Request) -> Response:
    """Health check."""
    client = engine_client(raw_request)
    if client is None:
        # Render-only servers have no engine; they are always healthy.
        return Response(status_code=200)
    try:
        await client.check_health()
        return Response(status_code=200)
    except EngineDeadError:
        return Response(status_code=503)

It answers exactly one question: has the engine died? It does not answer "is this replica warmed up", because the HTTP server does not start serving until startup completes. For load-balancer weighting there is a separate /load endpoint returning the in-flight request count (vllm/entrypoints/serve/instrumentator/basic.py:L30-L49), which is what §9.4's balancers consume.

SGLang generation health uses a one-token request; ordinary /health only follows that path when SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION enables it. Do not infer generation success from the ordinary short-circuit route:

@app.get("/health")
@app.get("/health_generate")
async def health_generate(request: Request) -> Response:
    """
    Check the health of the inference server by sending a special request to generate one token.

    If the server is running something, this request will be ignored, so it creates zero overhead.
    If the server is not running anything, this request will be run, so we know whether the server is healthy.
    """

    if _global_state.tokenizer_manager.gracefully_exit:
        logger.info("Health check request received during shutdown. Returning 503.")
        return Response(status_code=503)

    if _global_state.tokenizer_manager.server_status == ServerStatus.Starting:
        return Response(status_code=503)

Two behaviours to note. ServerStatus.Starting returns 503 — that is your readiness gate, for free. And gracefully_exit returns 503 before the process stops accepting work, which is exactly the drain semantic you want: SIGTERM flips the flag, the health check goes red, the load balancer stops sending new requests, in-flight requests finish, then the process exits.

    def sigterm_handler(self, signum=None, frame=None):
        logger.warning(
            f"SIGTERM received. {signum=} {frame=}. Draining requests and shutting down..."
        )
        self.tokenizer_manager.gracefully_exit = True

    def running_phase_sigquit_handler(self, signum=None, frame=None):
        logger.error(
            f"SIGQUIT received. {signum=}, {frame=}. It usually means one child failed."
        )
        # Stop subprocess watchdog before killing processes to prevent false-positive
        # crash detection during normal shutdown
        if self.tokenizer_manager._subprocess_watchdog is not None:
            self.tokenizer_manager._subprocess_watchdog.stop()
        self.tokenizer_manager.dump_requests_before_crash()
        kill_process_tree(os.getpid())

SIGTERM means drain; SIGQUIT means a child died, dump and kill the tree. If your orchestrator sends SIGKILL after a 30-second grace period and your requests routinely run for 60 seconds, you are truncating user responses on every deploy. Set the grace period from your own e2e_request_latency p99, not from the platform default.

Why restart is the primary recovery mechanism. In a stateless service you drop the bad replica and the request retries elsewhere. Here, three properties conspire against that. The GPU state (weights, CUDA graphs, KV pool) takes tens of seconds to rebuild, so failover is not free. The KV cache is per-replica, so a failover discards every cached prefix and the surviving replicas take a cold-cache throughput hit. And the collective has no quorum: there is no configuration in which three of four ranks keep serving. Given all that, the honest design is what both engines chose — detect the stall fast, die loudly, and let the orchestrator bring back a clean process. Your job on call is to make the detection fast (set --watchdog-timeout and its soft twin sensibly), make the restart clean (drain on SIGTERM, correct grace period), and make the next one preventable by capturing a py-spy dump before the process goes away.

Everything below runs against a single small model; none of it needs a big GPU, and the first two need no GPU at all.

V=~/Documents/other_git_repos/vllm
S=~/Documents/other_git_repos/sglang

# every Prometheus series vLLM registers, in registration order
grep -n 'name="vllm:' $V/vllm/v1/metrics/loggers.py

# ... and SGLang's, which is roughly 3x longer
grep -c 'name="sglang:' $S/python/sglang/srt/observability/metrics_collector.py

# the whole of vLLM's "fault tolerance"
wc -l $V/vllm/v1/fault_tolerance/*.py

vllm serve meta-llama/Meta-Llama-3-8B-Instruct --max-model-len 8192

# drive some load, then:
curl -s localhost:8000/metrics | grep '^vllm:inter_token_latency_seconds_bucket'
# expect: the le="0.01" bucket count equals the le="+Inf" count.
# Every sample is in bucket one. Now compute the honest mean instead:
curl -s localhost:8000/metrics | grep -E '^vllm:inter_token_latency_seconds_(sum|count)'

# Shrink the KV pool until the running set cannot fit, then send concurrent
# long-context requests. --kv-cache-memory-bytes takes bytes.
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --max-model-len 8192 --max-num-seqs 64 --kv-cache-memory-bytes 2147483648

# In another shell, watch for the field that only appears when non-zero:
#   "Preemptions: N"
# and confirm it in Prometheus:
watch -n1 "curl -s localhost:8000/metrics | grep '^vllm:num_preemptions_total'"

# Buckets tuned around a derived 4.48 ms decode floor:
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --enable-metrics \
  --bucket-inter-token-latency 0.003 0.004 0.005 0.006 0.008 0.012 0.02 0.05 0.2 \
  --bucket-time-to-first-token 0.01 0.02 0.05 0.1 0.25 0.5 1.0 2.5 10.0

# There is no fault injector at this SHA; the way to exercise the watchdog is
# to shrink its timeouts until an ordinary long batch trips them. The soft
# timeout dumps every rank's stack and keeps serving; the hard one dumps and
# then SIGQUITs the tree. Do NOT run this against anything you care about.
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --soft-watchdog-timeout 5 --watchdog-timeout 20

§10.3 covers generating load that resembles your workload rather than load that flatters your numbers; use its harness, not a for loop of curl.

• vLLM, docs/usage/troubleshooting.md — the first-party failure catalogue. The two-node NCCL sanity script (L83–L175) is the fastest way to decide whether a hang is vLLM's problem or the fabric's.
• vLLM metrics design document — the rationale for the V1 StatLoggerBase split between SchedulerStats (engine core) and IterationStats (frontend), which is why some metrics are per-engine and others are aggregated.
• sgl-project/sglang#18421 — the zombie-service issue that produced SubprocessWatchdog; a good short read on why a crashed child does not necessarily kill the parent.
• Prometheus, histogram_quantile — read the interpolation rules, especially the treatment of the lowest bucket, before you trust a percentile panel again. Native histograms remove this whole class of problem; neither engine emits them at these SHAs.
• OpenTelemetry GenAI semantic conventions — vLLM copies the attribute names into vllm/tracing/utils.py rather than importing them, to avoid version conflicts; worth knowing if your collector enforces the spec.
• §10.5 for turning these metrics into a capacity model, and §2.6 for the VRAM subtraction behind every KV threshold in this chapter.