MLOps & Serving: From Notebook to Something On Call

A model that only exists in a notebook has no value. The gap between "it scores 0.91 in cross-validation" and "it makes correct decisions for real users at 3 a.m." is filled by a set of unglamorous tools and disciplines. This page covers them in the order you meet them.

Every arrow is a place where projects fail. The loop closing — production feeding back into training data — is what separates a system from a script.

Experiment tracking

You will run hundreds of experiments. Without tracking, you will not be able to answer "what produced the model currently in production?"

import mlflow

mlflow.set_experiment("churn-v3")
with mlflow.start_run(run_name="lgbm-tuned"):
    mlflow.log_params(params)
    mlflow.log_metrics({"val_auc": auc, "val_ap": ap, "train_auc": train_auc})
    mlflow.log_artifact("confusion_matrix.png")
    mlflow.set_tags({"git_sha": sha, "data_version": data_hash, "owner": "risk-ml"})
    mlflow.sklearn.log_model(pipeline, "model", signature=signature,
                             input_example=X_val.head())

Log these every run, without exception:

The seed one matters more than people expect: on small datasets the spread across seeds often exceeds the improvement being claimed. Run 3–5 seeds and report mean ± std, not a single lucky number.

Data and model versioning

Git does not handle a 40 GB Parquet directory. The standard options:

Time travel is the feature that matters. "Reproduce the model we shipped in March" requires reading the data as it was in March, not as it is now. Delta and Iceberg support time-travel queries with format-specific syntax, but retention and vacuum policies can remove required history. Preserve snapshots explicitly; an old version identifier alone is not the data.

mlflow.register_model("runs:/<run_id>/model", "churn")
client = mlflow.MlflowClient()
client.set_registered_model_alias("churn", "candidate", "7")
resolved = client.get_model_version_by_alias("churn", "candidate")
immutable_uri = f"models:/churn/{resolved.version}"

A registry provides immutable versions, mutable aliases/tags, and lineage. Legacy stage-transition APIs are deprecated in MLflow; resolve an alias to the actual version before loading/logging a deployed artifact. See the registry workflow.

Feature stores

The problem a feature store solves is train/serve skew: the feature computed in a training SQL query and the feature computed in the serving Python are subtly different. An ordinary offline score may miss this, but historical replay and offline-versus-served feature parity tests can expose the mismatch.

The two properties that define a feature store:

1. One definition, two paths. The same transformation code produces the offline training table and the online serving value.
2. Point-in-time correctness. When building a training set, each row's features must be the values that were known at that row's timestamp — not the current values. This is an as-of join, and getting it wrong is the most damaging form of leakage in production ML, because it inflates offline metrics and can be detected offline through timestamp/availability assertions, historical replay, feature parity checks and held-out backtests.

If you build nothing else, build the point-in-time join correctly.

Serving patterns

Batch first. If the business can tolerate day-old predictions, batch scoring removes an entire class of operational problems: no latency budget, no autoscaling, easy retries, trivial rollback. Reach for online serving when freshness genuinely matters.

A latency budget

A 100 ms end-to-end budget spends roughly:

Feature retrieval is frequently the bottleneck, not the model. Optimising a 15 ms model to 8 ms while a feature lookup takes 40 ms is wasted effort — profile the whole path before optimising the part you find most interesting.

Serve p99, not the mean. A mean of 40 ms with a p99 of 900 ms means 1% of users have a bad experience, and in a fan-out architecture where one request touches 20 services, independent 1%-tail events imply probability \(1-0.99^{20}\approx18.2\%\) of at least one tail event, not nearly every request. Correlation changes that probability.

Serving frameworks

Dynamic batching is the single highest-leverage server feature. GPUs are throughput devices; serving one request at a time leaves them idle. Grouping concurrent requests for a few milliseconds trades a small latency increase for a large throughput multiplier. For LLMs, continuous batching goes further, admitting new requests into a running batch as others finish rather than waiting for the whole batch to complete.

Runnable local serving contract

This CPU fixture tests a probability API with FastAPI's in-process client. It does not start a network server, write artifacts, or contact a registry. A real loader must verify artifact integrity/trust and return an immutable version. A default MLflow sklearn pyfunc commonly calls predict, yielding labels; use an explicit probability wrapper or loaded estimator's predict_proba for a probability API.

from contextlib import asynccontextmanager
import numpy as np
from fastapi import FastAPI, HTTPException
from fastapi.testclient import TestClient
from pydantic import BaseModel, Field
from sklearn.linear_model import LogisticRegression

class Request(BaseModel):
    f0: float = Field(..., allow_inf_nan=False)
    f1: float = Field(..., allow_inf_nan=False)
    class Config:
        extra = "forbid"

def create_app(loader):
    @asynccontextmanager
    async def lifespan(app):
        model, version = loader()
        warm = model.predict_proba(np.zeros((1, 2)))
        if warm.shape != (1, 2) or not np.isfinite(warm).all():
            raise RuntimeError("warmup failed")
        app.state.model, app.state.version = model, version
        app.state.ready = True
        try:
            yield
        finally:
            app.state.ready = False

    app = FastAPI(lifespan=lifespan)
    app.state.ready = False

    @app.get("/health")
    def health():
        return {"ok": True}

    @app.get("/ready")
    def ready():
        if not app.state.ready:
            raise HTTPException(503, "not ready")
        return {"ok": True}

    @app.post("/predict")
    def predict(request: Request):
        if not app.state.ready:
            raise HTTPException(503, "not ready")
        positive = np.flatnonzero(app.state.model.classes_ == 1).item()
        score = app.state.model.predict_proba([[request.f0, request.f1]])[0, positive]
        return {"score": float(score), "model_version": app.state.version}
    return app

X = np.array([[-2., 0], [-1., 1], [1., -1], [2., 0]])
model = LogisticRegression(random_state=0).fit(X, [0, 0, 1, 1])
app = create_app(lambda: (model, "fixture-version-7"))
assert TestClient(app).get("/ready").status_code == 503
with TestClient(app) as client:
    assert client.get("/ready").status_code == 200
    result = client.post("/predict", json={"f1": 0., "f0": 2.}).json()
    assert result["model_version"] == "fixture-version-7"
    assert np.isclose(result["score"], model.predict_proba([[2., 0.]])[0, 1])
    assert client.post("/predict", json={"f0": 1.}).status_code == 422
    assert client.post("/predict", json={"f0": 1., "f1": 0., "secret": 3}).status_code == 422
assert not app.state.ready
print("readiness status, fixed schema, class probability and immutable version passed")

Log a minimized prediction identifier, immutable model/schema version and permitted diagnostics for delayed labels. Raw inputs may contain secrets or sensitive attributes: define redaction, retention, access controls and deletion instead of logging all inputs by default. Add authentication, authorization, request deadlines, bounded queues, overload responses and cancellation before network deployment. FastAPI recommends lifespan management.

Containerisation and orchestration

FROM python:3.11-slim AS build
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt

FROM python:3.11-slim
WORKDIR /app
COPY --from=build /install /usr/local
COPY src/ /app/src/
COPY model/ /app/model/
ENV PYTHONUNBUFFERED=1 OMP_NUM_THREADS=1
RUN useradd --create-home --uid 10001 appuser
USER appuser
EXPOSE 8080
HEALTHCHECK CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/ready', timeout=2)"
CMD ["uvicorn", "src.app:app", "--host", "0.0.0.0", "--port", "8080"]

On Kubernetes, the pieces that matter for ML specifically are HPA on a custom metric (queue depth or GPU utilisation, not CPU), pod disruption budgets so a node drain does not take out every replica, and a warm-up request in the readiness probe so the first real request does not pay JIT compilation cost.

Deployment strategies

Shadow mode first, always. It catches schema mismatches, latency regressions, and unexpected input distributions. With later outcome labels it can compare predictive quality, but cannot directly identify the causal effect of taking different actions. Resource isolation and privacy controls still matter; shadow mode does not certify safety.

Automate rollback on a metric gate. A deployment that requires a human to notice a problem at 3 a.m. is not a deployment strategy.

Monitoring

Four layers, in increasing difficulty:

1. Operational

Latency (p50/p95/p99), throughput, error rate, saturation, cost per prediction. Standard Prometheus/Grafana territory. Alert on p99 and error rate.

2. Data quality

Schema conformance, null rates, cardinality, range checks, duplicate rates, freshness. These fail more often than models do, and they fail silently.

import pandera as pa

schema = pa.DataFrameSchema({
    "age":    pa.Column(int,   pa.Check.in_range(18, 120)),
    "income": pa.Column(float, pa.Check.ge(0), nullable=True),
    "region": pa.Column(str,   pa.Check.isin(VALID_REGIONS)),
})
schema.validate(df, lazy=True)     # collect every violation, not just the first

Great Expectations, Pandera, Evidently, and dbt tests all cover this ground. Run the checks at the boundary — where data enters your system — and fail loudly.

3. Drift

The population stability index is the standard practical measure:

over binned feature values, with \(e\) the expected (training) proportion and \(a\) the actual. Conventional thresholds: \(< 0.1\) stable, \(0.1\)–\(0.25\) investigate, \(> 0.25\) significant shift.

A useful and under-used detector: train a classifier to distinguish training data from production data. Evaluate its AUC on held-out, appropriately grouped or temporal rows; training AUC can reflect memorization. Importances suggest diagnostic features but do not uniquely identify marginal changes, especially with correlated features or changed interactions.

Prediction drift is your early-warning system, because it needs no labels. If the mean predicted probability moves from 0.03 to 0.11 overnight, something happened — upstream, in the world, or in your code.

4. Model quality

The hard one, because labels arrive late or never.

Feedback loops are the subtle failure. A recommender only observes outcomes for items it chose to show. Training the next model on that log makes it more confident in what it already believed. Log the propensity (probability of showing each item) so you can inverse-propensity-weight the training data, and keep a small randomised exploration slice.

Retraining

A retraining pipeline must gate on evaluation, not just complete. The minimum gates:

1. Data validation passes.
2. New model beats the current production model on a held-out set and on critical slices.
3. No regression on protected segments.
4. Latency and model size within budget.
5. Shadow mode for a fixed period before promotion.

Warm-starting from the previous checkpoint is faster but accumulates drift and makes reproducibility harder; retraining from scratch is cleaner. Prefer from scratch unless training cost forbids it.

CI/CD for ML

The following is a workflow sketch: a runnable GitHub Actions job additionally needs runs-on, checkout, Python/dependency setup, credentials policy and the referenced repository commands/files. It has not been executed as CI here.

on: [pull_request]
jobs:
  test:
    steps:
      - run: ruff check . && mypy src/
      - run: pytest tests/unit
      - run: pytest tests/data          # schema and expectation checks
      - run: python -m src.train --smoke --max-steps 50
      - run: pytest tests/model         # behavioural tests on a fixed checkpoint
      - run: python -m src.eval --gate metrics.json --min-auc 0.85

Tests specific to ML, beyond ordinary unit tests:

The invariance/directional/minimum-functionality trio comes from behavioural testing (the CheckList methodology) and is far more useful than an aggregate metric for catching the failures users actually notice.

LLM-specific operations

Serving and monitoring language models differs enough to call out:

The most important operational habit: an evaluation set of real prompts from your own product, scored on every prompt or model change. Public benchmarks will not tell you whether your extraction prompt regressed.

Cost

Cascades deserve a mention: route every request to a small fast model, and only escalate to the large one when the small model's confidence is low. On many workloads this cuts cost by more than half at negligible quality loss, and it is easy to tune with a single confidence threshold.

Capstone: train, package, reload, serve

The earlier in-process example isolates the HTTP contract. This capstone connects the complete artifact path: a grouped text dataset, train-only preprocessing, development-set model selection, final evaluation, a serialized pipeline, an integrity-checked manifest, and the same predictions through FastAPI. It is a CPU-only offline fixture, with no model downloads, cloud account, registry, telemetry, or paid service.

Downloads: complete Python capstone and focused environment. The source includes training, loading, service construction, and the executable demonstration. These teaching pins reproduce the example; they are not a security-maintained production lockfile.

Run the complete workflow

Download the two files into one directory and run with Python 3.11:

python3.11 -m venv .venv
.venv/bin/python -m pip install -r requirements-artifact-capstone.txt
.venv/bin/python artifact_capstone.py demo --output ./ticket-artifact

The output directory must not already exist. The script refuses to overwrite a previous release. A failed run can leave an incomplete directory; inspect it and choose a new output path for the next run. In a registry implementation, build under a temporary version and promote it atomically only after validation.

The command prints the chosen regularization, development and test log loss, per-class scores, confusion matrix, immutable model digest, manifest digest, and roundtrip: passed. It executes HTTP requests through an in-process ASGI test client. It does not bind a port or deploy a public server. Use train instead of demo to write the artifact without running the HTTP demonstration.

To call that same artifact through a real local HTTP listener, use the manifest digest printed by your own trusted training run:

.venv/bin/python artifact_capstone.py serve --artifact ./ticket-artifact \
  --manifest-sha256 YOUR_TRUSTED_MANIFEST_SHA256 --trust-artifact --port 8001

The server binds only to 127.0.0.1, runs one worker, and disables access logs. Choose another port if 8001 is occupied. In another terminal:

curl --fail http://127.0.0.1:8001/ready
curl --fail http://127.0.0.1:8001/predict \
  -H 'Content-Type: application/json' \
  -d '{"texts":["please refund the invoice","reset my password"]}'

Stop the server with Ctrl+C. Startup loads and validates the artifact before accepting requests. The explicit trust flag is not an authentication mechanism; do not take the digest from a bundle whose producer you do not trust. This single-process development command does not expose the model publicly and has none of the production access controls described below.

This fixture contains no private data. A real release should not automatically embed raw training texts or customer identifiers in downloadable bundles. Store access-controlled references and approved redacted prediction fixtures when the data policy requires them. Hashes of sensitive texts are not anonymization.

Split isolation is an executable property

The task has three mutually exclusive labels: account, billing, and delivery. Each category has twelve synthetic incidents; every incident has two related messages. Splitting individual messages would allow the paired variant to land in another split. The fixed splitter instead assigns eight incident groups per class to training, two to development, and two to testing. The seed is 41, and all group IDs are saved.

Training uses only the text field. IDs, group labels, and targets are never passed as separate predictive features. Every incident also has a unique neutral marker token. Tests verify that training markers occur in the fitted vocabulary, while development and test markers do not. Fitting TfidfVectorizer on all texts before the split would fail this check, even if no labels were passed to it: vocabulary and document-frequency statistics are learned state.

The script fits three complete pipelines with C in {0.5, 2.0, 8.0} on the same training split. It chooses the smallest development log loss, with smaller C as a deterministic tie-breaker, then evaluates the selected model once on the test split. It deliberately does not refit on train plus development, so the train-only vocabulary invariant remains straightforward. A later refit is a different artifact: rerun the parity fixture and identify exactly which evaluation data remain untouched. Review pipelines and cross-validation and text evaluation for the wider protocol.

This is a plumbing dataset, not a language understanding benchmark. Message templates repeat across incident groups, and category words make the task easy. The reproduced fixture yields macro-F1 of 1.0 on just twelve test messages, four per class; development and test losses are also equal because of the symmetric templates. These values diagnose whether the example is wired correctly. They do not establish generalization, calibration, statistical significance, or robustness to new organizations, topics, languages, or time periods.

Trust is separate from a checksum

joblib persists Python objects using pickle machinery, so loading a malicious artifact may execute code. An attacker who controls a model and its adjacent checksum can replace both. The loader therefore requires two explicit inputs: trusted=True, meaning the producer is trusted to supply executable objects, and an expected SHA-256 for the manifest obtained from an independent trusted release channel. This example does not authenticate that channel or implement signatures. These constraints follow scikit-learn's model persistence guidance.

from artifact_capstone import train_artifact, load_artifact

# A new artifact produced locally by code you have inspected.
release = train_artifact("./another-ticket-artifact")
model, manifest = load_artifact(
    "./another-ticket-artifact",
    trusted=True,
    expected_manifest_sha256=release["manifest_sha256"],
)
probabilities = model.predict_proba(["please refund the invoice"])
print(manifest["classes"], probabilities.tolist())

Before deserialization, the loader checks the manifest digest, supported format and input schema, exact runtime version map, fixed filenames, every file digest, and the model's content identity. It reads the model bytes once, verifies those bytes, and deserializes that same in-memory buffer. Reopening the pathname after verification would introduce a replacement race. Fixed filenames also prevent a manifest from directing the loader to arbitrary paths.

After loading, it checks the expected Pipeline steps and class ordering, then warms up the probability path. These post-load checks catch incompatible trusted artifacts; they do not make unpickling untrusted bytes safe. For different trust boundaries, consider an appropriately inspected safer format or a supported runtime export, and test prediction parity again. Cross-version scikit-learn loading is unsupported; the example deliberately rejects version differences rather than relying on a successful deserialization as proof of compatibility.

artifact_version identifies the serialized model bytes. The manifest digest identifies the broader release, including evaluation and data references. Identical predictions need not imply identical serialized bytes across runtimes; preserve both content identity and behavioral fixtures. The source-code digest records provenance but is not a complete environment or supply-chain attestation.

An explicit HTTP contract

The request is a JSON object with exactly one field, texts, holding 1 to 32 strict strings. Each string must contain non-whitespace content and be at most 4,000 Python characters. Extra fields, numeric coercions, nulls, empty lists, oversized batches, and oversized strings receive HTTP 422. Character limits are not encoded-byte limits or tokenizer limits; an ingress proxy still needs its own request-body budget.

{"texts": ["please refund the invoice", "reset my password"]}

The response includes schema_version, artifact_version, ordered classes, and one prediction per input in the same order. Every probability vector uses the returned class ordering. Never assume column one means the business-positive class: map through classes_. The test compares saved, reloaded, and served probabilities with rtol=0 and atol=1e-12, appropriate for this unchanged CPU pipeline. A quantized or exported runtime needs a separately justified tolerance and task-level quality checks.

The app loads once in its lifespan context, then warms up before setting readiness. /health is a process-level liveness response; /ready returns 503 before initialization and 200 with the model version after success. Missing or corrupted artifacts abort startup. Shutdown clears readiness and the model reference. In the tests, entering with TestClient(app) executes startup and shutdown; constructing the client alone does not. This follows FastAPI's lifespan and testing guidance.

Failure cases and worked checks

The repository's site/test_artifact_capstone.py checks these boundaries, a fresh-process reload, deterministic predictions on repeat training, preserved batch order, HTTP parity, lifecycle behavior, CLI execution, and refusal to overwrite existing output. The standalone demo command provides a smaller smoke test without requiring the repository test harness.

Why is the unseen-word result not an exception? TF-IDF maps an all-OOV text to a zero feature row. Logistic regression can still return intercept-driven probabilities and an argmax label. That answer is mathematically valid but may be inappropriate for the application. Monitor OOV coverage and design a validated abstention policy on realistic unsupported inputs; never infer reliability solely from predict_proba being available. See classification robustness.

Does parity prove that the model is useful? No. It proves that these tested paths implement the same fitted computation within the chosen tolerance. A consistently wrong classifier can pass every artifact parity test. Quality, calibration, subgroup performance, and real service reliability need their own acceptance gates.

Extend toward a real deployment

Replace synthetic incidents with a licensed, versioned corpus and an annotation guide. Group by the actual leakage boundary, such as conversation, customer, or source document, and use a time-based holdout when predicting future traffic. Deduplicate before splitting without letting evaluation labels guide cleaning. Retain split counts, uncertain labels, language and length slices, and an error review set. Compare this linear baseline against a stronger encoder only under the same frozen evaluation protocol.

Next, measure calibration and cost-sensitive abstention on development data, then evaluate the frozen policy on held-out data with uncertainty intervals at the correct independent unit. Run the local ASGI server under a load generator to measure latency, concurrency, memory, startup time, overload behavior, timeouts, and graceful drain. An in-process TestClient neither measures network latency nor verifies multi-worker deployment or cancellation under load.

Finally add authentication, authorization, TLS termination, request-size limits, rate limits, privacy-safe logs, release signing, dependency scanning, resource budgets, rollback, and monitored quality with delayed labels. This capstone does not implement those controls, a container, an online feature store, a registry, or an actual public deployment. Treat each as a separate testable requirement, not as a property conferred by having a /predict route.

Self-check

1. What is train/serve skew, and what property of a feature store prevents it?
2. Explain point-in-time correctness and give two offline tests that detect violations.
3. Your model's p50 latency is 30 ms and p99 is 800 ms. Why does the p99 matter more, and what would you check first?
4. Name the four drift types and say which ones you can detect without labels.
5. Why is shadow deployment insufficient to decide whether to ship a new model?
6. Give three ML-specific CI tests that a normal unit-test suite would not have.
7. A recommender's metrics improve every retraining cycle while user engagement falls. What is happening?

Where to go next

• Scikit-learn — pipelines that serialise cleanly into a serving artefact.
• Hugging Face ecosystem — the model side of an LLM deployment.
• The Inference Engineering Course — the serving layer in depth.