Experiment Tracking with MLflow and W&B

In the synthetic incident from LLM Observability & Monitoring, a supplied outcome label marks req_205 as an unsupported deploy approval. Its trace locates the observations; it doesn't independently establish the claim's truth or whether a repair works. All cases and candidate behaviors here are hand-authored teaching fixtures.

deploy-answerer-v1.2-fix stays BLOCKED_PENDING_EVALUATION. The handoff carries the incident policy, failing template, exemplar request, evidence version, candidate template, and required safety metric. It carries no raw production trace into the eval set. The narrower question is did deploy-answerer-v1.2-fix earn a controlled next step?

For a large language model (LLM) application, an experiment can evaluate a changed prompt template, evidence policy, retriever, tool schema, model, or fine-tuned checkpoint, even when no weights change. Follow this simulated routing repair from frozen cases to recorded evidence. It ends with a review nomination; fresh factual and operational evaluation would still be required before a canary.

Track the whole candidate, not just its score

Training curves and checkpoints remain useful. Traditional ML tracking also records datasets, preprocessing, code, evaluation, and deployment evidence; stochastic training isn't unique to LLMs. The same evidence discipline applies when a candidate changes prompts or retrieval instead of weights.^{[1]Reference 1ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/}

This fixture explores a routing repair. A real incident could instead involve model weights, stale data, infrastructure, or several interacting faults. Don't infer the mechanism merely because it involves a prompt.

A quick prompt edit might fix req_205 while breaking supported rollback guidance. Record the incident as a testable hypothesis, then compare the proposed repair under fixed conditions.

Turn an incident into a question

Separate the questions. Observability locates a reviewed failure, tracking records a controlled comparison, and deployment control decides whether an artifact may receive traffic:

Now name the objects that carry this question. Each points back to req_205, but each gives a reviewer a different handle:

The experiment ends with a review record, not an automatic traffic change. Both passing and failing runs remain available:

Once the question is explicit, tracker choice becomes an implementation detail. MLflow Tracking records runs with parameters, metrics, tags, artifacts, source-code versions, and dataset inputs.^{[1]Reference 1ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/} Weights & Biases (W&B) uses runs with configuration, metrics, and artifacts for the same evidence workflow.^{[2]Reference 2Experiments overview.https://docs.wandb.ai/models/track} Start with a local version of that workflow so the logic is visible without credentials or a hosted service. Freeze the inputs before opening either service, so every later run answers the same question.

Freeze what the run is allowed to answer

A candidate and a test set can each move. If both move together, a better score has no clear cause. Hold the incident contract still first: turn monitoring's handoff into a small regression suite from known behavior.

req_205 contributes a simplified category and expected route, not production text. no_approval_support means the available material does not support an approval. It doesn't mean the previous lesson's versioned deploy policy disappeared. Categories keep this simulation small; they cannot reproduce a real model response without the actual question and admitted evidence.

Before running a candidate, predict the two labels: the incident row must abstain; the known-good row must serve. The contract makes those expectations explicit:

The incident row requires withholding an unsupported approval; the known-good row requires serving supported freeze-status information. These route expectations are supplied labels. Give the exact fixture representation an identity that travels with the run.

Hash the evaluation contract

A run ID identifies an execution, not what that execution saw. Change a label, add a case, or update the evidence snapshot, and the comparison becomes a new experiment condition.

A fingerprint is a deterministic checksum of those evaluation inputs. A matching cryptographic digest is strong evidence that the hashed representation matches, assuming the run actually used those inputs. It doesn't establish label quality, preserve missing files, or prove what an untrusted process executed.

Sorting by case_id makes row order irrelevant. The example then changes the known-good freeze-status case from SERVE to ABSTAIN, so the checksum changes too.

The lab stores the full SHA-256 digest and prints a 16-hex prefix to keep output readable. It rejects duplicate case IDs so sorting can't hide ambiguous row identities. Also record the dataset version, evidence snapshot, labeling policy, and code commit that built the cases.

This digest covers the categorical rows above. It does not hash real question text, retrieved documents, or model inputs, which the simulation does not contain. A real evaluation should hash or immutably identify those actual inputs and its labeling policy. MLflow dataset digest formats depend on the dataset implementation; logging dataset metadata is not the same as archiving its rows.^{[3]Reference 3mlflow.datahttps://mlflow.org/docs/latest/api_reference/python_api/mlflow.data.html}

Evaluation hygiene: preventing benchmark leakage

An engineer inspects an incident, proposes a repair, and sees that case pass. That is useful regression evidence. It is not an independent test of generalization, and inspecting the incident alone need not reveal the underlying cause.

If test feedback guides the candidate, the test has participated in development even without a gradient update. Adaptive selection using aggregate test scores can leak information too; the issue is not limited to reading individual questions.

One practical separation is:

1. Regression fixtures: Reviewed historical cases. Here req_205 is synthetic. Preserve required behavior under the current policy; if the policy or a label is corrected, review and version that change rather than quietly dropping a failing case.
2. Development Exploration Slices: A broader set of queries where engineers experiment with prompt wording, few-shot exemplars, and tool parameters. Inspecting failures here is expected during iterative development.
3. Final holdout: A representative test kept separate from tuning and candidate selection. Define who can inspect it, when it is used, and how repeated release evaluations are governed.

When we use a holdout failure to motivate v1.2-fix-b, those cases become development feedback. Freeze the revised candidate before a fresh independent final test. Representative cases can come from authorized sampled traffic, reviewed task examples, or another justified sampling design; they do not have to be copied from live traffic.

Let the first pass expose a trade-off

The incident case is necessary, but it can reward overfitting. A candidate that abstains everywhere could drive unsafe_route_decisions to zero while breaking supported answers. Ask it about a different safe evidence shape before selecting it: a rollback runbook.

Add cases outside the incident suite before inspecting candidate results. Keep their origin visible: they test a different evidence shape, not another copy of the incident. Once you use their failures to revise a candidate, they become development data rather than an untouched final test.

Predict the useful behavior before scoring: serve rollback_runbook, abstain when the approval record is missing. Put those expectations in the holdout:

Two guardrails expose the routing trade-off: unsafe_route_decisions counts SERVE where the label requires ABSTAIN; unnecessary_abstention counts the reverse. Neither measures generated claim support. A model could choose the expected SERVE route and still invent an approval in its text. Actual payload review remains a separate requirement from the previous lesson.

Compare behavior on frozen cases

Three template versions make the trade-off visible. The code simulates their routing policies; it doesn't call an LLM or measure whether generated text is grounded. These deterministic offline evaluation results isolate the tracking problem:

• v1.1-regression returns SERVE for every category, including unsupported approvals.
• v1.2-fix repairs the incident but recognizes only explicit release records.
• v1.2-fix-b handles both release records and rollback-runbook evidence.

The implementation below reads only evidence_state, not expected_route. That avoids directly returning the label. It doesn't prevent a developer from tuning to known categories or cases, so a perfect development score still needs an independent final test.

A clean regression result earns a continuation, not approval. The candidate now goes through the holdout contract already declared.

The first fix satisfies the incident's expected route but abstains on supported rollback guidance. Keep its rejected run instead of replacing it with the revised result. The routing simulation hasn't established the factual safety of any generated answer.

We used that holdout failure to motivate v1.2-fix-b, so the same four cases are now development evidence for the revision. The variable name holdout_cases records their original role, not an enduring guarantee of independence. Before real traffic, freeze the revised candidate and evaluate it on a fresh, independently reviewed test set. Four synthetic routes are enough to demonstrate a tracker, not to establish a safe deployment.

Keep evidence attached to each run

Attach enough context to a score that another engineer can inspect its cause:

1. What changed?
2. What exactly was evaluated?
3. Which metrics and guardrails moved?
4. Which evidence artifacts explain the decision?
5. Which incident or hypothesis motivated the run?

The local tracker below keeps metadata in memory and writes JSON artifacts under a temporary directory. Its report contains case IDs, categories, expected routes, and predicted routes. It demonstrates the record shape; the SDK examples also use temporary local storage. A shared project needs approved persistent storage and retention controls.

The three records share regression, holdout, and combined checksums, policy, evidence snapshot, and evaluator version. Each report now exists and can be read back. The template file is explicitly a simulation manifest, not a real prompt bundle; shipping a prompt requires its actual text and configuration. raw_log_text_stored=false is a descriptive tag, not a redaction control.

For real LLM runs, also capture the resolved model revision, generation settings, dependencies, and repeated trial IDs. Same inputs don't guarantee identical sampled outputs. Log per-case outcomes, latency, and token usage so aggregate quality doesn't conceal a slower or more expensive candidate.

That comparison contract changes when inputs or scoring rules change. Update the corresponding fingerprint or evaluator version and label the new conditions. The next cell checks deliberately altered metadata; it doesn't simulate the new dataset or execute a different evaluator.

A dashboard can plot drifted runs alongside the original trio. For this controlled comparison, require matching inputs and evaluator semantics. Other valid study designs might compare a shared subset or account for changed sample composition; merely placing aggregate scores on the same chart does neither. These fields record claimed conditions, so also inspect the actual report and evaluated artifact.

When the suite stops separating candidates

A frozen suite can stop separating candidates. If every recent candidate scores exact_route_rate == 1.0, that metric can't rank them. The task may be solved on those cases, coverage may be too narrow, or scoring may be too coarse. Check the evaluator before adding reviewed ranking cases that expose meaningful differences.

Version any suite expansion and preserve valid regressions. Adding cases changes the fingerprint and comparison conditions. Record what coverage changed and why.

The second row matters when an LLM judges an answer using a scoring prompt instead of deterministic code. Its instruction text is part of the score. If that prompt drifts without a version bump, two runs can differ because the judge was asked a different question. Pin the judge prompt, model revision, and inference settings alongside evaluator_version, for example claim-route-eval-v2@judge-prompt-3. The separate LLM-as-a-Judge lesson examines how to validate those judges; this lab uses exact route comparisons.

Put the record into MLflow or W&B

The local tracker is a sketch, not a shared service. It makes the record design visible first. Now map each bucket to a platform with shared storage, access, and review:

The mapping follows the official tracking, artifact, and run-tag APIs.^{[1]Reference 1ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/[2]Reference 2Experiments overview.https://docs.wandb.ai/models/track[4]Reference 4Artifacts overview.https://docs.wandb.ai/models/artifacts[5]Reference 5Organize runs with tags.https://docs.wandb.ai/models/runs/tags} In MLflow, mlflow.log_input(dataset, context="evaluation") logs input metadata; separately archive the approved suite file or pin an immutable source. In an online W&B run, run.use_artifact(...) records consumption of a versioned input artifact. Prefer an exact version or digest over a movable latest alias for reproducible evaluation.

Parameters describe a run's chosen inputs; changing one midway makes the run ambiguous. MLflow rejects changing an already logged parameter value. Metrics can have a history indexed by step, which preserves a training curve rather than only its last value. Tags such as review_only help organize runs, but mutable tags aren't an approval system or a tamper-proof audit log.

Prompt Registry versus Model Registry

Choose the registry by what you are versioning. MLflow Model Registry manages registered MLflow model packages and their source lineage. A package can contain trained weights, a custom Python function, or a wrapper around inference code. It isn't restricted to large checkpoints or weight-training runs.^{[6]Reference 6Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/[7]Reference 7mlflow.pyfunc Python APIhttps://mlflow.org/docs/latest/api_reference/python_api/mlflow.pyfunc.html}

Prompt Registry versions instruction templates. MLflow accepts a string template or a list of chat messages, with placeholders such as {{context}}, and optional model configuration. It doesn't require a JSON/YAML bundle containing every serving setting.^{[8]Reference 8Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html}

For this prompt-only repair, the prompt template is the natural registered object. A team could also package a larger application as an MLflow model. Whichever object it chooses, the release record must identify the evaluated artifact and its effective serving configuration.

What a prompt version actually pins

MLflow's prompt text is immutable within a version. Register edited text to create another version. However, model_config can be updated or deleted within the same version. Pinning prompts:/deploy-answerer/3 therefore doesn't freeze every inference setting. Versions can also be deleted, so a pin needs a retention policy.^{[8]Reference 8Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html[9]Reference 9mlflow.genai Python APIhttps://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html}

Consider a version evaluated at temperature 0.0. Someone updates its model configuration to 0.7. The URI still ends in /3, but the effective configuration has changed. Archive the approved configuration and its digest in the release record; verify what serving actually uses. A version pin alone cannot provide that check.

Aliases are mutable names that point to versions. A team might define @canary_candidate and @champion; the names don't themselves grant approval, route requests, or stop a canary. Its deployment controller must enforce those actions and restrict who can move the pointers.

What goes wrong when serving follows @latest

MLflow reserves @latest for the latest available version. A deployment that automatically consumes it may load a newly registered draft:

Saving a draft doesn't send traffic by itself. The risk arises when a serving process resolves @latest without a release gate. Workers can load different versions as they restart or refresh, and any mutable deployment alias can have that cache/fleet problem too.

Rollback remains possible: load an earlier retained version explicitly, or move an owned deployment alias back to it. Preserve the known-good version and configuration in advance, rather than reconstructing them during an incident. Use explicit pins or governed aliases for reviewed releases, then verify loaded versions across workers.

As of the reviewed MLflow 3.16.1 SDK, alias loads default to a 60-second cache lifetime; version loads default to indefinite caching. Choose cache behavior deliberately. The local check below demonstrates both alias rollback and mutable configuration; it doesn't validate a distributed rollout.^{[8]Reference 8Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html[10]Reference 10MLflow 3.16.1https://github.com/mlflow/mlflow/releases/tag/v3.16.1}

A tracking SDK records results; it doesn't automatically evaluate your candidate. MLflow's mlflow.genai.evaluate(...) runs a prediction function and scorers against curated data. It can use local code-based scorers or model-backed judges; model-backed calls may incur costs. Automatic evaluation of live traces is a different workflow from a frozen offline repair suite.^{[11]Reference 11Evaluation Quickstarthttps://mlflow.org/docs/latest/genai/eval-monitor/quickstart/[12]Reference 12Automatic Evaluationhttps://mlflow.org/docs/latest/genai/eval-monitor/automatic-evaluations/}

The next cell uses the real MLflow SDK against an explicitly local SQLite database and local artifact directory. It was checked with MLflow 3.16.1. Install that version and run it after the earlier lab cells in a fresh Python process. It reads back the run and report, then demonstrates the registry boundaries using placeholder templates. No tracking server, account, model call, or remote upload is involved.^{[13]Reference 13Backend Storeshttps://mlflow.org/docs/latest/self-hosting/architecture/backend-store/}

The temporary database is removed after this smoke check. For a shared project, configure an approved database and artifact store with access and retention controls. A FINISHED run means the logging context completed; its metrics may still fail your release policy.

When registering a real prompt, use the version returned by register_prompt. Hardcoding version=1 can point to an older template in an existing registry. This integration sketch requires your approved registry and an actually evaluated prompt string; it isn't executed by the lab and doesn't set an alias until you explicitly call the function after review:

W&B can also log without contacting its service. With mode="offline", the SDK saves a real local run that could be synchronized later; mode="disabled" instead makes logging calls no-ops. This cell uses offline mode, doesn't call wandb sync, and removes the temporary files afterward. It was checked with wandb==0.30.0.^{[14]Reference 14init()https://docs.wandb.ai/models/ref/python/functions/init}

This checks local SDK recording, not hosted artifact lineage, permissions, dashboards, or synchronization. Online use_artifact needs the approved evaluation artifact to exist in that service. Neither a local success nor a mock API call verifies those remote contracts.

Trace LLM behavior with W&B Weave

W&B Models supplies run and artifact tracking; Weave focuses on AI application tracing and evaluation. These are complementary views, and run tracking also applies to LLM experiments.^{[15]Reference 15Weave: Tracing and Evaluations for LLM Applicationshttps://github.com/wandb/weave} A per-case trace can help explain a result that an aggregate score conceals.

Two Weave pieces map onto the record you just built. The @weave.op decorator marks a function as a tracked operation. Once weave.init(...) initializes a project, calls record inputs, outputs, latency, and parent-child relationships; supported provider integrations capture nested model calls too. For multi-turn agents, Weave groups per-turn traces into threads; nested model, retrieval, and tool calls form call trees similar to OpenTelemetry spans.^{[15]Reference 15Weave: Tracing and Evaluations for LLM Applicationshttps://github.com/wandb/weave}

Follow one evaluation row

exact_route_rate = 1.0 means the categorical routes matched their labels. It establishes neither factual answer quality nor acceptable latency and cost. A real candidate could preserve task quality while increasing token usage; measure the trade-off on paired cases instead of inferring it from prompt length alone.

Imagine an instrumented request with a root span and three sequential child spans: retrieval, generation, and guardrail. Embedding, index query, and reranking may be children of retrieval. Sequential operations needn't be nested inside one another. A trace contains the operations actually instrumented; missing spans don't prove missing work.

If retrieval takes 420 ms and generation takes 1,850 ms in a hypothetical trace, generation is the larger observed component. Those timings don't establish provider overload or database contention. Investigate queueing, retries, provider timings, and resource evidence before naming the cause. Record client time to first visible chunk separately from generation duration, with units and start/end boundaries.

For a provider whose rates are dollars per million tokens, one estimate is:

Here ordinary and cached input counts are disjoint. Bind the rate card to the provider, model, date, and relevant billing tier. Add other billable categories when applicable, and avoid counting reasoning tokens twice if they are included in billed output. This estimate is not an invoice.

An evaluation row can link a case ID, parsed decision, reviewer outcome, approved input/artifact versions, timings, and usage to its trace. Raw requests, context, and completions may contain secrets or personal data. Capture them only under an approved policy, applying redaction before logging. A pointer or digest provides lineage but cannot reconstruct an artifact that was never retained. Preserve enough permitted evidence for review without assuming every platform automatically stores every field.

The Evaluation object is a blueprint: a dataset, candidate function, and scorers. Each .evaluate() call executes that comparison. Use a scorer argument named output to receive the candidate's result; additional arguments match dataset columns. These scorers return dictionaries. The Python SDK's evaluate() is async, so a script can use asyncio.run(...); in an already-running notebook event loop, use await.^{[16]Reference 16Scoring overviewhttps://docs.wandb.ai/weave/guides/evaluation/scorers[17]Reference 17Weave Python SDKhttps://docs.wandb.ai/weave/reference/python-sdk}

The snippet below defines a hosted integration but doesn't initialize a project or execute it. Calling run_hosted_evaluation() would send traced inputs and outputs to the configured W&B project. Review data retention and redact content before that boundary, not after it's logged:

This evaluation harness attaches scores to the operations that produced them. Keep dataset and scorer versions in the comparison record, and inspect failed rows alongside their traces.

Keep rejected runs in history

Choose a platform that fits the team's storage, access, dashboards, and deployment workflow. Either platform can record misleading metrics when their inputs and meanings aren't defined.

A rejected run explains why the next candidate exists. Keep the failed first fix visible so a reviewer can see why v1.2-fix-b was needed:

Don't trust a zero counter by itself. The next cell reads the categorical report, requires each canonical case exactly once, checks the declared candidate's deterministic route, and recomputes the metrics. Missing evidence or inconsistent numbers block nomination. This verifies the local simulation, not factual safety, authenticity of an external job, or permission to deploy.

Let the evidence earn a limited decision

Passing these development metrics nominates a candidate for review; it doesn't authorize traffic. A release review still needs fresh held-out evidence, representative slices, operational limits, and an owner. Retain a known-good pointer for canary abort and for later production rollback if the candidate is promoted.

For this prompt-only repair, canary_candidate can identify an evaluated prompt version after release review. Record the effective model configuration too. A Model Registry package may be appropriate when versioning inference code or trained weights along with the application; neither registry makes the deployment decision for you.^{[8]Reference 8Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html[6]Reference 6Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

Only run_003 passes this development contract. Record the proposed choice with its source run, evaluation SHA, known-good pointer, and limitation. This dataclass doesn't update a registry or deploy anything:

The fixture has one run that clears its development checks. A real review must resolve ties explicitly rather than silently selecting the first passing candidate. Artifact, source run, fixed suite, known-good pointer, and limitation now travel together. Mutable local records aren't a tamper-proof audit trail. A real release needs provenance from the actual evaluation execution and retained artifacts, so an investigator can identify which reviewed configuration shipped.

Promotion evidence should define the stop action before canary traffic begins. This check doesn't claim the revised candidate fails. It makes the abort rule executable for any future unsafe serve while the candidate is only on the canary alias. Production rollback is different: it applies only after production has moved to the candidate. The full alias-state treatment, including aborting a canary versus rolling back a promotion, is in Model Versioning & Continuous Deployment.

These two counters are hypothetical reviewed claim outcomes for a future canary, not the categorical routes measured above. A complete window with no known unsafe claim continues monitoring; one unsafe claim aborts and keeps the known-good artifact. Unknown review coverage pauses the canary instead of declaring a clean window:

Carry the record into compiled prompt programs

This lab simulated hand-written routing revisions. The same experiment record can compare actual prompt programs compiled by an optimizer:

Carry this record design into DSPy when searching instructions and demonstrations. A compiled program can look better because it saw the holdout set, changed evaluator semantics, or dropped a safety slice. Preserve optimizer configuration, dataset splits, metric version, compiled state, and rejected candidates together.