Experiment Tracking with MLflow and W&B

A live failure anchors this lesson: deploy-answerer-v1.1-regression served two deploy-approval claims that ReleaseOps records didn't support. Monitoring created a proposed fix, deploy-answerer-v1.2-fix, but correctly left its status as BLOCKED_PENDING_EVALUATION.

Monitoring exported the fields this experiment needs: incident policy, failing template, exemplar request, evidence version, candidate template, and required safety metric. The experiment can test the repair without copying raw production traces into its dataset.

Now you need to answer a different question: did the proposed fix earn a controlled next step? Experiment tracking records the hypothesis, frozen test inputs, changed artifact, measured outputs, and decision so that someone else can reproduce that answer.

For a large language model (LLM) application, an experiment isn't limited to training new weights. A run can evaluate a changed prompt template, evidence policy, retriever, tool schema, model, or fine-tuned checkpoint. This lesson uses a prompt-and-policy fix because it's the immediate continuation of the production incident.

From alert to experiment

The monitoring page established what happened in production. It didn't test any fix. Keep that boundary clear:

The objects in an experiment record are straightforward once they're tied to this failure.

Why isn't deploy-answerer-v1.2-fix already approved when it was created from a clear incident?

Answer

An incident identifies a failing behavior, not whether a proposed repair works. The candidate must run on fixed regression and holdout cases, satisfy defined metrics, and leave an evidence record before promotion.

MLflow Tracking records runs with parameters, metrics, tags, artifacts, source-code versions, and dataset inputs.^{[1]Reference 1ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/} Weights & Biases (W&B) uses runs with configuration, metrics, and artifacts for the same evidence workflow.^{[2]Reference 2Experiments overview.https://docs.wandb.ai/models/track} Start with a local version of that workflow so the logic is visible without credentials or a hosted service.

Freeze the question before running candidates

Changing both the candidate and the test cases at the same time makes a win hard to interpret. Start from the handoff that monitoring produced, then create a small regression suite from known behavior.

The production exemplar contains no raw log here. The experiment needs stable evidence categories and expected routes, not durable copies of log text.

The first case checks the incident: no ReleaseOps record supports an approval claim, so the application must abstain. The second case prevents the blunt "fix" of always refusing to answer when a trustworthy release record is present.

Fingerprint the evaluation inputs

A run id isn't enough to reproduce a decision. If someone quietly changes a label, adds a case, or updates the evidence snapshot, the comparison is a new experiment condition.

A fingerprint is a deterministic checksum of the evaluation inputs. Equal fingerprints mean two runs were measured against the same frozen representation. They don't prove the labels are correct; review still has to establish that.

The row order isn't meaningful, so the fingerprint normalizes order. The example deliberately changes the known-good freeze-status case from SERVE to ABSTAIN; that mistake must create a different checksum. In a real project, also record dataset version, evidence snapshot version, labeling policy, and the code commit that built the cases.

A regression pass is necessary, not sufficient

Suppose the first fix rejects unsupported approval claims and serves explicit ReleaseOps records. It passes the two known cases. That doesn't show how it behaves when a rollback runbook provides a safe next step through a different evidence shape.

Add representative holdout cases before selecting a candidate. Keep them separate from the original incident suite so reviewers can tell whether a candidate repaired the known failure or generalized beyond it.

Notice the second guardrail. An application that abstains on every answer is factually safe on this metric, but it isn't useful. unsafe_claim_served protects trust; unnecessary_abstention protects the product from an overly restrictive repair.

Evaluate the behavior, not the proposed story

The next cell models three template versions. These functions stand in for deterministic offline evaluation results so you can focus on run comparison:

• v1.1-regressed serves every claimed approval, including unsupported ones.
• v1.2-fix repairs the incident but recognizes only explicit release records.
• v1.2-fix-b handles both release records and rollback-runbook evidence.

The candidate implementation doesn't get to read expected_route. It makes a rule from evidence_state, and the evaluator compares that output with the frozen expectation.

A clean regression result is progress, not approval. It means the candidate deserves evaluation on the holdout contract that was already declared.

Tracking isn't a scoreboard for flattering metrics. The first fix satisfies the incident's safety condition but introduces a holdout experience regression. Its run should remain visible and rejected, not silently replaced by the revised fix.

Two runs use the same model and prompt but different evaluation-set fingerprints. Are their aggregate scores directly comparable?

Answer

Not without accounting for the dataset change. The fingerprint identifies which cases were scored, so a score movement may come from changed inputs rather than changed model behavior.

Store one evidence record per run

Now put the experiment structure around those results. A tracking service should let a reviewer answer:

1. What changed?
2. What exactly was evaluated?
3. Which metrics and guardrails moved?
4. Which evidence artifacts explain the decision?
5. Which incident or hypothesis motivated the run?

The local tracker below uses the same conceptual buckets as MLflow and W&B. It deliberately logs stable identifiers and a redacted exemplar report, not raw logs.

All three runs point to the same regression, holdout, and combined evaluation checksums plus policy, evidence snapshot, and evaluator version. Each run also preserves its redacted report and versioned template artifact. That makes their metric comparison meaningful. If a new test case or scoring rule is needed tomorrow, update the corresponding fingerprint or evaluator version and label subsequent runs as a new comparison set.

Before ranking runs, audit that they are genuinely comparable. A dashboard can display results with different evaluation inputs or scoring logic side by side, but the reviewer must refuse to call that a controlled comparison.

A saturated suite stops discriminating

There's a subtler failure than a changed suite: a suite that no longer separates candidates. When every recent candidate scores exact_route_rate == 1.0 on frozen cases, metric can't rank them. Keep those cases as regression guards, but don't use their ceiling score to choose among candidates. Add harder cases that expose current differences when ranking requires more signal.

That rotation is a deliberate, versioned change, not a quiet edit. Adding harder cases creates a new suite fingerprint, so runs before and after the rotation belong to different comparison sets. Record the reason for the rotation next to the new fingerprint so a later reviewer can see that coverage got stricter on purpose.

The second row matters whenever the evaluator is itself an LLM judge rather than deterministic code. The judge's instruction text is an input to the score. If that prompt drifts without a version bump, two runs can look different only because the judge was asked a slightly different question. Pin the judge prompt inside evaluator_version (for example, claim-route-eval-v2@judge-prompt-3) so an unversioned judge edit can't silently redefine the baseline.

Every candidate now scores 100% on the regression suite. What should happen before using it to rank the next change?

Answer

Add reviewed cases from recent failures or harder slices while preserving existing regressions. A saturated suite can still block known bugs, but it no longer discriminates among candidates.

How this maps to MLflow and W&B

The local tracker isn't a substitute for a shared tracking service. It exposes the record design first. Once that design is correct, the platform mappings are small.

Those APIs are documented by their respective tracking, artifact, and run-tag guides.^{[1]Reference 1ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/[2]Reference 2Experiments overview.https://docs.wandb.ai/models/track[3]Reference 3Artifacts overview.https://docs.wandb.ai/models/artifacts[4]Reference 4Organize runs with tags.https://docs.wandb.ai/models/runs/tags} Pin the frozen suite as a dataset input alongside its SHA string: MLflow's current shape is mlflow.log_input(dataset, context="evaluation") after building an mlflow.data dataset (or equivalent) from the suite file; W&B's matching habit is to run.use_artifact(...) a versioned evaluation artifact so later reviews can open the exact input rows alongside the fingerprint. A real integration would require installing the chosen SDK and configuring storage and authentication, so these snippets are intentionally not executed here:

The W&B Weave path for LLM and agent runs

W&B splits its product: W&B Models tracks classic training runs, while Weave is the toolkit aimed at LLM and agent applications. Weave records execution traces rather than only scalar metrics.^{[5]Reference 5Weave: Tracing and Evaluations for LLM Applicationshttps://github.com/wandb/weave} It fits this lesson because a prompt-and-policy candidate is judged by what it did on each case, not by a single loss curve.

Two Weave pieces map onto the record design you just built. The @weave.op decorator marks a function as a tracked operation: calling it logs inputs, outputs, latency, and parent-child relationships. After weave.init(...), supported provider integrations capture nested model calls too.^{[5]Reference 5Weave: Tracing and Evaluations for LLM Applicationshttps://github.com/wandb/weave} For multi-turn agents, Weave groups per-turn traces into threads; nested model, retrieval, and tool calls form call trees similar to OpenTelemetry spans.^{[5]Reference 5Weave: Tracing and Evaluations for LLM Applicationshttps://github.com/wandb/weave} The weave.Evaluation class ties a frozen dataset, candidate function, and scorers into an aggregate report. This integration needs SDK and W&B account, so snippet is illustrative rather than executed here:

The discipline doesn't change with the tool. Weave gives you traces and an evaluation harness, but you still choose the frozen dataset, define the guardrail scorers, and refuse to rank runs whose inputs or scorers drifted.

Choose a platform because it fits your team's storage, access, dashboards, and deployment workflow. Don't confuse tool selection with experimental rigor. Either platform can preserve an under-specified or misleading run if you don't define inputs and guardrails first.

Don't delete rejected experiments. Their failure reason is part of the evidence. A future reviewer should see that the first repair eliminated unsafe claims but failed a usefulness guardrail.

Turn passing metrics into a limited decision

Even a passing offline run isn't authority to deploy broadly. This lab's cases are synthetic and tiny. A correct decision says exactly what the evidence supports: the revised candidate may enter canary review, with a known-good pointer retained for canary abort (and later production rollback only if the candidate was already promoted), plus continued monitoring of the same safety invariant.

For this prompt-only change, the registered object in your application's artifact registry is a template artifact. Don't pretend that every LLM application change produces a new model. If a later experiment fine-tunes model weights, MLflow Model Registry can link the registered model version to its source run and use aliases to identify a deployment target.^{[6]Reference 6Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

The fixture has one eligible run. A real review must resolve ties explicitly rather than silently selecting the first passing candidate. The artifact, source run, fixed suite, known-good pointer, and limitation now travel together. If canary observability later reports another unsupported deploy-approval claim, the investigator can trace back to this exact decision rather than guessing which "fix" shipped.

Promotion evidence should also define the stop action before canary traffic begins. This canary check doesn't claim the revised candidate fails. It makes the abort rule executable for any future unsafe serve while the candidate is only on the canary alias. Production rollback is a different action: it applies only after production has already moved to the candidate. The full alias-state treatment (abort a canary vs roll back a promotion) is in Model Versioning & Continuous Deployment.

The same record applies to compiled prompt programs

This lab changed one template by hand because that's what the incident demanded. The same experiment record can compare prompt programs compiled by an optimizer:

The next lesson uses DSPy to search instructions and demonstrations. Without this run record, a compiled program can look better because it saw the holdout set, changed evaluator semantics, or dropped a safety slice. Keep optimizer configuration, dataset splits, metric version, compiled state, and rejected candidates together.