Deep Dive - MLflow

At 09:12 in this illustrative incident, hybrid-reranker-v3 answers a return-window question with "30 days" and no citation. Git commit 8b2f9e1 is unchanged, yet a prompt alias, retrieval index, provider model, or evaluator rubric may have moved. Checking the commit alone won't tell us which inputs this request actually used.

Ray showed where that work runs: tasks, actors, object refs, and recovery. MLflow records the evidence around it: experiments, parameters, metrics, artifacts, models, prompts, request traces, and evaluation results, all tied to stable identities. GPU scheduling and attention kernels stay elsewhere. Those links give a team something it can compare, reproduce, promote, debug, and audit.

Keep one policy assistant in view. It retrieves 40 candidates, reranks to 8, and answers from prompt support-answer version 17. Git commit 8b2f9e1 and dataset support-eval-2026-08-01 (short label eval-08-01) identify two more inputs. When a request about the return window cites nothing, its trace becomes case RET-104. Names, scores, and timings here are teaching fixtures, not measurements from a deployed assistant.

The local SDK lab below pins MLflow 3.15.2, released August 26, 2026. The source-reading path uses that release too. Documentation under /latest/ can move ahead of an installed server, so verify feature availability and client/server compatibility before adopting its examples.^{[1]Reference 1MLflow 3.15.2https://github.com/mlflow/mlflow/releases/tag/v3.15.2}

Linked identities, not just a Git commit

Two runs can share a source commit while using different data, parameters, environments, random seeds, or checkpoints. Replay only the commit and you can still get a different model. The original MLflow paper described open interfaces for tracking experiments, packaging reproducible projects, and representing models in several deployment flavors without forcing teams into one framework.^{[2]Reference 2Accelerating the Machine Learning Lifecycle with MLflow.https://www.usenix.org/conference/opml19/presentation/tut5}

That open-interface idea still anchors the project. The current tree extends classical experiment tracking into LLM and agent work. Each asset below answers a different reconstruction question:

These records earn their value through links. A trace can record the resolved prompt and model, a failure can become an evaluation case, and a candidate evaluation can point back to its model and code. Those links aren't automatic for arbitrary application code. Stable IDs also don't make every field immutable: tags, assessments, aliases, and some configuration can change. Require approval in your promotion workflow; the registry API itself won't refuse every unevaluated alias move.

Why isn't a Git commit enough to reproduce an LLM evaluation?

Answer

Code is only one input. Reproduction also needs model and prompt versions, data or trace cases, parameters, dependencies, evaluator definitions, and artifacts. MLflow records those changing inputs under linked identities.

Metadata and artifacts take different paths

Suppose every request adds a few searchable fields but a checkpoint adds gigabytes. Putting both in one store makes query traffic compete with large-byte transfers. MLflow separates searchable metadata from large bytes: the backend store holds experiment, run, model, prompt, trace, metric, parameter, and tag records, while the artifact store holds model weights, checkpoints, plots, tables, files, and archived trace payloads.^{[3]Reference 3MLflow Architecture Overviewhttps://mlflow.org/docs/latest/self-hosting/architecture/overview/}

That split leaves one routing question: who moves artifact bytes? The tracking server provides an HTTP boundary in front of both stores. Clients can send metadata to the server while either proxying artifact uploads through it or writing directly to object storage. Those modes have different credential and throughput consequences.

SQLite is the default local backend. A production tracking server normally uses a managed relational database plus object storage. The file backend is a legacy path in maintenance mode, and Model Registry requires a database-backed store. Shared workspaces, which group experiments, models, prompts, and artifacts for teams, are opt-in and also need SQL.^{[4]Reference 4Backend Storeshttps://mlflow.org/docs/latest/self-hosting/architecture/backend-store/[3]Reference 3MLflow Architecture Overviewhttps://mlflow.org/docs/latest/self-hosting/architecture/overview/}

Artifact proxying keeps storage credentials on the server, but it also widens the server's authority. Scope that role and enforce access checks on artifact requests; proxying isn't tenant isolation by itself. Direct uploads reduce proxy load, but every client now needs storage access. Existing experiments retain their artifact locations, so changing server flags doesn't automatically migrate their bytes or access paths.^{[3]Reference 3MLflow Architecture Overviewhttps://mlflow.org/docs/latest/self-hosting/architecture/overview/[5]Reference 5ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/}

A diagram of stores still doesn't tell you what one experiment recorded. To answer that, follow one run.

Runs are evidence containers

A run groups parameters, metrics, tags, artifacts, and source metadata. In our example, retrieval depth and rerank width are parameters, nDCG@8 is a metric, the commit and dataset revision are tags, and error_slices.json is an artifact. Each field has a different job when you compare or replay a run.

The MLflow API makes those categories explicit. This example uses a temporary SQLite database and local artifacts, so it needs no tracking server. The metric values are fixtures; reading them back verifies persistence, not retrieval quality. Install mlflow==3.15.2 in an isolated environment before running it:

Those calls prove that the API accepted records, not that the records describe a valid experiment. Without a dataset revision, a metric can be impossible to compare. Missing model signatures can let the wrong schema reach an artifact, while mutable environment dependencies can defeat reproduction. MLflow stores evidence; the team still has to decide which evidence a release requires.^{[5]Reference 5ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/}

MLflow's model abstraction packages files with a flavor-specific loader, environment metadata, and an optional input/output signature. A logged model has a model ID and can point back to a source run; that's distinct from a registered model's integer version. A registry version points to an artifact, while tags and aliases remain mutable. Approve the resolved artifact and its digest, then move an alias separately. A version number isn't protection against someone overwriting bytes in the artifact store.^{[6]Reference 6Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

The run can say nDCG@8 is 0.81. That aggregate still can't show which span of a live request went wrong, so we need request-level lineage.

LLM traces add request-level lineage

A trace represents one request as a tree of spans. Each span records a named operation, timing, status, attributes, and optional inputs or outputs.

MLflow Tracing is compatible with OpenTelemetry (OTel), an open telemetry standard. It adds LLM-oriented structures while preserving export and ingestion through OTel-compatible systems.^{[7]Reference 7LLM Tracing and Agent Observabilityhttps://mlflow.org/docs/latest/genai/tracing/[8]Reference 8OpenTelemetry GenAI Semantic Conventionshttps://github.com/open-telemetry/semantic-conventions-genai}

The policy assistant's illustrative trace has this shape. Retrieval reports DOC-12, but generation answers "30 days" with no citation. Before reading the timings, predict where you'd investigate first. The root's wall-clock duration is 1,420 ms:

The direct children sum to $145+110+1{,}130=1{,}385$ ms, leaving 35 ms of root work outside those children if they ran sequentially. Never add all nested durations: that double-counts parent time. Overlapping spans also need a timeline rather than a sum to explain the critical path.

Root metadata helps search the trace, while child spans explain it. Model spans can carry token counts and provider metadata. Retrieval spans can record document IDs and scores; tool spans can record arguments, results, and exceptions. Assessments attach feedback or scores. A recorded document ID is evidence of what the instrumentation observed, not proof that the chunk reached the final model context. Check context packing as well as retrieval.

Instrumentation can be manual with @mlflow.trace or span contexts, automatic through integration hooks, or ingested from OpenTelemetry. Automatic tracing is fast to adopt, but manual spans are still useful around business decisions that a library integration can't name.

A trace shows a correct retrieved document but an unsupported answer. Which boundary failed?

Answer

Inspect the retrieved content, final context packing, instructions, model output, and citation validation. A correct document in retrieval output doesn't prove it survived truncation or reached generation. The trace narrows the investigation; it doesn't establish the root cause by itself.

One failure trace doesn't define a release test by itself. Freeze its inputs and write an expected outcome before changing the candidate, so later comparisons can tell whether the fix helped.

Traces become evaluation data

Call that frozen request RET-104. The operations loop starts by capturing development or production traces, then selecting failures and representative successes. Add expectations, labels, or human feedback, such as "the answer must cite a retrieved doc," and store those cases in a versioned evaluation dataset (eval-08-01).

Now run each candidate application version against the same cases. Apply deterministic scorers and calibrated LLM judges, then inspect per-case failures alongside aggregate slices. Our release policy requires every critical case to pass, so a better average can't hide a broken RET-104. Keep separate representative holdout cases: repeatedly tuning against incident cases can overfit the regression suite.

Offline evaluation runs when a team calls mlflow.genai.evaluate with cases, a prediction function, and scorers. Use deterministic scorers for exact contracts such as JSON validity, citation existence, tool policy, or numerical tolerance. LLM judges help with relevance or tone when exact code can't express the rubric.

Automatic evaluation applies configured LLM judges asynchronously to sampled or filter-matched traces or conversations. It keeps judge calls off the request's synchronous path, but logging overhead and shared server load still exist. The documented server workflow supports LLM judges rather than code scorers, uses AI Gateway endpoints, considers recent traces within a one-hour window, and doesn't automatically retry failed evaluations. Confirm these version-sensitive limits for your deployment. Monitor the fraction of eligible cases actually scored, not just the average among successful assessments.^{[9]Reference 9Automatic Evaluationhttps://mlflow.org/docs/latest/genai/eval-monitor/automatic-evaluations/}

Judge output isn't ground truth. It changes with judge model, prompt, temperature, context, and rubric. Align judges against human labels, pin configurations, sample disagreements, and keep exact safety rules in code where possible.

Run the evidence loop locally

The downloadable SQLite SDK lab exercises native MLflow APIs with deterministic functions instead of provider calls. It reads back a run, artifact, and two-span trace; registers two prompt versions; moves an alias; then evaluates three response fixtures with mlflow.genai.evaluate and a custom code scorer.^{[10]Reference 10mlflow.genai Python APIhttps://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html}

Run the file with uv run mlflow_local_lab.py. Its dependency metadata pins Python 3.12 and MLflow 3.15.2. It disables MLflow usage telemetry and uses a temporary local database and artifact directory, which it removes after checking them. Package installation may need internet access; the lab itself needs neither credentials nor a server.

The three cases have no citation, an unknown citation DOC-99, and an allowed citation DOC-12. The scorer accepts only a nonempty list of allowed document IDs. Thus one of three cases passes, and our all-required-cases release policy blocks promotion. This checks citation membership, not whether “30 days” is supported by the document's text.

Expected lab summaries, excluding MLflow's generated run IDs and progress messages:

The fresh registry numbers these prompts 1 and 2; they play the roles of v17 and v18 in the incident. Setting cache_ttl_seconds=0 forces fresh alias resolution, so the lab tests alias movement and already-loaded objects, not elapsed-time cache expiry. We'll isolate expiry with a deterministic clock below.

Why should schema validity use a code scorer instead of an LLM judge?

Answer

Schema validity is deterministic and exactly computable. A code scorer is cheaper, repeatable, and easier to debug. Save judges for criteria that genuinely require semantic interpretation.

Instructions are one of the assets that evaluation will promote. They need the same versioning rules as models, which brings us to prompt management.

Prompt Registry makes instructions deployable assets

Prompt text behaves like code and configuration at once. A wording change can alter tool selection, output shape, latency, and safety. MLflow Prompt Registry stores templates with variables, optional model configuration, tags, and commit messages. A created template version is immutable, but its model configuration can be updated. Record the exact resolved configuration or its digest alongside the version; the prompt version alone can't reproduce a historical temperature setting.^{[11]Reference 11Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html}

Aliases provide mutable names such as candidate or production. Clients can load prompts:/support-answer@production while the alias points to a numbered version. Moving the alias changes subsequent resolution; it doesn't rewrite an already loaded object or hot-reload a worker.

That convenience includes a cache, so an alias move isn't instantly visible to every worker. Version-based loads (prompts:/support-answer/17) have an infinite default time-to-live (TTL); alias-based loads (prompts:/support-answer@production) default to 60 seconds. A process can keep serving v17 until it loads again after expiry. Because attached model configuration is mutable, an indefinitely cached version object can also retain old configuration. Choose an explicit refresh policy and record the resolved configuration used on each request.^{[11]Reference 11Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html}

The next example isolates cache behavior without calling MLflow. Its integer clock makes expiry reproducible. Versions reject replacement, aliases are scoped by prompt name, and cache entries are keyed by both name and alias. It models immutable templates only, not MLflow's mutable model configuration or concurrent registry writes.

Trace RET-104 must store version 17 if that's what the process used, even while the alias already names 18. The alias describes intended routing; the resolved version describes observed behavior. The toy records template text for inspection; a real trace should apply redaction or record a digest when the template contains sensitive material.

Model Registry uses a similar alias model. Fixed stages such as Staging and Production have been deprecated since MLflow 2.9.0 in favor of tags, aliases, and environment-specific registered models. Aliases separate a deployable name from an immutable model version, while CI/CD owns approval and environment promotion.^{[6]Reference 6Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

Provider calls are another mutable input. The gateway sees them, but it still isn't the source of application truth.

AI Gateway owns provider access, not application truth

MLflow also includes an OpenAI-compatible gateway surface for provider endpoints. Unified chat-completions live under /gateway/mlflow/v1, while passthrough routes expose a provider's native API. The gateway can centralize credentials, routing, fallbacks, rate limits, budgets, guardrails, and usage records.^{[12]Reference 12MLflow AI Gatewayhttps://mlflow.org/docs/latest/genai/governance/ai-gateway/}

The gateway can answer which provider served a request, how many tokens it used, and whether a routing rule applied. It can't decide whether DOC-12 was the right policy or whether a business action was authorized. Keep those decisions in application and evaluation contracts.

Use this distinction when assigning release ownership:

Linking these layers doesn't transfer responsibility. A trace exists because a request ran. An assessment exists because something scored it. A registry alias moved because a workflow changed it. Each event needs its own authorization and evidence.

Those links only hold up when the tracking server, object store, and exporters are operated like production systems.

Production architecture needs operations work

Database capacity and migrations

Every run metric, trace, span, prompt, alias, and assessment adds write and query load. Use a production database, monitor connection pools and slow queries, and rehearse schema upgrades. Back up before migrations. Large trace payloads may need archival into object storage while lightweight trace metadata remains searchable.^{[4]Reference 4Backend Storeshttps://mlflow.org/docs/latest/self-hosting/architecture/backend-store/}

Artifact storage and credentials

Model checkpoints and evaluation tables can dwarf metadata. Use lifecycle rules, encryption, checksums, and explicit retention. Decide whether clients upload directly or the server proxies them. Don't give a public tracking server a broad object-store role.

Trace privacy

LLM spans can contain prompts, outputs, retrieved documents, tool arguments, secrets, personal data, and source code. Redact before export, sample intentionally, apply access controls, and set retention by data class. Disabling input capture for one integration doesn't automatically sanitize custom span attributes.^{[13]Reference 13Production Tracinghttps://mlflow.org/docs/latest/genai/tracing/prod-tracing/[14]Reference 14Handling sensitive datahttps://opentelemetry.io/docs/security/handling-sensitive-data/}

Async loss and backpressure

Asynchronous tracing protects request latency, and OSS MLflow enables it by default outside Databricks notebooks. A process crash can still drop buffered spans. Current docs warn that a full export queue discards new traces and that failed exports are discarded after the retry window. Flush on graceful shutdown and monitor exporter failures. Then decide whether telemetry loss should fail open or fail closed: most applications should serve traffic while alerting on observability loss, while regulated workflows may choose differently.^{[13]Reference 13Production Tracinghttps://mlflow.org/docs/latest/genai/tracing/prod-tracing/}

For instrumentation-only images, MLflow offers the slimmer mlflow-tracing distribution. Package composition is version-sensitive: the published mlflow==3.15.2 package itself depends on matching mlflow-skinny==3.15.2 and mlflow-tracing==3.15.2. Let the resolver install that supported set; don't independently mix mismatched releases based on older packaging advice.^{[15]Reference 15MLflow 3.15.2 Distribution Metadatahttps://pypi.org/pypi/mlflow/3.15.2/json}

Identity and tenancy

A shared tracking server contains proprietary artifacts and request data. Put authentication, TLS, workspace or experiment permissions, and object-store policies into the threat model. Registry write access is deployment power because moving an alias can change what future clients load.

Strengths and weaknesses

Read each row as a tradeoff, not a feature tally:

MLflow isn't a general distributed scheduler like Ray or Kubernetes. It offers deployment integrations and agent-serving features, but those don't replace the underlying resource scheduler, recovery protocol, or autoscaling system. Identify which component actually owns a GPU allocation or retry before treating a logged event as execution control.

It also doesn't replace a full observability platform. Infrastructure metrics, logs, profiles, and distributed application traces may live elsewhere. OpenTelemetry compatibility helps correlate them, but teams still need consistent trace and deployment identifiers across systems. If a GPU is saturated, MLflow can link the request to its model and release; a GPU profiler still has to explain the kernel.

Project identity

MLflow was created at Databricks and announced in 2018. The original paper's authors included Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Andy Konwinski, and other Databricks engineers.^{[2]Reference 2Accelerating the Machine Learning Lifecycle with MLflow.https://www.usenix.org/conference/opml19/presentation/tut5} It focused on experimentation, reproducibility, and deployment through open interfaces rather than a locked framework.

Databricks contributed MLflow to the Linux Foundation on June 25, 2020, giving the project a vendor-neutral governance home.^{[16]Reference 16The MLflow Project Joins Linux Foundationhttps://www.linuxfoundation.org/press/press-release/the-mlflow-project-joins-linux-foundation} Databricks remains a major contributor and offers managed MLflow integrations, while the public project can be self-hosted and accepts community contributions.^{[17]Reference 17MLflow: Open Source AI Engineering Platformhttps://github.com/mlflow/mlflow[18]Reference 18MLflow on Databrickshttps://docs.databricks.com/gcp/en/mlflow/}

The foundational paper explains the original design, not every modern API. Tracing also builds on OpenTelemetry specifications. For evaluation, prompt registry, gateway, and model-lineage behavior, use the documentation and source matching your release.

A code-reading path through MLflow

The public repository makes architectural boundaries visible. Trace one request from client call to store and span, then pin the commit because GenAI code moves quickly:

1. Start at mlflow/tracking/client.py to see the stable MlflowClient facade.
2. Follow a log call into mlflow/store/tracking/ and compare REST, file, and SQLAlchemy stores.
3. Follow an artifact call into mlflow/store/artifact/ to see why large bytes use separate repositories.
4. Open mlflow/server/handlers.py and trace one REST route to a store operation.
5. Read mlflow/entities/logged_model.py for model identity and lineage.
6. Follow mlflow/tracing/provider.py and mlflow/tracing/client.py from span creation to persistence.
7. Inspect mlflow/genai/ for evaluation datasets, scorers, and prompts; follow gateway-specific handling into mlflow/gateway/.

Use tag v3.15.2 for this reading exercise instead of moving master. Start with MlflowClient.log_param, follow the tracking client to SqlAlchemyStore.log_param, and find the database constraint that rejects changing an existing parameter value. Then compare it with mutable run tags and prompt aliases. Public concepts are more stable than the implementation paths around them.^{[1]Reference 1MLflow 3.15.2https://github.com/mlflow/mlflow/releases/tag/v3.15.2}

Test that distinction before reading deeper. The local lab tries to overwrite retrieval_k=40 with 50, then changes a run tag and a prompt version's temperature. The parameter update must fail; the tag and configuration updates must succeed. A useful source-reading explanation identifies where each rule is enforced and which historical values remain recoverable. Naming three API methods without explaining their different mutability rules misses the important boundary.

A prompt version's template stayed unchanged, but its temperature changed from 0.2 to 0.8. Can an old evaluation receipt containing only the version number reproduce the earlier request?

Answer

No. The template version doesn't identify the historical mutable configuration. Retain the exact configuration used, or its digest plus retrievable content. Resolve configuration deliberately when loading prompts; an already-loaded object may still contain the earlier value.

Designing an evidence contract

Start an MLflow deployment with required fields and release receipts, not with a dashboard. For the policy assistant, require:

• Application release, source commit (8b2f9e1), environment, and dependency lock.
• Resolved model provider and exact model identifier.
• Resolved prompt version alongside its alias (support-answer@production and v17 or v18), plus the model configuration actually used.
• Retrieval index and corpus revision (eval-08-01 isn't a substitute for the live index hash).
• Root trace ID plus tool, retrieval, and model spans.
• Token, latency, failure, and cost attributes with known units.
• Evaluation dataset revision and scorer versions.
• Per-case assessments plus aggregate slices, including RET-104.
• Approval identity, candidate version, and digests of the evaluated artifact and configuration.
• Alias movement event and rollback target.

That receipt lets a team answer four different questions: what ran, what it observed, how it scored, and why it was promoted. Without those distinctions, one green dashboard can hide missing cases, stale aliases, or dropped traces.