Deep Dive - MLflow

An LLM application changes even when its Python function doesn't. A prompt alias moves, a retrieval index refreshes, a provider model updates, or an evaluator changes its rubric. When quality drops, a Git commit alone can't tell you which combination served the bad response.

MLflow records that missing evidence. It connects experiments, parameters, metrics, artifacts, models, prompts, request traces, and evaluation results through stable identities. Its job isn't to schedule GPUs or execute attention kernels. Its job is to preserve enough lineage that a team can compare, reproduce, promote, debug, and audit what those systems produced.

Why MLflow exists

Machine-learning work has more moving inputs than ordinary application code. Two runs can use the same source commit but different data, parameters, environments, random seeds, or checkpoints. The original MLflow paper described open interfaces for tracking experiments, packaging reproducible projects, and representing models in several deployment flavors without forcing teams into one framework.^{[1]Reference 1Accelerating the Machine Learning Lifecycle with MLflow.https://people.eecs.berkeley.edu/~alig/papers/mlflow.pdf}

That open-interface idea still anchors the project. Modern MLflow extends classical experiment tracking into LLM and agent work:

The system becomes useful when those rows point to one another. A trace can reveal the prompt and model used. Production failures can become evaluation cases. A candidate's evaluation run can point back to its logged model and code. A deployment alias can move only after evidence passes.

Why isn't a Git commit enough to reproduce an LLM evaluation?

Answer

Code is only one input. Reproduction also needs model and prompt versions, data or trace cases, parameters, dependencies, evaluator definitions, and artifacts. MLflow records those changing inputs under linked identities.

Metadata and artifacts take different paths

MLflow separates searchable metadata from large bytes. The backend store holds experiment, run, model, prompt, trace, metric, parameter, and tag records. The artifact store holds model weights, checkpoints, plots, tables, files, and archived trace payloads.^{[2]Reference 2MLflow Architecture Overviewhttps://mlflow.org/docs/latest/self-hosting/architecture/overview/}

The tracking server provides an HTTP boundary in front of both. Clients can send metadata to the server while either proxying artifact uploads through it or writing directly to object storage. Those modes have different credential and throughput consequences.

SQLite works for a local start. A production tracking server normally uses a managed relational database plus object storage. The file backend is a legacy path, and Model Registry requires a database-backed store.^{[3]Reference 3Backend Storeshttps://mlflow.org/docs/latest/self-hosting/architecture/backend-store/}

Artifact proxying keeps storage credentials on the server, but it also widens the server's authority. A user who can send an allowed artifact request may act through the server's storage role. Direct uploads reduce proxy load, but every client now needs storage access. Choose one security boundary deliberately.

Runs are evidence containers

A run groups parameters, metrics, tags, artifacts, and source metadata. Parameters describe configuration such as learning rate or chunk size. Metrics record numeric observations over steps or time. Tags add searchable context. Artifacts hold larger files.

The simplest tracking code makes these categories explicit:

Logging doesn't prove correctness. Without a dataset revision, a metric can be impossible to compare. Missing model signatures can let the wrong schema reach an artifact, while mutable environment dependencies can defeat reproduction. MLflow stores evidence; the team must decide which evidence is required.

MLflow's model abstraction packages files with a flavor-specific loader, environment metadata, and an optional input/output signature. A logged model can point back to its source run. Model Registry then assigns registered names, immutable versions, tags, and aliases for promotion workflows.^{[4]Reference 4Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

LLM traces add request-level lineage

Aggregate metrics can say latency rose or quality fell. They can't show which tool call, retrieval result, or model response caused it. A trace represents one request as a tree of spans. Each span records a named operation, timing, status, attributes, and optional inputs or outputs.

MLflow Tracing is compatible with OpenTelemetry (OTel), an open telemetry standard. It adds LLM-oriented structures while preserving export and ingestion through OTel-compatible systems.^{[5]Reference 5LLM Tracing and Agent Observabilityhttps://mlflow.org/docs/latest/genai/tracing/[6]Reference 6Semantic conventions for generative AI systemshttps://opentelemetry.io/docs/specs/semconv/gen-ai/}

A document assistant might emit this tree:

Root metadata helps search the trace, while child spans explain it. Model spans can carry token counts and provider metadata. Retrieval spans can carry document IDs and scores, and tool spans can record arguments, result status, and exceptions. Assessments attach human feedback, code checks, or judge scores.

Instrumentation can be manual with @mlflow.trace or span contexts, automatic through integration hooks, or ingested from OpenTelemetry. Automatic tracing is fast to adopt, but manual spans are still useful around business decisions that a library integration can't name.

A trace shows a correct retrieved document but an unsupported answer. Which boundary failed?

Answer

Retrieval recall succeeded for that request. Inspect generation instructions, context packing, model output, and citation validation. Improving the vector index alone doesn't address evidence the generator already received.

Traces become evaluation data

The strongest LLM operations loop turns production evidence into repeatable tests:

1. Capture traces from development or production.
2. Select failures and representative successes.
3. Add expectations, labels, or human feedback.
4. Store them in a versioned evaluation dataset.
5. Run candidate application versions against the same cases.
6. Apply deterministic scorers and calibrated LLM judges.
7. Compare per-case failures and aggregate slices.
8. Promote only when required checks pass.

Offline evaluation runs when a team calls mlflow.genai.evaluate with cases, a prediction function, and scorers. Deterministic scorers should own exact contracts such as JSON validity, citation existence, tool policy, or numerical tolerance. LLM judges help with criteria such as relevance or tone when exact code can't express the rubric.

Automatic evaluation applies configured LLM judges asynchronously to sampled or filter-matched traces or conversations. It supports continuous quality signals without blocking request latency. Current MLflow docs also make its limits explicit: automatic evaluation supports LLM judges rather than code scorers, and sampling, filters, or judge and export failures can leave incomplete evidence. Monitor evaluation coverage and failures.^{[7]Reference 7Automatic Evaluationhttps://mlflow.org/docs/latest/genai/eval-monitor/automatic-evaluations/}

Judge output isn't ground truth. It changes with judge model, prompt, temperature, context, and rubric. Align judges against human labels, pin configurations, sample disagreements, and keep exact safety rules in code where possible.

Why should schema validity use a code scorer instead of an LLM judge?

Answer

Schema validity is deterministic and exactly computable. A code scorer is cheaper, repeatable, and easier to debug. Save judges for criteria that genuinely require semantic interpretation.

Prompt Registry makes instructions deployable assets

Prompt text behaves like code and configuration at once. A small wording change can alter tool selection, output shape, latency, and safety. MLflow Prompt Registry stores versioned prompt templates with variables, optional model configuration, tags, and commit messages. Template and version metadata are immutable, while model configuration can be updated for an existing version.^{[8]Reference 8Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html}

Aliases provide mutable names such as candidate or production. Clients can load prompts:/support-answer@production while the alias points to one immutable version. Moving the alias changes future loads without rewriting application code.

That convenience needs rollout discipline. Alias-based clients can cache prompt values. A process may serve an older version until its cache refreshes. Record resolved prompt version alongside alias string on every trace so requests remain attributable during rollout.

Model Registry uses a similar alias model. Fixed stages such as Staging and Production have been deprecated in favor of tags, aliases, and environment-specific registered models. Aliases separate a deployable name from an immutable model version, while CI/CD owns approval and environment promotion.^{[4]Reference 4Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

AI Gateway owns provider access, not application truth

MLflow also includes an OpenAI-compatible gateway surface for provider endpoints. It centralizes credentials, routing, fallbacks, rate limits, budgets, guardrails, and usage records.^{[9]Reference 9MLflow AI Gatewayhttps://mlflow.org/docs/latest/genai/governance/ai-gateway/}

The gateway can answer which provider served a request, how many tokens it used, and whether a routing rule applied. It can't decide whether the retrieved policy was correct or whether a business action was authorized. Keep those decisions in application and evaluation contracts.

This distinction prevents a common design mistake:

MLflow connects these layers, but connection isn't equivalence. A trace exists because a request ran. An assessment exists because something scored it. A registry alias moved because a workflow changed it. Each event needs its own authorization and evidence.

Production architecture needs operations work

Database capacity and migrations

Run metrics, traces, spans, prompts, aliases, and assessments create write and query load. Use a production database, monitor connection pools and slow queries, and rehearse schema upgrades. Back up before migrations. Large trace payloads may need archival into object storage while lightweight trace metadata remains searchable.^{[3]Reference 3Backend Storeshttps://mlflow.org/docs/latest/self-hosting/architecture/backend-store/}

Artifact storage and credentials

Model checkpoints and evaluation tables can dwarf metadata. Use lifecycle rules, encryption, checksums, and explicit retention. Decide whether clients upload directly or the server proxies them. Don't give a public tracking server a broad object-store role.

Trace privacy

LLM spans can contain prompts, outputs, retrieved documents, tool arguments, secrets, personal data, and source code. Redact before export, sample intentionally, apply access controls, and set retention by data class. Disabling input capture for one integration doesn't automatically sanitize custom span attributes.^{[10]Reference 10Production Tracinghttps://mlflow.org/docs/latest/genai/tracing/prod-tracing/[11]Reference 11Handling sensitive datahttps://opentelemetry.io/docs/security/handling-sensitive-data/}

Async loss and backpressure

Asynchronous tracing protects request latency, but a process crash can drop buffered spans. Flush on graceful shutdown, monitor exporter failures, and decide whether telemetry loss should fail open or fail closed. Most applications should serve traffic while alerting on observability loss; regulated workflows may choose differently.

Identity and tenancy

A shared tracking server contains proprietary artifacts and request data. Put authentication, TLS, workspace or experiment permissions, and object-store policies into the threat model. Registry write access is deployment power because moving an alias can change what future clients load.

Strengths and weaknesses

MLflow is broad, but it isn't an orchestrator like Ray, Airflow, or Kubernetes. It won't allocate a GPU, rerun a failed task graph, or autoscale model replicas by itself. It records and manages the assets and evidence around those systems.

It also doesn't replace a full observability platform. Infrastructure metrics, logs, profiles, and distributed application traces may live elsewhere. OpenTelemetry compatibility helps correlate them, but teams still need consistent trace and deployment identifiers across systems.

Project identity

MLflow was created at Databricks and announced in 2018. The original paper's authors included Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Andy Konwinski, and other Databricks engineers.^{[1]Reference 1Accelerating the Machine Learning Lifecycle with MLflow.https://people.eecs.berkeley.edu/~alig/papers/mlflow.pdf} It focused on experimentation, reproducibility, and deployment through open interfaces rather than a locked framework.

Databricks contributed MLflow to the Linux Foundation in 2020, giving the project a vendor-neutral governance home.^{[12]Reference 12The MLflow Project Joins Linux Foundationhttps://www.linuxfoundation.org/press/press-release/the-mlflow-project-joins-linux-foundation} Databricks remains a major contributor and offers managed MLflow integrations, while the public project can be self-hosted and accepts community contributions.^{[13]Reference 13MLflow: Open Source AI Engineering Platformhttps://github.com/mlflow/mlflow[14]Reference 14MLflow on Databrickshttps://docs.databricks.com/gcp/en/mlflow/}

MLflow has one foundational system paper rather than one paper for every modern feature. Current tracing builds on OpenTelemetry specifications. Evaluation, prompt registry, gateway, and model-lineage features are engineering systems whose primary descriptions live in project documentation and code.

A code-reading path through MLflow

The local clone makes architectural boundaries visible. Read one vertical request before scanning the full repository:

1. Start at mlflow/tracking/client.py to see the stable client facade.
2. Follow a log call into mlflow/store/tracking/ and compare REST, file, and SQLAlchemy stores.
3. Follow an artifact call into mlflow/store/artifact/ to see why large bytes use separate repositories.
4. Open mlflow/server/handlers.py and trace one REST route to a store operation.
5. Read mlflow/entities/logged_model.py for model identity and lineage.
6. Follow mlflow/tracing/provider.py and mlflow/tracing/client.py from span creation to persistence.
7. Inspect mlflow/genai/ for evaluation datasets, scorers, prompts, and gateway boundaries.

Pin a commit before following internal paths. MLflow's public concepts are stable, but its GenAI implementation changes quickly. The local workspace clone used for this reading path was behind upstream, so current claims were checked against official documentation rather than inferred from that checkout.

Designing an evidence contract

A useful MLflow installation begins with required fields and release receipts, not with a dashboard. For an LLM agent, require:

• Application release, source commit, environment, and dependency lock.
• Resolved model provider and exact model identifier.
• Resolved prompt version alongside its alias.
• Retrieval index and corpus revision.
• Root trace ID plus tool, retrieval, and model spans.
• Token, latency, failure, and cost attributes with known units.
• Evaluation dataset revision and scorer versions.
• Per-case assessments plus aggregate slices.
• Approval identity and immutable candidate model or prompt version.
• Alias movement event and rollback target.

That receipt lets a team answer four different questions: what ran, what it observed, how it scored, and why it was promoted. Without those distinctions, one green dashboard can hide missing cases, stale aliases, or dropped traces.

What to remember

• MLflow preserves evidence and lineage; it doesn't schedule distributed compute.
• Searchable metadata belongs in a backend store, while large artifacts belong in an artifact store.
• Runs, models, prompt versions, traces, datasets, and assessments need stable identities and explicit links.
• OpenTelemetry-compatible traces reveal request-level behavior across retrieval, tools, and model calls.
• Production traces become useful when selected failures turn into repeatable evaluation cases.
• Code scorers own exact contracts; calibrated judges support semantic criteria.
• Prompt and model aliases are mutable deployment pointers to immutable versions.
• Privacy, storage credentials, migrations, retention, and exporter loss are production design concerns.