Deep Dive - MLflow

At 09:12 in this illustrative incident, hybrid-reranker-v3 answers a return-window question with "30 days" and no citation. Git commit 8b2f9e1 is unchanged, yet a prompt alias, retrieval index, provider model, or evaluator rubric may have moved. Checking the commit alone won't tell us which inputs this request actually used.

Ray showed where that work runs. MLflow records the evidence around it, including request traces, under stable identities, so two executions that share a commit can still be compared. GPU scheduling and attention kernels stay elsewhere.

Keep one policy assistant in view. It retrieves 40 candidates, reranks to 8, and answers from prompt support-answer version 17. Git commit 8b2f9e1 and dataset support-eval-2026-08-01 (short label eval-08-01) identify two more inputs. When a request about the return window cites nothing, its trace becomes case RET-104. Names, scores, and timings here are teaching fixtures, not measurements from a deployed assistant.

The local SDK lab below pins MLflow 3.16.1, released September 17, 2026 and verified September 23. The source-reading path uses that release too. Documentation under /latest/ can move ahead of an installed server, so verify feature availability and client/server compatibility before adopting its examples.^{[1]Reference 1MLflow 3.16.1https://github.com/mlflow/mlflow/releases/tag/v3.16.1}

Link every artifact to its Git revision

Two runs can share a source commit while using different data, parameters, environments, random seeds, or checkpoints. Replay only the commit and you can still get a different model. The original MLflow paper described open interfaces for tracking experiments, packaging reproducible projects, and representing models in several deployment flavors without forcing teams into one framework.^{[2]Reference 2Accelerating the Machine Learning Lifecycle with MLflow.https://people.eecs.berkeley.edu/~alig/papers/mlflow.pdf}

That open-interface idea still anchors the project. The current tree extends classical experiment tracking into LLM and agent work. Each asset below answers a different reconstruction question:

These records earn their value through links. A trace can record the resolved prompt and model, a failure can become an evaluation case, and a candidate evaluation can point back to its model and code. Those links aren't automatic for arbitrary application code. Stable IDs also don't make every field immutable: tags, assessments, aliases, and some configuration can change. Require approval in your promotion workflow; the registry API itself won't refuse every unevaluated alias move.

Metadata and artifacts take different paths

Suppose every request adds a few searchable fields but a checkpoint adds gigabytes. Putting both in one store makes query traffic compete with large-byte transfers. MLflow separates searchable metadata from large bytes: the backend store holds experiment, run, model, prompt, trace, metric, parameter, and tag records, while the artifact store holds model weights, checkpoints, plots, tables, files, and archived trace payloads.^{[3]Reference 3MLflow Architecture Overviewhttps://mlflow.org/docs/latest/self-hosting/architecture/overview/}

That split leaves one routing question: who moves artifact bytes? The tracking server provides an HTTP boundary in front of both stores. Clients can send metadata to the server while either proxying artifact uploads through it or writing directly to object storage. Those modes have different credential and throughput consequences.

View larger

SQLite is the default local backend. A production tracking server normally uses a managed relational database plus object storage. The file backend is a legacy path in maintenance mode, and Model Registry requires a database-backed store. Shared workspaces, which group experiments, models, prompts, and artifacts for teams, are opt-in and also need SQL.^{[4]Reference 4Backend Storeshttps://mlflow.org/docs/latest/self-hosting/architecture/backend-store/[3]Reference 3MLflow Architecture Overviewhttps://mlflow.org/docs/latest/self-hosting/architecture/overview/}

Artifact proxying keeps storage credentials on the server, but it also widens the server's authority. Scope that role and enforce access checks on artifact requests; proxying isn't tenant isolation by itself. Direct uploads reduce proxy load, but every client now needs storage access. Existing experiments retain their artifact locations, so changing server flags doesn't automatically migrate their bytes or access paths.^{[3]Reference 3MLflow Architecture Overviewhttps://mlflow.org/docs/latest/self-hosting/architecture/overview/[5]Reference 5ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/}

View larger

A diagram of stores still doesn't tell you what one experiment recorded. To answer that, follow one run.

Runs are evidence containers

A run groups parameters, metrics, tags, artifacts, and source metadata. In our example, retrieval depth and rerank width are parameters, nDCG@8 is a metric, the commit and dataset revision are tags, and error_slices.json is an artifact. Each field has a different job when you compare or replay a run.

The MLflow API makes those categories explicit. This example uses a temporary SQLite database and local artifacts, so it needs no tracking server. The metric values are fixtures; reading them back verifies persistence, not retrieval quality. Install mlflow==3.16.1 in an isolated environment before running it:

import os
from pathlib import Path
from tempfile import TemporaryDirectory

os.environ["MLFLOW_DISABLE_TELEMETRY"] = "true"
import mlflow

with TemporaryDirectory() as directory:
    root = Path(directory)
    mlflow.set_tracking_uri(f"sqlite:///{root / 'tracking.db'}")
    experiment_id = mlflow.create_experiment(
        "retrieval-ranking", artifact_location=(root / "artifacts").as_uri()
    )
    with mlflow.start_run(experiment_id=experiment_id, run_name="hybrid-reranker-v3") as run:
        mlflow.log_params({"retrieval_k": 40, "rerank_k": 8})
        mlflow.log_metrics({"recall_at_40": 0.94, "ndcg_at_8": 0.81})
        mlflow.set_tags({"dataset_revision": "eval-08-01", "git_commit": "8b2f9e1"})
        mlflow.log_dict({"case": "RET-104", "failure": "missing citation"}, "error_slices.json")
    stored = mlflow.get_run(run.info.run_id)
    assert stored.data.params["retrieval_k"] == "40"
    assert stored.data.metrics["ndcg_at_8"] == 0.81
    print("read back: retrieval_k=40; ndcg_at_8=0.81")

read back: retrieval_k=40; ndcg_at_8=0.81

Those calls prove that the API accepted records, not that the records describe a valid experiment. Without a dataset revision, a metric can be impossible to compare. Missing model signatures can let the wrong schema reach an artifact, while mutable environment dependencies can defeat reproduction. MLflow stores evidence; the team still has to decide which evidence a release requires.^{[5]Reference 5ML Experiment Tracking.https://mlflow.org/docs/latest/ml/tracking/}

MLflow's model abstraction packages files with a flavor-specific loader, environment metadata, and an optional input/output signature. A logged model has a model ID and can point back to a source run; that's distinct from a registered model's integer version. A registry version points to an artifact, while tags and aliases remain mutable. Approve the resolved artifact and its digest, then move an alias separately. A version number isn't protection against someone overwriting bytes in the artifact store.^{[6]Reference 6Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

The run can say nDCG@8 is 0.81. That aggregate still can't show which span of a live request went wrong, so we need request-level lineage.

LLM traces add request-level lineage

A trace represents one request as a tree of spans. Each span records a named operation, timing, status, attributes, and optional inputs or outputs. A parent-child edge describes nesting. Since MLflow 3.16.0, separate span links can also connect related operations, such as a downstream trace or another request sharing a retrieval step. A link doesn't make that operation a nested child or justify adding its duration to the parent's time.^{[7]Reference 7MLflow 3.16.0 Releasehttps://github.com/mlflow/mlflow/releases/tag/v3.16.0}

MLflow Tracing is compatible with OpenTelemetry (OTel), an open telemetry standard. It adds LLM-oriented structures while preserving export and ingestion through OTel-compatible systems.^{[8]Reference 8LLM Tracing and Agent Observabilityhttps://mlflow.org/docs/latest/genai/tracing/[9]Reference 9OpenTelemetry GenAI Semantic Conventionshttps://github.com/open-telemetry/semantic-conventions-genai}

The policy assistant's illustrative trace has this shape. Retrieval reports DOC-12, but generation answers "30 days" with no citation. Before reading the timings, predict where you'd investigate first. The root's wall-clock duration is 1,420 ms:

answer_question                         1,420 ms
├── retrieve_policy_docs                 145 ms
│   ├── embed_query                       24 ms
│   └── vector_search                     91 ms
├── rerank                                110 ms
└── generate_answer                     1,130 ms
    ├── model_call                       980 ms
    └── validate_citations               105 ms

The direct children sum to $145+110+1{,}130=1{,}385$ ms, leaving 35 ms of root work outside those children if they ran sequentially. Never add all nested durations: that double-counts parent time. Overlapping spans also need a timeline rather than a sum to explain the critical path.

Root metadata helps search the trace, while child spans explain it. Model spans can carry token counts and provider metadata. Retrieval spans can record document IDs and scores; tool spans can record arguments, results, and exceptions. Assessments attach feedback or scores. A recorded document ID confirms instrumentation observed retrieval; inspect context packing to confirm what reached the final model context.

Instrumentation can be manual with @mlflow.trace or span contexts, automatic through integration hooks, or ingested from OpenTelemetry. Automatic tracing is fast to adopt, but manual spans are still useful around business decisions that a library integration can't name.

One failure trace doesn't define a release test by itself. Freeze its inputs and write an expected outcome before changing the candidate, so later comparisons can tell whether the fix helped.

Traces become evaluation data

Call that frozen request RET-104. The operations loop starts by capturing development or production traces, then selecting failures and representative successes. Add expectations, labels, or human feedback, such as "the answer must cite a retrieved doc," and store those cases in a versioned evaluation dataset (eval-08-01).

Here, eval-08-01 names a frozen comparison snapshot we retain. A native MLflow evaluation dataset has an ID, but its records and expectations can be updated through APIs such as merge_records; the ID alone doesn't identify historical contents. Retain the case inputs, expectations and a content digest used for each evaluation. The dataset feature requires a SQL backend; a list of dictionaries, as used in the local lab, is also valid evaluation input without creating a stored dataset.^{[10]Reference 10Building MLflow Evaluation Datasetshttps://mlflow.org/docs/latest/genai/datasets/}

Now run each candidate application version against the same cases. Apply deterministic scorers and calibrated LLM judges, then inspect per-case failures alongside aggregate slices. Our release policy requires every critical case to pass, so a better average can't hide a broken RET-104. Keep separate representative holdout cases: repeatedly tuning against incident cases can overfit the regression suite.

View larger

Offline evaluation runs when a team calls mlflow.genai.evaluate with cases, a prediction function, and scorers. Use deterministic scorers for exact contracts such as JSON validity, citation existence, tool policy, or numerical tolerance. LLM judges help with relevance or tone when exact code can't express the rubric.

Automatic evaluation applies configured LLM judges asynchronously to sampled or filter-matched traces or conversations. It keeps judge calls off the request's synchronous path, but logging overhead and shared server load still exist. The documented server workflow supports LLM judges rather than code scorers, uses AI Gateway endpoints, considers recent traces within a one-hour window, and doesn't automatically retry failed evaluations. Confirm these version-sensitive limits for your deployment. Track the share of eligible cases scored and their average assessment outcome; the average alone can hide low coverage.^{[11]Reference 11Automatic Evaluationhttps://mlflow.org/docs/latest/genai/eval-monitor/automatic-evaluations/}

Judge output isn't ground truth. It changes with judge model, prompt, temperature, context, and rubric. Align judges against human labels, pin configurations, sample disagreements, and keep exact safety rules in code where possible.

Run the evidence loop locally

The downloadable SQLite SDK lab exercises native MLflow APIs with deterministic functions instead of provider calls. It reads back a run, artifact, and two-span trace; registers two prompt versions; moves an alias; then evaluates three response fixtures with mlflow.genai.evaluate and a custom code scorer.^{[12]Reference 12mlflow.genai Python APIhttps://mlflow.org/docs/latest/api_reference/python_api/mlflow.genai.html}

Run the file with uv run mlflow_local_lab.py. Its dependency metadata pins Python 3.12 and MLflow 3.16.1. It disables MLflow usage telemetry and uses a temporary local database and artifact directory, which it removes after checking them. Package installation may need internet access; the lab itself needs neither credentials nor a server.

The three cases have no citation, an unknown citation DOC-99, and an allowed citation DOC-12. The scorer accepts only a nonempty list of allowed document IDs. Thus one of three cases passes, and our all-required-cases release policy blocks promotion. This checks citation membership, not whether “30 days” is supported by the document's text.

MLflow's result.passed summarizes scorer outcomes; it isn't a receipt that every required case and scorer ran. In 3.16.1, its implementation skips missing values without a scorer error and returns True when no result table is present. The lab separately checks the expected three rows and three Boolean scores. A production controller should join results to required case IDs and reject missing assessments before considering the scores.^{[13]Reference 13MLflow 3.16.1: GenAI EvaluationResult and Passing Criteriahttps://github.com/mlflow/mlflow/blob/v3.16.1/mlflow/genai/evaluation/entities.py}

Expected lab summaries, excluding MLflow's generated run IDs and progress messages:

tracking: run, metric, artifact, and two-span trace read back
prompts: old object=v1; fresh alias=v2
mutability: parameter fixed; tag and prompt model config changed
evaluation: missing=False; wrong=False; valid=True; mean=1/3
release: blocked under the all-required-cases policy
PASS: local SDK contracts; no server, provider, or judge calls

The fresh registry numbers these prompts 1 and 2. They play the roles of v17 and v18 in the incident. Setting cache_ttl_seconds=0 forces fresh alias resolution, so the lab tests alias movement and already-loaded objects, not elapsed-time cache expiry.

Evaluation can promote a new instruction the same way it promotes a model. Is a prompt version, by itself, enough to replay the request?

Prompt Registry makes instructions deployable assets

A prompt version is not enough by itself. Prompt text behaves like code and configuration at once. A wording change can alter tool selection, output shape, latency, and safety.

MLflow Prompt Registry stores templates with variables, optional model configuration, tags, and commit messages. A created template version is immutable, but its model configuration can be updated. Record the exact resolved configuration or its digest alongside the version. The prompt version alone can't reproduce a historical temperature setting.^{[14]Reference 14Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html}

Aliases provide mutable names such as candidate or production. Clients can load prompts:/support-answer@production while the alias points to a numbered version. Moving the alias changes subsequent resolution; it doesn't rewrite an already loaded object or hot-reload a worker.

That convenience includes a cache, so an alias move isn't instantly visible to every worker. Version-based loads (prompts:/support-answer/17) have an infinite default time-to-live (TTL); alias-based loads (prompts:/support-answer@production) default to 60 seconds. A process can keep serving v17 until it loads again after expiry. Because attached model configuration is mutable, an indefinitely cached version object can also retain old configuration. Choose an explicit refresh policy and record the resolved configuration used on each request.^{[14]Reference 14Prompt Registryhttps://mlflow.org/docs/latest/genai/prompt-registry/index.html}

The next example isolates cache behavior without calling MLflow. Its integer clock makes expiry reproducible. Versions reject replacement, aliases are scoped by prompt name, and cache entries are keyed by both name and alias. It models immutable templates only, not MLflow's mutable model configuration or concurrent registry writes.

from dataclasses import dataclass

@dataclass(frozen=True)
class PromptVersion:
    name: str
    version: int
    template: str

class PromptRegistry:
    def __init__(self) -> None:
        self.versions: dict[tuple[str, int], PromptVersion] = {}
        self.aliases: dict[tuple[str, str], tuple[str, int]] = {}

    def register(self, prompt: PromptVersion) -> None:
        key = (prompt.name, prompt.version)
        if key in self.versions:
            raise ValueError("version already exists")
        self.versions[key] = prompt

    def set_alias(self, alias: str, prompt: PromptVersion) -> None:
        key = (prompt.name, prompt.version)
        if key not in self.versions:
            raise KeyError(key)
        self.aliases[(prompt.name, alias)] = key

    def resolve(self, name: str, alias: str) -> PromptVersion:
        return self.versions[self.aliases[(name, alias)]]

class CachedClient:
    """Alias loads refresh after ttl_ticks. MLflow's default alias TTL is 60s."""

    def __init__(self, registry: PromptRegistry, ttl_ticks: int = 1) -> None:
        if type(ttl_ticks) is not int or ttl_ticks < 0:
            raise ValueError("TTL must be a nonnegative integer")
        self.registry = registry
        self.ttl_ticks = ttl_ticks
        self._cached: dict[tuple[str, str], tuple[PromptVersion, int]] = {}
        self._last_tick = -1

    def load(self, name: str, alias: str, tick: int) -> PromptVersion:
        if type(tick) is not int or tick < 0 or tick < self._last_tick:
            raise ValueError("clock must be nonnegative and monotonic")
        self._last_tick = tick
        key = (name, alias)
        cached = self._cached.get(key)
        if cached is None or tick >= cached[1]:
            cached = (self.registry.resolve(name, alias), tick + self.ttl_ticks)
            self._cached[key] = cached
        return cached[0]

def record_trace(alias: str, resolved: PromptVersion) -> dict[str, object]:
    return {
        "prompt_alias": alias,
        "prompt_name": resolved.name,
        "prompt_version": resolved.version,
        "template": resolved.template,
    }

v17 = PromptVersion("support-answer", 17, "Cite retrieved docs.")
v18 = PromptVersion("support-answer", 18, "Cite retrieved docs. Refuse if none.")
registry = PromptRegistry()
registry.register(v17)
registry.register(v18)
registry.set_alias("production", v17)

client = CachedClient(registry, ttl_ticks=1)
served = client.load("support-answer", "production", tick=0)
assert record_trace("production", served)["prompt_version"] == 17

registry.set_alias("production", v18)
stale = client.load("support-answer", "production", tick=0)
assert stale.version == 17

fresh = client.load("support-answer", "production", tick=1)
assert fresh.version == 18
print(f"cached={stale.version} then resolved={fresh.version}")

other = PromptVersion("summarize", 1, "Summarize without adding facts.")
registry.register(other)
registry.set_alias("production", other)
assert client.load("summarize", "production", tick=1) == other
try:
    registry.register(PromptVersion("support-answer", 17, "Silently changed."))
except ValueError:
    print("version overwrite rejected; another prompt's cache stays separate")
else:
    raise AssertionError("immutable template was replaced")

cached=17 then resolved=18
version overwrite rejected; another prompt's cache stays separate

Trace RET-104 must store version 17 if that's what the process used, even while the alias already names 18. The alias describes intended routing; the resolved version describes observed behavior. The toy records template text for inspection; a real trace should apply redaction or record a digest when the template contains sensitive material.

Model Registry uses a similar alias model. Fixed stages such as Staging and Production have been deprecated since MLflow 2.9.0 in favor of tags, aliases, and environment-specific registered models. Aliases separate a deployable name from an immutable model version, while CI/CD owns approval and environment promotion.^{[6]Reference 6Model Registry Workflows | MLflow AI Platformhttps://mlflow.org/docs/latest/ml/model-registry/workflow/}

Provider calls are another mutable input. The gateway sees them, but it still isn't the source of application truth.

AI Gateway owns provider access, not application truth

MLflow also includes an OpenAI-compatible gateway surface for provider endpoints. Unified chat-completions live under /gateway/mlflow/v1, while passthrough routes expose a provider's native API. The gateway can centralize credentials, routing, fallbacks, rate limits, budgets, guardrails, and usage records.^{[15]Reference 15MLflow AI Gatewayhttps://mlflow.org/docs/latest/genai/governance/ai-gateway/}

The gateway can answer which provider served a request, how many tokens it used, and whether a routing rule applied. It can't decide whether DOC-12 was the right policy or whether a business action was authorized. Keep those decisions in application and evaluation contracts.

Use this distinction when assigning release ownership:

Linking these layers doesn't transfer responsibility. A trace exists because a request ran. An assessment exists because something scored it. A registry alias moved because a workflow changed it. Each event needs its own authorization and evidence.

Those links only hold up when the tracking server, object store, and exporters are operated like production systems.

Production architecture needs operations work

Database capacity and migrations

Every run metric, trace, span, prompt, alias, and assessment adds write and query load. Use a production database, monitor connection pools and slow queries, and rehearse schema upgrades. Back up before migrations. Large trace payloads may need archival into object storage while lightweight trace metadata remains searchable.^{[4]Reference 4Backend Storeshttps://mlflow.org/docs/latest/self-hosting/architecture/backend-store/}

Artifact storage and credentials

Model checkpoints and evaluation tables can dwarf metadata. Use lifecycle rules, encryption, checksums, and explicit retention. Decide whether clients upload directly or the server proxies them. Don't give a public tracking server a broad object-store role.

Deleting a run is a soft deletion so it can be restored; it doesn't automatically remove its metadata or artifact bytes. MLflow provides mlflow gc for permanent cleanup of deleted runs. Retention work must account for that cleanup and for independently copied artifacts and backups, rather than treating a deleted UI row as erased evidence.^{[4]Reference 4Backend Storeshttps://mlflow.org/docs/latest/self-hosting/architecture/backend-store/}

Trace privacy

LLM spans can contain prompts, outputs, retrieved documents, tool arguments, secrets, personal data, and source code. Redact before export, sample intentionally, apply access controls, and set retention by data class. Disabling input capture for one integration doesn't automatically sanitize custom span attributes.^{[16]Reference 16Production Tracinghttps://mlflow.org/docs/latest/genai/tracing/prod-tracing/[17]Reference 17Handling sensitive datahttps://opentelemetry.io/docs/security/handling-sensitive-data/}

Async loss and backpressure

Asynchronous tracing protects request latency, and OSS MLflow enables it by default outside Databricks notebooks. A process crash can still drop buffered spans. Current docs warn that a full export queue discards new traces and that failed exports are discarded after the retry window. Flush on graceful shutdown and monitor exporter failures. Then decide whether telemetry loss should fail open or fail closed: most applications should serve traffic while alerting on observability loss, while regulated workflows may choose differently.^{[16]Reference 16Production Tracinghttps://mlflow.org/docs/latest/genai/tracing/prod-tracing/}

For instrumentation-only images, MLflow offers the slimmer mlflow-tracing distribution. Package composition is version-sensitive: the published mlflow==3.16.1 package itself depends on matching mlflow-skinny==3.16.1 and mlflow-tracing==3.16.1. Let the resolver install that supported set; don't independently mix mismatched releases based on older packaging advice.^{[18]Reference 18MLflow 3.16.1 Distribution Metadatahttps://pypi.org/pypi/mlflow/3.16.1/json}

Identity and tenancy

A shared tracking server contains proprietary artifacts and request data. Put authentication, TLS, workspace or experiment permissions, and object-store policies into the threat model. Registry write access is deployment power because moving an alias can change what future clients load. MLflow 3.16.0 made basic-auth authorization fail closed by default; 3.16.1 removed the shipped default admin password and fixed UI authorization for non-admin users. These changes apply to that configured authentication path, not to an assurance that every server has authentication enabled.^{[7]Reference 7MLflow 3.16.0 Releasehttps://github.com/mlflow/mlflow/releases/tag/v3.16.0[1]Reference 1MLflow 3.16.1https://github.com/mlflow/mlflow/releases/tag/v3.16.1}

Strengths and weaknesses

Read each row as a tradeoff, not a feature tally:

MLflow isn't a general distributed scheduler like Ray or Kubernetes. It offers deployment integrations and agent-serving features, but those don't replace the underlying resource scheduler, recovery protocol, or autoscaling system. Identify which component actually owns a GPU allocation or retry before treating a logged event as execution control.

It also doesn't replace a full observability platform. Infrastructure metrics, logs, profiles, and distributed application traces may live elsewhere. OpenTelemetry compatibility helps correlate them, but teams still need consistent trace and deployment identifiers across systems. If a GPU is saturated, MLflow can link the request to its model and release; a GPU profiler still has to explain the kernel.

Project identity

MLflow was created at Databricks and announced in 2018. The original paper's authors included Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Andy Konwinski, and other Databricks engineers.^{[2]Reference 2Accelerating the Machine Learning Lifecycle with MLflow.https://people.eecs.berkeley.edu/~alig/papers/mlflow.pdf} It focused on experimentation, reproducibility, and deployment through open interfaces rather than a locked framework.

Databricks contributed MLflow to the Linux Foundation on June 25, 2020, giving the project a vendor-neutral governance home.^{[19]Reference 19The MLflow Project Joins Linux Foundationhttps://www.linuxfoundation.org/press/press-release/the-mlflow-project-joins-linux-foundation} Databricks remains a major contributor and offers managed MLflow integrations, while the public project can be self-hosted and accepts community contributions.^{[20]Reference 20MLflow: Open Source AI Engineering Platformhttps://github.com/mlflow/mlflow[21]Reference 21MLflow on Databrickshttps://docs.databricks.com/gcp/en/mlflow/}

The foundational paper explains the original design, not every modern API. Tracing also builds on OpenTelemetry specifications. For evaluation, prompt registry, gateway, and model-lineage behavior, use the documentation and source matching your release.

A code-reading path through MLflow

The public repository makes architectural boundaries visible. Trace one request from client call to store and span, then pin the commit because GenAI code moves quickly:

1. Start at mlflow/tracking/client.py to see the stable MlflowClient facade.
2. Follow a log call into mlflow/store/tracking/ and compare REST, file, and SQLAlchemy stores.
3. Follow an artifact call into mlflow/store/artifact/ to see why large bytes use separate repositories.
4. Open mlflow/server/handlers.py and trace one REST route to a store operation.
5. Read mlflow/entities/logged_model.py for model identity and lineage.
6. Follow mlflow/tracing/provider.py and mlflow/tracing/client.py from span creation to persistence.
7. Inspect mlflow/genai/ for evaluation datasets, scorers, and prompts; follow gateway-specific handling into mlflow/gateway/.

Use tag v3.16.1 for this reading exercise instead of moving master. Start with MlflowClient.log_param, follow the tracking client to SqlAlchemyStore.log_param, and find the database constraint that rejects changing an existing parameter value. Then compare it with mutable run tags and prompt aliases. Public concepts are more stable than the implementation paths around them.^{[1]Reference 1MLflow 3.16.1https://github.com/mlflow/mlflow/releases/tag/v3.16.1}

Test that distinction before reading deeper. The local lab tries to overwrite retrieval_k=40 with 50, then changes a run tag and a prompt version's temperature. The parameter update must fail; the tag and configuration updates must succeed. A useful source-reading explanation identifies where each rule is enforced and which historical values remain recoverable. Naming three API methods without explaining their different mutability rules misses the important boundary.

Designing an evidence contract

Start an MLflow deployment with required fields and release receipts, not with a dashboard. For the policy assistant, require:

• Application release, source commit (8b2f9e1), environment, and dependency lock.
• Resolved model provider and exact model identifier.
• Resolved prompt version alongside its alias (support-answer@production and v17 or v18), plus the model configuration actually used.
• Retrieval index and corpus revision (eval-08-01 isn't a substitute for the live index hash).
• Root trace ID plus tool, retrieval, and model spans.
• Token, latency, failure, and cost attributes with known units.
• Evaluation dataset revision and scorer versions.
• Per-case assessments plus aggregate slices, including RET-104.
• Approval identity, candidate version, and digests of the evaluated artifact and configuration.
• Alias movement event and rollback target.

That receipt lets a team answer four different questions: what ran, what it observed, how it scored, and why it was promoted. Without those distinctions, one green dashboard can hide missing cases, stale aliases, or dropped traces.