AI Lab System Design Interview

At minute twelve, your board already has an API gateway, GPU workers, and a vector index. The interviewer asks: an under-quota tenant faces a queue that will miss its first-token SLO; what does the client receive, who owns that decision, and what can support show later? A polished diagram has no answer until each boundary names a number, an owner, a receipt, and a recovery path.

The previous lesson used a documentation crawler, quotas, and worker handoffs to practice ownership of shared state. This round puts those pieces under a product promise. Start with who uses the system and the constraint that can break that promise, then add infrastructure only when the constraint calls for it.

We'll keep one running product: an internal model gateway that teams use to call multiple LLM providers. Later follow-ups add permission-aware retrieval and a long-running coding agent, so the same first sketch has to survive privacy, latency, cancellation, and rollout pressure.

These are practice cases, not a report of any lab's question bank or scoring rubric. OpenAI's public guide, checked October 10, 2026, emphasizes clear reasoning and communication, but the actual format depends on the role and team. Confirm the duration and scope with your recruiter; use the rehearsal schedule below as an adjustable exercise.^{[1]Reference 1Interview guidehttps://openai.com/interview-guide/}

Keep the control plane separate from the data plane in your explanation. Entitlements, route policy, model registry, quota configuration, and rollout state change relatively slowly; admission, retrieval, provider calls, and agent execution carry requests. When a policy changes or its control store is unavailable, point to how the data plane receives the decision and what fails closed. That question turns a box into an operating boundary.

Start with a constraint

Open with a constraint, not a component list:

I'll keep the first design simple, size the constraints early, then add queues, caches, sharding, model routing, or eval gates only when the requirement forces them.

Treat the board as a chain of commitments. Each step supplies a fact for the next: a latency SLO bounds queue and GPU capacity, while a retrieval requirement earns an index. Say what artifact will prove each commitment when the system runs.

1. Goal: who uses it and what success means.
2. Requirements: functional and non-functional.
3. Scale: QPS, tokens, tenants, documents, latency, retention.
4. API: external contract and important internal interfaces.
5. Data model: entities, indexes, isolation, retention.
6. Architecture: simplest request path first.
7. Reliability: retries, idempotency, backpressure, overload, failover.
8. Safety/security: permissions, audit, abuse controls, rollback.
9. Observability: metrics, logs, traces, support views.
10. Rollout: beta gates, canaries, eval gates, kill switches.

The order also gives you a way to communicate under interruption. If the interviewer jumps to sharding, tie it back to the model, traffic shape, and SLO; if they jump to safety, tie it back to the actor, permission boundary, and audit receipt. The answer stays understandable because each new box has a reason to exist.

Let the pressure choose the pattern

An AI/backend design can combine several patterns, but one constraint usually deserves attention first. A quota problem calls for admission control, an access-control list (ACL) calls for eligibility checks, and an irreversible action calls for durable state plus explicit authorization.

Use the table to keep the first design narrow. A model gateway earns a vector database only when retrieval enters the product; retrieval earns agent planning only when the user asks the system to act. Our running product crosses the first four rows in that order, so each follow-up adds a pressure instead of replacing the original boundary.

Let follow-ups move a boundary

Treat each follow-up as a changed requirement, not a request for another component. Name what changed, move the boundary that owns it, and state the receipt or rollback signal that proves the move worked.

That three-part answer keeps the design conversational. The interviewer can challenge a trade-off, while you can point to the queue, policy record, metric, or recovery action that would settle it.

Put one number on the board

One back-of-the-envelope calculation gives the diagram a load-bearing assumption. Keep units visible, name what the estimate leaves out, and say which boundary changes when the number moves.

Use this phrasing when an assumption is still rough:

I'll size the constraint that most affects the design. If that assumption changes, the architecture boundary I would revisit is X.

Size model memory before drawing workers

The gateway might call hosted providers only. In that case, provider quotas and latency measurements matter more than GPU sizing. If the interviewer adds self-hosted inference, put two budgets on the board: fixed weight memory and per-request key/value (KV) cache growth. These are storage estimates, not throughput benchmarks.

Eight billion parameters at two bytes each occupy 16 billion bytes, or 16 decimal GB. Double the concurrent requests and those loaded weights stay the same size. That fixed term is the parameter count times the bytes per parameter:

$M_\text{weights} = P \times b$

Here $P$ is the parameter count and $b$ is bytes per stored parameter: 2 for 16-bit floating point (FP16) or brain floating point (BF16), 1 for 8-bit floating point (FP8). Quantization scales, unquantized tensors, and runtime allocations add overhead. Decimal GB uses $10^9$ bytes; GiB uses $2^{30}$ bytes.

The cache grows as requests retain more tokens. In a 32-layer model with 8 KV heads, each 128 values wide, one BF16 token needs $2 \times 32 \times 8 \times 128 \times 2 = 131{,}072$ bytes: 128 KiB. Count keys and values in each layer to generalize that calculation:

$M_\text{token} = 2 \times n_\text{layers} \times n_\text{KV heads} \times d_\text{head} \times p_\text{bytes}$

The leading 2 counts key and value; $n_\text{layers}$ counts layers, $n_\text{KV heads}$ counts stored KV heads, $d_\text{head}$ is values per head, and $p_\text{bytes}$ is bytes per value. For $B$ requests each retaining $S$ tokens, multiply by $B S$:

$M_\text{KV} = 2 \times B \times S \times n_\text{layers} \times n_\text{KV heads} \times d_\text{head} \times p_\text{bytes}$

For unequal request lengths, replace $B S$ with their sum. This full-attention estimate assumes no prefix sharing, sliding-window eviction, compression, or cache sharding; block allocation also rounds storage upward. Count retained prompt and generated tokens, not just input length.

Use a Llama 3 8B-shaped sketch so the scale is concrete: 32 layers, 32 query heads, head dimension 128, BF16. The first KV line is a multi-head-attention counterfactual with 32 key/value heads. Llama 3 8B uses grouped-query attention (GQA), sharing 8 key/value heads across those 32 query heads, which is the memory lever to test.^{[2]Reference 2The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783[3]Reference 3GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.https://arxiv.org/abs/2305.13245}

Before running the fixture, predict which bar grows with batch and context. The answer should be KV, while the weight bar stays fixed for a loaded model. That prediction is the bridge from arithmetic to admission policy.

params = 8_000_000_000
bytes_per_param = 2  # FP16 or BF16
weights_gb = params * bytes_per_param / 1e9

n_layers, d_head, p_bytes = 32, 128, 2
n_kv_heads_mha = 32  # MHA counterfactual: one KV head per query head
kv_bytes_per_token = 2 * n_layers * n_kv_heads_mha * d_head * p_bytes
batch, seq = 16, 2048
kv_gb = kv_bytes_per_token * batch * seq / 1e9

n_kv_heads_gqa = 8  # Llama 3 8B GQA
kv_bytes_per_token_gqa = 2 * n_layers * n_kv_heads_gqa * d_head * p_bytes
kv_gb_gqa = kv_bytes_per_token_gqa * batch * seq / 1e9

print("weights_gb:", weights_gb)
print("kv_bytes_per_token:", kv_bytes_per_token)
print("kv_mb_per_token:", round(kv_bytes_per_token / 1e6, 2))
print("kv_gb:", round(kv_gb, 1))
print("kv_gb_gqa:", round(kv_gb_gqa, 1))
assert weights_gb == 16.0
assert kv_bytes_per_token == 524288
assert round(kv_gb, 1) == 17.2
assert round(kv_gb_gqa, 1) == 4.3

weights_gb: 16.0
kv_bytes_per_token: 524288
kv_mb_per_token: 0.52
kv_gb: 17.2
kv_gb_gqa: 4.3

The figure compares two architectures, not a serving flag: you can't change a checkpoint's KV-head count and assume its learned behavior stays intact. First confirm that weights and runtime overhead fit; then KV cache often sets marginal batch and context capacity. Activations and workspace still need a slice of high-bandwidth memory (HBM), so leave headroom instead of planning to fill the card to 100%.

The numbers describe a model shape and workload fixture, not a measured serving result. A real capacity claim needs GPU and software versions, model and precision, prompt/output distribution, batch or concurrency, warmup, baseline, and correctness criteria beside the number. Without that packet, “17 GB” supports a memory argument but offers no throughput promise.

Leave a decision trail

Architecture choices become useful when a follow-up changes one assumption. Keep a visible decision log that records the chosen boundary, the rejected alternative, the reason, and the signal that would make you reverse it:

This leaves the interviewer something concrete to challenge. They can disagree with the choice and still see its owner, trade-off, and reversal signal; in a running system, the same record helps support explain why the path behaved that way.

Rehearse the board as a sequence

For a 45-minute rehearsal, spend early minutes making the promise and bottleneck testable, then reserve time to show what happens when the promise fails. This allocation is a practice aid, not a hiring standard.

If the interviewer interrupts early, jump to the dominant constraint:

The part that most changes the design is constraint. I'll size that first, then show the request path it forces.

Turn requirements into boundaries

Use each prompt as a short design case. State the user, promise, dominant constraint, and support receipt first; ask only questions that could change the boundary. Read the solution guide as an evidence trail, not as a list of fashionable components.

Prompt 1: model gateway for enterprise teams

Design an API gateway for teams calling multiple LLM providers through one company platform.

Start with this situation:

• Each organization has workspaces, users, API keys, and model access rules.
• The gateway must enforce requests/minute, tokens/minute, and monthly spend limits.
• Support needs a request ID that can explain which route, model, policy decision, and overload state happened.
• Some models are beta-only and must be gated.
• Traffic can spike 10x during customer-support incidents.

Ask questions that can change the boundary:

• Are clients streaming responses, batch jobs, or both?
• Should spend limits be hard stops, soft alerts, or tier-dependent?
• Which decision must support explain first: auth failure, quota failure, route choice, or provider failure?

The gateway case establishes the receipt: request ID, policy version, route, quota, and provider outcome. Carry that receipt into retrieval, where the most dangerous failure is an eligible-looking document that should never enter model context.

Prompt 2: permission-aware enterprise retrieval

Design retrieval for an internal assistant that answers employee questions from company documents.

Start with this situation:

• Documents come from multiple systems with different ACL formats.
• Permission changes and deletions must take effect quickly.
• Answers must cite sources.
• The assistant must not retrieve and then filter private documents after generation.
• Admins need auditability for "why did this answer use this document?"

Ask questions that can change the boundary:

• What is the revocation target: seconds, minutes, or hours?
• Can indexes be physically separated by tenant, or must filters enforce isolation?
• Should the assistant fail closed when ACL freshness is unknown?

Here the request receipt gains an eligibility decision: user identity, ACL version, filters, retrieved IDs, and citations. The same discipline matters when the “request” runs for thirty minutes and can write to a repository.

Prompt 3: long-running coding agent service

Design a service that accepts repository tasks and runs a coding agent asynchronously.

Start with this situation:

• Users submit a repo, branch, task prompt, and tool permissions.
• Jobs can run for 30 minutes and may need retries or human review.
• The agent can create commits, run tests, and leave artifacts.
• Users need live progress, cancellation, logs, and final diff review.
• Secrets and production systems must be protected.

Ask questions that can change the boundary:

• Is the service allowed to push branches, or should it only produce a patch artifact?
• Which tools need network access, and which should run offline?
• What happens when a user cancels during a tool call or test run?

Request admission owns overload

Start with one request at the front door. Assign a server-generated request ID before authentication so rejected requests remain traceable. Then authenticate the key, authorize model access, check workspace and organization limits, reserve budget, and attempt admission. Predict two cases: an over-quota tenant while GPUs sit idle, and an eligible tenant facing a saturated fleet. Our gateway contract distinguishes those rejection causes.

What this boundary must answer:

• API keys map to workspace and organization.
• Limits apply across requests/minute, tokens/minute, model, and tier.
• Request IDs appear in responses and logs.
• In this contract, caller rate limits return 429; shared capacity exhaustion returns 503. Both carry a request ID and a machine-readable cause. Include a retry hint only when there's a defensible estimate.
• Beta features are gated by explicit version or feature flags.
• Rollout has canaries, kill switches, and regression checks.

Use Python to sanity-check rate math before drawing capacity boxes. The values below are a synthetic arithmetic fixture: they translate an assumed request mix into a token budget, not a production throughput claim.

requests_per_minute = 4_000
avg_input_tokens = 1_200
avg_output_tokens = 450
tokens_per_minute = requests_per_minute * (avg_input_tokens + avg_output_tokens)
tokens_per_second = tokens_per_minute / 60

print("tokens_per_minute:", tokens_per_minute)
print("tokens_per_second:", round(tokens_per_second))

tokens_per_minute: 6600000
tokens_per_second: 110000

That 6.6-million-token figure is aggregate demand, not a safe per-request reservation: averages won't enforce a hard spend cap. Reserve each request's tokenized input plus maximum output for its token budget. A money reservation also needs the route's full tariff: ordinary input, cache reads/writes where priced separately, generated tokens including billable hidden reasoning, and tool charges. Use conservative bounds when the cache hit or tool count is unknown, and separately reserve any authorized retry or fallback. For example, OpenAI's current cache-write rate can exceed its ordinary input rate; pricing only an uncached input/output pair can underreserve.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching} The next fixture uses a single token budget and shows a refund when actual use is smaller:

remaining_before = 10_000
reserved_tokens = 1_800
actual_tokens = 1_520

remaining_after_admission = remaining_before - reserved_tokens
remaining_after_reconcile = remaining_after_admission + (reserved_tokens - actual_tokens)

print("after admission:", remaining_after_admission)
print("after reconcile:", remaining_after_reconcile)
assert remaining_after_admission == 8_200
assert remaining_after_reconcile == 8_480

after admission: 8200
after reconcile: 8480

The subtraction above assumes one request. A real quota ledger must atomically check and reserve across concurrent callers, then reconcile once per request ID. Duplicate completion events must not refund twice. If a stream disconnects before usage arrives, keep the reservation pending and reconcile from provider accounting; treating missing usage as zero would bypass the cap. Request-count limits, token limits, and monetary limits need separate accounting windows.

Queueing changes the latency promise

An inference scheduler is where a product promise meets GPU physics. Imagine a long prompt arriving while short responses are streaming: admitting it may improve utilization while delaying first tokens, but rejecting it may waste capacity. NVIDIA documents in-flight batching as a way to interleave context and generation work, with token and batch limits that still need tuning against latency.^{[6]Reference 6Paged Attention, IFB, and Request Scheduling.https://nvidia.github.io/TensorRT-LLM/features/paged-attention-ifb-scheduler.html}

Separate prefill from decode

Before choosing a policy, predict which request phase can hurt which user. Prefill builds prompt KV and the logits used to choose the first token; a matching prefix cache can supply existing KV, leaving only the uncached suffix to process.^{[5]Reference 5Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching} Ordinary autoregressive decode then extends the sequence one token per step. Measure both with time to first token (TTFT), inter-token latency (ITL), and time per output token (TPOT).

That split gives you two promises to test. A fleet can report high mean tokens/sec while first-token UX or stream feel fails in the tail. DistServe defines goodput using TTFT and per-request TPOT constraints, then separates resource allocation for the phases.^{[7]Reference 7DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} A long colocated prefill can delay ongoing decode. Chunked prefill limits the prefill work per step; disaggregation separates compute but adds KV transfer, network contention, and pool-balancing costs. Neither guarantees that the SLO will pass.

Name the metric that can fail

When someone says “throughput improved,” ask which user-visible promise moved. Name the metric, percentile, unit, and boundary before accepting the comparison:

• Time to first token (TTFT): arrival → first output token (queue + prefill + first token).
• Inter-token latency (ITL) / time between tokens (TBT): each gap after the first token; use the distribution (including tail) for stream feel.
• Time per output token (TPOT): for $N>1$ output tokens, time from first to last token divided by $N-1$. This per-request average can hide a long individual ITL; it's undefined for a one-token response.
• Queue wait time before admission or first scheduled work.
• p95 and p99 end-to-end latency, plus p95/p99 of TTFT and ITL when the product streams.
• Fleet goodput: successful requests per second satisfying the chosen latency constraints. Count rejections separately; admitting requests doesn't mean they met the SLO. Choose explicitly whether decode pacing is evaluated by TPOT, ITL, or both.
• Tokens per second only with a label: per-request decode rate, fleet aggregate, or prefill rate. Undifferentiated TPS is not a substitute for ITL or TPOT.
• GPU utilization.
• Error rate and overload rejections.
• Cost per successful request.

Attach an evidence packet to any capacity or performance number: GPU and software versions, model and precision, serving algorithm, prompt and output distribution, arrival rate, batch or concurrency, warmup, baseline, and correctness criteria. A naked TPS value leaves unclear whether the batch, sequence mix, implementation, or acceptance rule changed.

Choose a serving lever by failure mode

Once the failing metric is named, choose the smallest mechanism that changes it. The toolkit table distinguishes memory layout, admission timing, caching, and phase placement:

Continuous batching and paged attention solve different failures. Continuous batching decides when sequences join or leave the batch; paged attention decides how KV blocks are laid out and freed. State both mechanisms separately, then say which measured metric should move.

A self-hosted prefix cache must match token IDs and positions, model/adapter revision, and relevant cache configuration. Put tenant and authorization scope in the lookup policy too: identical text doesn't authorize sharing private state. Hosted providers have their own matching and retention rules; changing a route doesn't carry its old cache hit with it. A cache miss changes both prefill work and the billing estimate.

Model-fit scale math

If weights plus KV don't fit one GPU, name two decisions separately. The logical shard plan can use tensor parallelism (TP) for layer shards, pipeline parallelism (PP) across stages, or expert parallelism when mixture-of-experts (MoE) routing requires it. The physical route decides whether those exchanges cross high-bandwidth NVLink, PCIe, or a network link. TP can fit a model and still miss latency if the route is slow; PP can fit across nodes and still pay pipeline bubbles. Say which communication cost the SLO can afford.

Use a measured workload-specific fleet capacity only when you can describe how it was measured. The following 125,000-token value is a planning fixture, not a benchmark; a real result needs the evidence packet described above:

assumed_capacity_tokens_per_second = 125_000
expected_demand_tokens_per_second = 110_000

headroom = assumed_capacity_tokens_per_second - expected_demand_tokens_per_second
headroom_percent = headroom / assumed_capacity_tokens_per_second * 100

print("headroom_tokens_per_second:", headroom)
print("headroom_percent:", round(headroom_percent, 1))

headroom_tokens_per_second: 15000
headroom_percent: 12.0

Here 12% is unused capacity divided by assumed capacity, not spare capacity divided by demand. Input and output tokens have different costs, so the combined rate is useful only for a fixed request mix. The next approximation treats queued tokens as homogeneous work; a real scheduler tracks prefill backlog, active decode sequences, free KV blocks, and measured latency tails separately.

HTTP defines 429 Too Many Requests for rate limiting and 503 Service Unavailable for temporary inability to serve, including overload or maintenance. Retry-After is optional, not a required promise that the next attempt succeeds.^{[8]Reference 8Additional HTTP Status Codeshttps://www.rfc-editor.org/rfc/rfc6585.html#section-4[9]Reference 9HTTP Semantics: 503 Service Unavailablehttps://www.rfc-editor.org/rfc/rfc9110.html#section-15.6.4} Our gateway maps caller limits to 429 and shared capacity exhaustion to 503. That's a documented API policy, not a universal provider convention. Preserve upstream error details internally and expose a stable cause code; clients should use bounded backoff with jitter and retry only safe operations.

The figure uses illustrative policy inputs, not a benchmark. The wait check below is its 503 side: 20,000 queued tokens clear a 0.25 s budget at 125,000 tokens/s; 50,000 exceed it. The two-resource sketch then adds decode slots and free KV blocks, so a request can fail admission even when the prefill backlog looks fine.

from math import isfinite

def admit(queued_tokens: int, service_tokens_per_second: int, max_queue_wait_seconds: float) -> bool:
    if not all(isfinite(value) for value in (queued_tokens, service_tokens_per_second, max_queue_wait_seconds)):
        raise ValueError("non-finite queue capacity input")
    if queued_tokens < 0 or service_tokens_per_second <= 0 or max_queue_wait_seconds < 0:
        raise ValueError("invalid queue capacity input")
    estimated_wait = queued_tokens / service_tokens_per_second
    return estimated_wait <= max_queue_wait_seconds

def admit_two_resource(
    prefill_backlog_tokens: int,
    prefill_tokens_per_second: int,
    active_decode_sequences: int,
    max_decode_sequences: int,
    free_kv_blocks: int,
    kv_blocks_needed: int,
    max_prefill_wait_seconds: float,
) -> bool:
    """Sketch dual capacity: prefill backlog and decode/KV slots both must clear."""
    counts = (active_decode_sequences, max_decode_sequences, free_kv_blocks, kv_blocks_needed)
    if any(type(count) is not int or count < 0 for count in counts):
        raise ValueError("resource counts must be nonnegative integers")
    prefill_ok = admit(prefill_backlog_tokens, prefill_tokens_per_second, max_prefill_wait_seconds)
    decode_ok = active_decode_sequences < max_decode_sequences
    kv_ok = free_kv_blocks >= kv_blocks_needed
    return prefill_ok and decode_ok and kv_ok

print(admit(queued_tokens=20_000, service_tokens_per_second=125_000, max_queue_wait_seconds=0.25))
print(admit(queued_tokens=50_000, service_tokens_per_second=125_000, max_queue_wait_seconds=0.25))
print(
    admit_two_resource(
        prefill_backlog_tokens=10_000,
        prefill_tokens_per_second=80_000,
        active_decode_sequences=48,
        max_decode_sequences=64,
        free_kv_blocks=120,
        kv_blocks_needed=16,
        max_prefill_wait_seconds=0.25,
    )
)

# A telemetry adapter must not turn missing capacity into infinity or fractional slots.
try:
    admit_two_resource(10_000, 80_000, 48, 64, float("inf"), 16, 0.25)
except ValueError as error:
    print("invalid capacity:", error)
else:
    raise AssertionError("infinite free KV was admitted")

try:
    admit_two_resource(10_000, 80_000, 48.5, 64, 120, 16, 0.25)
except ValueError:
    pass
else:
    raise AssertionError("fractional decode count was admitted")

True
False
True
invalid capacity: resource counts must be nonnegative integers

The predicate estimates backlog wait, not complete TTFT: the new request's own prefill, routing, and network time still consume the latency budget. It also doesn't reserve resources. A concurrent scheduler must check and claim decode/KV capacity atomically, then release it on completion or cancellation; otherwise two individually passing requests can oversubscribe the same slots.

Permissions belong before retrieval

Picture a user whose finance access was revoked seconds ago. If a private chunk reaches an unauthorized model service, filtering the final answer can't undo that exposure. Enforce eligibility before content crosses a trust boundary, including before an external reranker or generator. A trusted search engine may compute internal similarity scores before its ACL filter; the requirement is that unauthorized content never escapes that authorized boundary.

Draw the data path and the control decision together:

• Connector ingestion workers with backpressure.
• Per-document ACLs or delegated auth checks.
• Tenant-isolated indexes or strict metadata filters.
• Hybrid retrieval plus reranking.
• Citation output with source IDs.
• Deletion and retention jobs.
• Offline evals for recall and answer faithfulness.
• Support traces that show which documents were eligible.

The ACL snapshot and hybrid index both feed candidate selection. A snapshot alone still leaves a revocation window: a permission removed one second ago may remain in a 20-second-old snapshot. For immediate revocation, check an authoritative source at the relevant boundary or enforce an invalidation protocol with a documented delay. Apply the same policy to cached answers, shared prefixes, and support traces; those are also data access paths.

For a sensitive source, predict the stale-snapshot branch before reading the fixture. Encode that freshness decision as a fail-closed policy:

from math import isfinite

def snapshot_is_usable(snapshot_age_seconds: float | None, max_age_seconds: float) -> bool:
    if not isfinite(max_age_seconds) or max_age_seconds < 0:
        raise ValueError("invalid freshness budget")
    if snapshot_age_seconds is None or not isfinite(snapshot_age_seconds) or snapshot_age_seconds < 0:
        return False
    return snapshot_age_seconds <= max_age_seconds

print(snapshot_is_usable(snapshot_age_seconds=20, max_age_seconds=60))
print(snapshot_is_usable(snapshot_age_seconds=90, max_age_seconds=60))
print(snapshot_is_usable(snapshot_age_seconds=None, max_age_seconds=60))

True
False
False

Passing that freshness check doesn't grant access. The caller must still pass tenant, user/group, and document policy checks. Public sources may use a different explicit freshness policy; unknown ACL state should never silently turn private content into public content.

Now test ordering inside one tenant with a tiny candidate set. Scores are fixtures supplied by a trusted index, not an external reranker. A high score can't authorize a private document, and a tombstone removes a deleted document before ranking the eligible results:

documents = [
    {"id": "public-access-policy", "groups": {"employees"}, "deleted": False, "score": 0.82},
    {"id": "finance-plan", "groups": {"finance"}, "deleted": False, "score": 0.99},
    {"id": "old-handbook", "groups": {"employees"}, "deleted": True, "score": 0.95},
]
user_groups = {"employees"}

eligible = [
    document
    for document in documents
    if not document["deleted"] and document["groups"] & user_groups
]
ranked_ids = [document["id"] for document in sorted(eligible, key=lambda item: item["score"], reverse=True)]

print(ranked_ids)
assert ranked_ids == ["public-access-policy"]

['public-access-policy']

Durable state bounds agent side effects

Consider a worker restarting after an agent created a commit but before it reported completion. If intent, tool calls, artifacts, checkpoints, logs, and permissions lived only in process memory, support has no way to tell whether a retry will duplicate the write. Durable state turns long-running work into an inspectable record; scoped execution keeps that record from becoming uncontrolled execution.

Make the state and permission boundaries explicit:

• Task states: queued, provisioning, running, blocked, review, failed, canceled, complete.
• Checkpoints for resumability.
• Tool permission scopes and audit logs.
• Secret redaction.
• Git branch and conflict handling.
• Streaming progress.
• Cancellation and deadlines.
• Evals for task success and regression.

Before discussing worker count, write legal transitions down. Predict whether a completed task can return to running, then let the state machine make that answer explicit:

ALLOWED_TRANSITIONS = {
    "queued": {"provisioning", "canceled"},
    "provisioning": {"running", "failed", "canceled"},
    "running": {"blocked", "review", "failed", "canceled"},
    "blocked": {"running", "review", "failed", "canceled"},
    "review": {"running", "complete", "canceled"},
    "failed": set(),
    "canceled": set(),
    "complete": set(),
}

def can_transition(current: str, target: str) -> bool:
    return target in ALLOWED_TRANSITIONS.get(current, set())

print(can_transition("running", "review"))
print(can_transition("complete", "running"))

True
False

This predicate validates a transition but doesn't persist it. Workers also need an atomic expected-state update or lease so two workers can't both claim the same attempt. Keep cancellation intent separate from confirmed termination, and record a new attempt rather than silently reopening a terminal result.

Retry policy follows side effects. A key helps only if the receiver durably deduplicates that key, rejects changed payloads, and retains the record throughout the retry window. Provisioning is a write too. The policy below assumes a transient error, an authorized action, and remaining retry budget; it chooses a recovery method, not permission to retry forever.^{[4]Reference 4Making retries safe with idempotent APIshttps://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/} The idempotency flag comes from verified receiver configuration. A nonempty string such as "False" is truthy in Python, so only the actual boolean True enables automatic write replay.

def retry_mode(action: str, receiver_enforces_idempotency: bool = False) -> str:
    if action == "repo_read":
        return "automatic"
    if action in {"sandbox_provision", "create_commit", "post_comment"} and receiver_enforces_idempotency is True:
        return "automatic-with-idempotency"
    return "reconcile-or-review"

print(retry_mode("repo_read"))
print(retry_mode("create_commit"))
print(retry_mode("post_comment", receiver_enforces_idempotency=True))
print(retry_mode("sandbox_provision"))
print(retry_mode("create_commit", receiver_enforces_idempotency="False"))
assert retry_mode("create_commit", receiver_enforces_idempotency="False") == "reconcile-or-review"

automatic
reconcile-or-review
automatic-with-idempotency
reconcile-or-review
reconcile-or-review

Evaluation becomes a release boundary

An evaluation monitor earns its place when it changes a release decision. Fixed expectations and incident discoveries become a regression suite; a failed or missing required check blocks launch instead of disappearing into an average. Here recall means the fraction of labeled relevant documents retrieved, and citation faithfulness means the fraction of scored citations that support their associated claim. These are synthetic aggregate scores with declared thresholds, not confidence estimates. Predict what a below-threshold citation score should do even when recall passes. Then try an empty result packet: no failures were reported, but no required checks ran either.

from math import isfinite

REQUIRED_MINIMA = {"retrieval_recall": 0.92, "citation_faithfulness": 0.95}

def launch_failures(checks: dict, permission_leaks: int | None) -> list[str]:
    failures = []
    for name, required_minimum in REQUIRED_MINIMA.items():
        result = checks.get(name)
        if not isinstance(result, tuple) or len(result) != 2:
            failures.append(name)
            continue
        observed, minimum = result
        valid = all(
            type(value) in (int, float) and isfinite(value) and 0 <= value <= 1
            for value in (observed, minimum)
        )
        if not valid or minimum != required_minimum or observed < required_minimum:
            failures.append(name)
    if type(permission_leaks) is not int or permission_leaks != 0:
        failures.append("permission_leaks")
    return failures

checks = {
    "retrieval_recall": (0.94, 0.92),
    "citation_faithfulness": (0.93, 0.95),
}
failures = launch_failures(checks, permission_leaks=0)

print("launch_allowed:", not failures)
print("failures:", failures)

missing = launch_failures({}, permission_leaks=None)
print("missing results:", missing)
assert missing == ["retrieval_recall", "citation_faithfulness", "permission_leaks"]

altered_policy = launch_failures({
    "retrieval_recall": (0.94, 0.92),
    "citation_faithfulness": (0.93, 0.0),
}, permission_leaks=0)
print("altered policy:", altered_policy)
assert altered_policy == ["citation_faithfulness"]
assert launch_failures({
    "retrieval_recall": (0.99, float("nan")),
    "citation_faithfulness": (0.99, 0.95),
}, permission_leaks=0) == ["retrieval_recall"]
assert launch_failures({
    "retrieval_recall": (0.99, 0.92),
    "citation_faithfulness": (0.99, 0.95),
}, permission_leaks=float("nan")) == ["permission_leaks"]

launch_allowed: False
failures: ['citation_faithfulness']
missing results: ['retrieval_recall', 'citation_faithfulness', 'permission_leaks']
altered policy: ['citation_faithfulness']

Changing the citation tuple from (0.93, 0.95) to (0.93, 0.0) would make the evaluator approve its own failing score if the gate trusted the supplied minimum. The host owns REQUIRED_MINIMA; the tuple only echoes that policy. In a deployed gate, load a versioned policy independently of the evaluator result and bind the result to that version.

A passing fixture doesn't establish a low leak rate: zero observed leaks in a small or unrepresentative sample isn't proof of absence. This adapter receives two-element metric tuples and a measured integer leak count; absent results, invalid thresholds, and unknown leak counts block launch. Specify dataset versions, critical slices, sample sizes, uncertainty, and judge calibration before using the thresholds for a launch.

Keep online canaries reversible. The next policy assumes a validated measurement window for the candidate release, including adequate sample counts and fresh metrics. A confirmed permission leak blocks expansion and triggers containment regardless of latency. Rollback can stop further exposure but can't retract leaked data; revoke access and investigate as needed. Here permission_leaks means confirmed unauthorized exposure, not correctly denied requests.

from math import isfinite

def rollout_decision(error_rate: float | None, p95_latency_ms: float | None, permission_leaks: int | None) -> str:
    if type(permission_leaks) is int and permission_leaks > 0:
        return "rollback"
    if type(permission_leaks) is not int or permission_leaks < 0:
        return "hold"
    if error_rate is None or p95_latency_ms is None:
        return "hold"
    if not isfinite(error_rate) or not 0 <= error_rate <= 1 or not isfinite(p95_latency_ms) or p95_latency_ms < 0:
        return "hold"
    if error_rate > 0.01 or p95_latency_ms > 900:
        return "hold"
    return "expand"

print(rollout_decision(error_rate=0.004, p95_latency_ms=720, permission_leaks=0))
print(rollout_decision(error_rate=0.004, p95_latency_ms=720, permission_leaks=1))
print(rollout_decision(error_rate=None, p95_latency_ms=720, permission_leaks=0))

expand
rollback
hold

Test attractive shortcuts

Shortcut questions help only when they expose a mechanism and its operating cost. Use the table to replace each appealing claim with a failure mode, then ask what measurement would prove the replacement works.

For each row, name the old approach's failure and the mechanism that replaces it. “We use vLLM now” is not an operating argument until you say which metric, workload, and correctness check moved.

Make failure visible on the board

End by testing the design against six omissions that turn a convincing diagram into an unsafe system:

• Implementation technology appears before user value, so scale math can optimize the wrong product.
• A cache has no invalidation, privacy, or freshness boundary, so fast answers can become stale or unauthorized.
• Monitoring names dashboards without exact metrics, units, percentiles, or owners, so regressions stay ambiguous.
• Overload and support/debug paths are missing, so callers retry blindly and operators lack an explanation for a rejection.
• Safety is a slogan instead of evals, permissions, staged rollout, and rollback, so a policy violation reaches users.
• Agent actions are not reversible and auditable, so retries can duplicate side effects without a reviewable record.