AI Lab System Design Interview

AI lab system design rounds test whether you can turn ambiguous model-product requirements into a reliable backend. The winning answer is rarely the most complex architecture. It is the design that names the product goal, sizes the hard constraint, then adds queues, caches, model routing, evals, permissions, or human review only where requirements force them.

Use ecommerce, delivery, and customer-support examples when you need a concrete neutral domain: order lookup, shipment tracking, returns support, catalog search, and internal agent workflows have real latency, privacy, and support-debug constraints.

The design round script

Open with:

I will keep the first design simple, size the constraints early, then add queues, caches, sharding, model routing, or eval gates only when the requirement forces them.

Then follow this order:

1. Goal: who uses it and what success means.
2. Requirements: functional and non-functional.
3. Scale: QPS, tokens, tenants, documents, latency, retention.
4. API: external contract and important internal interfaces.
5. Data model: entities, indexes, isolation, retention.
6. Architecture: simplest request path first.
7. Reliability: retries, idempotency, backpressure, overload, failover.
8. Safety/security: permissions, audit, abuse controls, rollback.
9. Observability: metrics, logs, traces, support views.
10. Rollout: beta gates, canaries, eval gates, kill switches.

Prompt bank

Drill 1: API gateway and rate-control plane

The gateway is the front door. It authenticates API keys, resolves workspace and organization limits, estimates request cost, routes to a model or queue, and emits a request ID that support can follow.

Design checklist:

• API keys map to workspace and organization.
• Limits apply across requests/minute, tokens/minute, model, and tier.
• Request IDs appear in responses and logs.
• Overload returns useful 429 or degraded fallback, not silent queue growth.
• Beta features are gated by explicit version or feature flags.
• Rollout has canaries, kill switches, and regression checks.

Use Python to sanity-check rate math before drawing capacity boxes:

Drill 2: inference scheduler

For LLM serving, the scheduler is where latency, cost, and fairness meet. Production serving systems use request scheduling and in-flight batching to balance throughput and latency under GPU memory constraints.^[1] Mention these metrics:

• Time to first token.
• Tokens per second.
• Queue wait time.
• p95 and p99 end-to-end latency.
• GPU utilization.
• Error rate and overload rejections.
• Cost per successful request.

Drill 3: permission-aware retrieval

For enterprise retrieval, the critical rule is: do not retrieve private data and filter it after generation. Permission constraints must be part of candidate selection, ranking, and auditing.

Architecture pieces:

• Connector ingestion workers with backpressure.
• Per-document ACLs or delegated auth checks.
• Tenant-isolated indexes or strict metadata filters.
• Hybrid retrieval plus reranking.
• Citation output with source IDs.
• Deletion and retention jobs.
• Offline evals for recall and answer faithfulness.
• Support traces that show which documents were eligible.

Drill 4: long-running coding agents

Long-running agent infrastructure has to persist intent, tool calls, artifacts, checkpoints, logs, and permissions. The main design risk is not just failed execution. It is uncontrolled execution.

Cover:

• Task states: queued, running, blocked, failed, canceled, complete.
• Checkpoints for resumability.
• Tool permission scopes and audit logs.
• Secret redaction.
• Git branch and conflict handling.
• Streaming progress.
• Cancellation and deadlines.
• Evals for task success and regression.

Failure modes to avoid

• Starting with implementation technology before naming user value.
• Caching without invalidation, privacy, or freshness.
• Monitoring without exact metrics.
• Ignoring overload and support/debug needs.
• Treating safety as a slogan instead of evals, permissions, staged rollout, and rollback.
• Forgetting that agent systems need reversible actions and audit trails.

Mastery checklist

• Design an API gateway with request IDs, workspaces, rate limits, beta gates, and overload behavior.
• Design an inference scheduler with queue metrics, batching, fairness, and 429 behavior.
• Design a permission-aware retrieval system with ACLs, deletion, citations, and evals.
• Design long-running coding-agent infrastructure with durable state, permissions, logs, and cancellation.
• Design an evaluation monitor that turns incidents into regression tests and launch gates.