AI Engineer Portfolio Projects for Interviews

The interesting part of a document assistant often starts after its first good answer. It cites a policy, but the policy was superseded. You fix retrieval, then discover that the new prompt follows an instruction hidden in a retrieved comment. The repository now contains something more useful than a demo: a sequence of decisions another engineer can examine.

Employers value different portfolio signals. Anthropic welcomes independent research, writing, and open-source work on resumes; OpenAI's engineering interview guide emphasizes design, code, performance, and tests.^{[1]Reference 1Careershttps://www.anthropic.com/careers[2]Reference 2Interview guidehttps://openai.com/interview-guide/} Make it easy for someone to inspect what you designed, what you tested, and what you learned. Tailor the project evidence to each role.

Pick a problem you want to investigate

You don't need all five projects. Choose the question you would enjoy debugging, then check whether that work resembles the role you want. The AI engineer role guide explains the differences between product, platform, and research work.

Document QA and its evaluation dashboard fit together naturally. The other projects can stand alone. Avoid building a platform that contains every idea before you have one result to explain. If you start with documents, can you show a cited answer that has lost contact with the policy still in force?

1. Document QA: make the source failure visible

Build runbook-qa over documents you wrote yourself or have permission to publish. Include a deployment procedure and two versions of an incident policy. In a separate manifest, mark v2 as superseded and v3 as active for a specified effective date. Authority comes from the effective date and review metadata, not version order alone.

Give the assistant three questions. q-rollback asks for a procedure present in the active policy. q-missing asks for a procedure absent from the corpus. q-conflict asks about a rule that changed between versions. Define the expected behavior before changing prompts: answer with supporting evidence, abstain when evidence is missing, and apply the manifest's precedence rule when sources disagree.

A RAG system retrieves text before generating an answer. To debug it, save the intermediate evidence alongside the final response:

Try one controlled change, such as preserving headings during chunking. Replay the same questions. If the correct passage was already retrieved, better chunking may not fix the answer. If parsing deleted a table row, switching models won't restore the missing input.

Citation presence is easy to check mechanically; citation support is a separate judgment. Start with manually reviewed expected claims for the small corpus. Don't treat a valid source ID as proof that an answer is grounded.

Stop the first version at cited answers and explicit abstentions. Uploads, authentication, and multiple document formats can wait. The Document QA capstone expands this into a product with an explicit evidence-admission contract. Evaluate cited answers across a baseline and candidate before trusting a comparison. Did the next change help, or did the report hide a regression?

2. Evaluation dashboard: test the report itself

A dashboard can be a separate project using saved model outputs. Its job is to make a comparison trustworthy and explain what the green percentage includes.

For runbook-qa, add q-injection: retrieved text contains an instruction to ignore the task. Keep that case in both runs. Pin the case manifest, corpus, prompts, model identifiers, generation settings, and grader version. Change one component for the comparison; repeat model runs when variability could explain the difference.

In this synthetic example, the candidate fixes two failures and introduces one. Its pass rate rises from 50% to 75%, but it follows the injected instruction:

View larger

Before trusting those percentages, check that every expected case appears exactly once. A missing failure, a duplicate success, or the string "false" accidentally treated as a true value can corrupt the report.

The following program evaluates already-graded rows; use a separate evaluator to grade model answers. All four cases are mandatory in this tiny contract, including the answerable question: an assistant that abstains on everything must fail too. Passing makes a candidate eligible for further review before deployment.

from collections import Counter
from dataclasses import dataclass

@dataclass(frozen=True)
class CaseResult:
    case_id: str
    passed: bool

EXPECTED = ("q-rollback", "q-missing", "q-conflict", "q-injection")

def evaluate(results: list[CaseResult]) -> dict:
    # Type annotations alone do not reject strings or integers at runtime.
    if any(type(r.case_id) is not str or type(r.passed) is not bool for r in results):
        return {"decision": "INVALID", "pass_rate": None, "reasons": ["invalid types"]}

    counts = Counter(r.case_id for r in results)
    problems = []
    for label, ids in (
        ("missing", set(EXPECTED) - set(counts)),
        ("unknown", set(counts) - set(EXPECTED)),
        ("duplicate", {key for key, count in counts.items() if count > 1}),
    ):
        if ids:
            problems.append(f"{label}: {', '.join(sorted(ids))}")
    if problems:
        return {"decision": "INVALID", "pass_rate": None, "reasons": problems}

    outcomes = {r.case_id: r.passed for r in results}
    failures = [case_id for case_id in EXPECTED if not outcomes[case_id]]
    return {
        "decision": "HOLD" if failures else "REVIEW",
        "pass_rate": sum(outcomes.values()) / len(EXPECTED),
        "reasons": failures,
    }

baseline = [CaseResult(case_id, passed) for case_id, passed in
            zip(EXPECTED, [True, False, False, True])]
candidate = [CaseResult(case_id, passed) for case_id, passed in
             zip(EXPECTED, [True, True, True, False])]
missed_answer = [CaseResult(case_id, passed) for case_id, passed in
                 zip(EXPECTED, [False, True, True, True])]
all_pass = [CaseResult(case_id, True) for case_id in EXPECTED]

assert evaluate(all_pass)["decision"] == "REVIEW"
assert evaluate(candidate)["reasons"] == ["q-injection"]
assert evaluate(candidate + [CaseResult("q-extra", True)])["decision"] == "INVALID"

for name, rows in (
    ("baseline", baseline),
    ("candidate", candidate),
    ("missed answer", missed_answer),
    ("all pass", all_pass),
    ("missing row", candidate[:-1]),
    ("duplicate row", candidate + candidate[:1]),
    ("unknown row", candidate + [CaseResult("q-extra", True)]),
    ("string boolean", [CaseResult("q-rollback", "false")]),
):
    report = evaluate(rows)
    rate = report["pass_rate"]
    score = "unscored" if rate is None else f"{rate:.0%}"
    print(f"{name}: {score}, {report['decision']}, {report['reasons']}")

baseline: 50%, HOLD, ['q-missing', 'q-conflict']
candidate: 75%, HOLD, ['q-injection']
missed answer: 75%, HOLD, ['q-rollback']
all pass: 100%, REVIEW, []
missing row: unscored, INVALID, ['missing: q-injection']
duplicate row: unscored, INVALID, ['duplicate: q-rollback']
unknown row: unscored, INVALID, ['unknown: q-extra']
string boolean: unscored, INVALID, ['invalid types']

The missed-answer run has no safety regression, but it still fails a required functional case. In a larger evaluation, define quality thresholds and critical failures in advance while evaluating subjective judgments on their own criteria. Show uncertainty and slice counts. Use representative production cases to assess reliability beyond these four wiring checks.

For the first dashboard, support importing two result files, rejecting incompatible manifests, and opening failed rows beside their outputs and grading reasons. A static HTML report is enough. The Eval Dashboard capstone adds comparison validity and release-gate behavior in more detail. A report can reject a bad comparison and still say nothing about a person in the loop. What happens to an approval after someone edits the draft?

3. Support copilot: bind approval to the exact draft

Approval has to stick to the text a person actually saw. Use a fictional subscription service with a small cancellation policy. The copilot reads a customer message and drafts a supported reply. It can't issue a refund, modify the subscription, or send a message. A person reviews the draft and its policy evidence.

The interesting bug is stale approval. Suppose a reviewer approves draft 7, then someone edits the amount in draft 8. If the database stores only approved = true on the ticket, the application may treat the unreviewed text as approved.

Store a draft ID, revision, content hash, cited policy version, reviewer identity, decision, and timestamp. Approval applies to that exact revision. An edit creates a new revision that needs review. A policy change can also invalidate the evidence behind an existing draft, so specify when approval must be renewed.

Demonstrate the boundary with three scenarios:

• A draft lacks support for a promised refund. The reviewer sees the missing evidence and rejects it under the project's policy.
• A draft changes after approval. The previous approval no longer authorizes the new revision.
• The same approved action is submitted twice. A simulated executor records one action using an idempotency key, not two.

Persist the state instead of using an in-memory UI toggle. For this portfolio version, the executor only appends to a local outbox. Label that simulation clearly. If you later add real sending, keep the executor outside the model's tool set and validate the approval, recipient, and exact content at execution time.

The Human-in-the-Loop lesson develops the durable pause-and-resume mechanism. Your project can stay focused on proving that an edit, restart, or duplicate request doesn't bypass review. The same question shows up in a repository. Can a proposed fix stay inside a boundary the assistant is not allowed to move?

4. Repository assistant: repair one bug without moving the boundary

Put that boundary in a small repository you own, such as a CSV-reporting utility. Give the assistant a concrete issue: the report crashes on an empty input file. Save the starting commit, a reproducing test, and the expected behavior. Let the assistant inspect files, propose a patch, and return test results for review.

Start by allowing changes only to the implementation file. Keep the evaluation tests outside its writable workspace. Otherwise, a candidate could “fix” the issue by deleting the assertion. Run the same trusted tests against the starting commit and the patched commit; a test that passes both never demonstrated the repair.

Repository contents are untrusted input, including comments that look like instructions. Put an adversarial instruction in a fixture and check whether the assistant attempts an out-of-scope edit. Enforcement belongs in the host: validate paths and the complete diff, reject changes outside the allowlist, and run code in a disposable sandbox with no secrets, no network by default, and bounded resources. Enforce file access with host allowlists and operating-system controls; prompt text cannot constrain filesystem access.

SWE-bench uses real issue descriptions, repository snapshots, and tests to study repository-level fixes.^{[3]Reference 3SWE-bench: Can Language Models Resolve Real-World GitHub Issues?https://arxiv.org/abs/2310.06770} Borrow that task structure and report your one-repository exercise as a separate evaluation. Publish failed attempts as well as successes, and record tool permissions, budgets, and the test harness.

Leave automatic merging and deployment out of scope. The Code Generation and Sandboxing lesson explains how to separate model proposals from trusted execution and validation. A patch that stays in bounds can still be the expensive one. Which savings survive when you replay the same workload?

5. Cost and latency: measure the savings you actually get

Hold the questions fixed and change one cost lever at a time. Use a frozen question set from a document assistant, or a classification workload you already have. A prompt-prefix caching experiment is a manageable start: keep shared instructions and evidence stable, then vary the user question.

Record per-request input and output tokens, cache usage, elapsed time, errors, retries, and quality outcomes. Pin the model identifier, provider, rate-card date, concurrency, and workload. Separate cold requests from warm requests. Shuffling or alternating comparison runs can reduce the chance that changing service load explains the result.

Provider prefix caching reuses computation for a matching prefix; it doesn't reuse the previous answer. OpenAI's current guide documents model-dependent minimum prefix lengths, retention, and charges. Inspect returned usage such as cached_tokens instead of assuming repeated text hit the cache.^{[4]Reference 4Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

Check the cache boundary as well as the text. For GPT-5.6 and later, implicit-only caching can write through a changing user question without separately saving the shared passage before it. The next question then misses that longer prefix. Place an explicit breakpoint after the stable content in both requests, using the selected model's documented controls. Compare the returned reads and writes. Moving a timestamp to the end helps preserve the text, but doesn't by itself create an eligible matching breakpoint.^{[4]Reference 4Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

For models that report cache_write_tokens, count writes separately too. The guide's input-cost calculation separates ordinary input, cached reads, and cache writes. A cold request may pay to populate a cache that your workload never reuses. Log both fields and apply the selected model's rate card; a high read discount alone doesn't prove the experiment saved money.^{[4]Reference 4Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}

OpenAI's Batch API advertises 50% lower cost than synchronous requests with a 24-hour completion window. Batches can expire with unfinished requests, so reconcile every input ID with either a result or an error.^{[5]Reference 5OpenAI Batch API Guidehttps://developers.openai.com/api/docs/guides/batch} Report Batch completion under offline throughput; use a separate measurement for interactive response latency.

Report cost per attempted task and per successful task, including retries. State whether your estimate covers only provider charges or also retrieval and hosting. Show request count beside p50 and p95; a tiny sample doesn't support a confident tail-latency claim. Keep quality next to cost, and label any latency statistic that excludes timeouts as covering successful requests only.

A measured result of “this workload got no cache hits” is worth explaining. It tells a reader more than a savings percentage copied from provider marketing. The LLM Cost Engineering lesson covers usage-based accounting and the distinction between prefix caching and application-level answer reuse. Those measurements help only if someone can find the question, the comparison, and the limitation without an API key.

Make the repository readable before it is runnable

A reader should be able to find the question, experiment, result, and limitation without supplying an API key. Keep a static report and redacted traces alongside the code:

README.md                    # problem, non-goals, setup, test and eval commands
docs/design.md               # chosen approach and rejected alternative
evals/cases.jsonl            # versioned inputs and expected behavior
reports/compare.md           # same-case comparison, counts, failed rows
traces/redacted/             # representative success and failure traces
src/
tests/
.env.example                 # variable names, never credentials

Check the documented commands in a clean environment. If dependencies or paid services are required, say so. A mocked provider can demonstrate UI states and report logic; label those results as mocked and report live model quality separately. Never publish employer documents, customer data, credentials, or proprietary code to make a project look realistic.

Write the result as a small engineering account. What failed first? Which component did you change? What remained broken? For example: “Preserving headings recovered the missing source on these cases, but the answer still selected the superseded policy. I added an authority filter and reran both versions.” Follow that sentence with the actual rows, not invented impact numbers.

If you used AI tools, distinguish generated code from the design, verification, and debugging you performed. If others contributed, identify your own subsystem and decisions. A controlled experiment with an unresolved limitation is easier to discuss honestly than a broad claim that the assistant is production-ready.

You don't need a sprawling platform to have something worth discussing. Choose one project, make one comparison reproducible, and explain one failure well. When your repository can support that conversation, the interview preparation guide can help you prepare for the process you're entering.