Python for AI Engineering

Most AI engineering starts before the model call.

You have files on disk. You have examples with labels. You have model predictions that may be wrong, malformed, or missing. Before you can trust a training run, an evaluation report, or a demo, you need Python code that can read those examples, check the shape of each row, compute one clear metric, and fail when the input is broken.

Python's job here is to turn loose files into named data, named data into measurements, and measurements into something another person can reproduce. This chapter builds that first loop from the ground up: read a JSONL evaluation file, parse each row into a typed object, score exact-match accuracy, and protect the behavior with tests.^[1]

Git and Docker protected the file and runtime. Now we open the scorer and build the code those earlier chapters were carrying. This chapter also hardens that behavior into a reproducible contract with tests, snapshots, leakage checks, and CI.

The first useful loop

For this Python chapter, don't think about model APIs yet. Think about the smallest reliable AI workflow:

That sequence is the foundation for later work. RAG evaluations, fine-tuning datasets, judge results, and agent traces all need the same habit: make the data visible before turning it into a number.

Here is the whole flow:

The diagram stays small on purpose. A beginner should be able to trace every arrow before the curriculum moves into arrays, tensors, training loops, and LLM systems.

Start with the three-row contract you already own

The Git and Docker chapters already gave you a protected three-row evaluation file at eval/support_tickets.jsonl. We reuse it here so the Python scorer operates on the exact same contract the previous lessons locked down.

Two predictions match. One doesn't.

The exact-match score is:

$$\frac{2}{3} = 0.667$$

That hand calculation still matters. The Python code must produce this number on this file because the earlier Git and Docker chapters locked the same contract. This is the same fixture the pre-commit gate, the Docker container, and the later CI workflow will protect.

The file lives in the eval directory the repo already tracks

JSONL (JSON Lines) is the format used throughout the curriculum because it streams cleanly, supports appending new eval rows, and stays human-inspectable with cat or head.

Inspect one line (the order-102 failure)

It carries the three fields the scorer contract requires:

The row is a valid failure case. The scorer must treat it as incorrect. A row missing prediction or carrying a non-string value must fail before any metric is computed.

Name the pieces before writing code

The script uses a few Python ideas. Keep them attached to the evaluation task.

A type hint helps humans and editors read your intent. It doesn't validate JSON from disk by itself. A static type checker such as mypy or pyright verifies hints before the code runs, and a linter and formatter such as ruff keeps the style consistent. None of them inspect the actual file at runtime, so the runtime checks below still matter because real files can contain missing fields, wrong types, blank lines, or invalid JSON.

Build the scorer

The eval/support_tickets.jsonl file is already present from the Git chapter and already travels through the Docker setup. Create the Python module that reads it. Put this code in python_eval_demo.py.

If you are testing this block in a blank scratch directory, create the same three-row fixture first:

Path comes from the standard library module pathlib. It turns a file path into an object you can read, check, and pass around. Using Path instead of a plain string makes file operations explicit and behaves the same way on Linux, macOS, and Windows.

Read the code in five blocks:

1. Example names one row.
2. require_str checks one field.
3. parse_example turns text into a checked row.
4. load_examples reads the file and rejects empty input.
5. score_examples turns row checks into one metric.

Run it

The printed score matches the hand calculation. Use that as your first sanity check.

Trace the order 102 row through the script

The best way to read the program is to follow one row all the way through.

The word "averaged" is literal here. In Python, True behaves like 1 and False behaves like 0 when summed:

The article rounds that final value to three decimal places, so it prints 0.667.

Read the code by boundaries

Good AI scripts make each boundary explicit. If the final number looks wrong, you can walk backward through the pipeline and inspect the smallest suspicious step.

That structure is more important than exact-match accuracy itself. Exact match is a simple first metric. Later chapters will use richer metrics, tensors, loss functions, eval sets, and model outputs. The boundary habit stays the same.

When debugging, print one value at each boundary:

Don't debug the aggregate score first. Debug one row first.

Make bad input fail loudly

Now break the input on purpose. Run this in a small scratch file or REPL after saving python_eval_demo.py in the same folder:

Both failures are useful. The first row is missing a required field. The second row has the right field name but the wrong value type. In both cases, the script stops before it prints a misleading metric.

Treat this as a production habit. Silent bad data can make an evaluation dashboard look healthy while the input is nonsense.

Turn the script into a command

The first version hardcodes the protected eval/support_tickets.jsonl path so the example stays easy to run on the curriculum fixture. The next version should accept the file path from the command line so the same module can score any later eval set.

Add import sys near the top of the file, then replace main():

sys.argv is a list that holds the words you type after the script name. sys.argv[1] is the first argument, which we expect to be the JSONL file path.

Run it on the curriculum fixture

That small change makes the script reusable across any JSONL file that follows the same three-field schema. A teammate (or your future self on a fresh clone) can point it at the protected fixture or any later eval set and get a trustworthy number. The earlier fixture tests still pass because the behavior on this exact file is locked.

The input contract is now easy to state:

Add tests before the script grows

Once python_eval_demo.py is a module, tests can import its functions directly. Save the test file as tests/test_python_eval_demo.py so the project uses a standard pytest layout.

tmp_path is a pytest fixture. pytest automatically creates a temporary directory and passes it as the tmp_path argument, so the test doesn't leave files behind after it finishes.

Run the tests

Run pytest through an isolated tool command instead of installing packages into your global Python environment.

Expected output

These tests protect the three behaviors most likely to break first:

The point isn't to write many tests. The point is to test the boundaries that would make a metric lie.

Notice that the tests don't call main(). They test the smaller functions. That makes failures easier to understand. If parse_example fails, you know the parsing boundary broke. If exact_match fails, you know the scoring rule broke.

The same checks are easy to run directly while learning:

Make the tests strong enough for ML work

Three unit tests catch parser mistakes. Real ML evaluation code usually needs four more protections: prompt snapshots, simple property checks, train/eval leakage detection, and seeded CI runs.^[2][3]

Snapshot the prompt template

Prompt drift is one of the fastest ways to make a metric move without anybody noticing why. Lock the exact prompt string with a small snapshot-style test:

If one word changes, the test fails before the scorer or model call hides the reason.

Add a property check and a leakage detector

ML tests should protect invariants, not only one fixed example. Accuracy should never leave [0, 1], and the eval set should never overlap with the training prompts:

This is how you turn "that score looks suspiciously high" into one reproducible red test instead of a debugging argument.^[4]

Seed test runs and let CI enforce them

Notebook state and random seeds are another common source of fake progress. Put the seeding contract in one place, then run the same test command in CI:

Run the fuller suite with:

Now the contract is stronger:

Common bugs and how to find them

Beginner Python evaluation scripts usually fail in predictable ways.

Notebook-only code becomes fragile for the same reason. A notebook can hide state in old cells, local paths, and copied snippets. A small module with tests is easier to rerun and review.

Practice on the same file

Try these in order. Predict the result before running code.

1. Add a fourth row where prediction equals expected.
2. Predict the new exact-match score.
3. Add extra spaces around one correct prediction.
4. Confirm exact_match still treats it as correct.
5. Replace one prediction with the number 7.
6. Confirm parse_example rejects that row.
7. Create an empty file and confirm load_examples rejects it.
8. Change main() so the file path comes from sys.argv[1].

Use these checks after you try:

If your answer only says "the code works," make it more specific. Name the input, the checked fields, the metric, and the failure behavior.

How this becomes production Python

This first script stays deliberately small, but it points at the same Python skills used in production AI systems.

The important part is the boundary, not the library. Whether the next system uses a dataclass, a Pydantic model, an async client, or a cache, the script should still make inputs explicit, validate shape early, and fail before a bad result looks trustworthy.

Before moving on, you should be able to explain the finished script in four sentences:

1. The input is a JSONL file with one evaluation example per line.
2. The parser checks that each row has string fields named prompt, expected, and prediction.
3. The metric scores one row with exact match, then averages the row scores.
4. The tests prove whitespace handling, missing fields, and empty files behave as expected.

What this pattern unlocks

This chapter isn't about memorizing every Python feature. It's about one engineering pattern:

Read a data file, check one row, turn it into a named object, compute one metric, and protect the behavior with a test.

You will reuse that pattern constantly. NumPy will turn rows into arrays. PyTorch will turn arrays into tensors and gradients. Retrieval systems will turn documents into ranked candidates. Agent systems will turn tool calls into traces. Each later step is larger, but the first question stays familiar: what enters, what shape should it have, what operation happens, and what fails before the result can be trusted?^[5][6]

Mastery check

Key concepts

• functions and modules
• dataclasses and type hints
• pathlib file IO
• JSONL datasets
• runtime validation
• pytest tests
• prompt snapshot tests
• property-style checks for metric invariants
• train/eval leakage detection
• seeded reproducibility and CI gates
• command-line scripts
• error handling

Evaluation rubric

• Foundational: Explains Python evaluation scripts as repeatable data pipelines from file path to checked row to metric
• Intermediate: Builds a dataclass parser, file loader, exact-match scorer, command-line entrypoint, and pytest suite with visible output
• Advanced: Catches missing fields, wrong types, empty files, prompt drift, train/eval leakage, and hidden notebook state before scoring

Follow-up questions

Common pitfalls

• Notebook-only code works once, then breaks when another person changes paths, rerun order, or input shape.
• Type hints help readers, but they don't validate runtime JSON on their own. Parse and check required fields before scoring.
• A metric is only useful if malformed rows and empty files fail before the score is printed.
• Prompt wording can drift silently unless one test locks the exact string the scorer or judge sees.
• A high score can be fake if the training prompts overlap the eval prompts or if random seeds change between runs.