Python for AI Engineering

Most AI engineering starts before the model call.

There are files on disk, labeled examples, and model predictions that may be wrong, malformed, or missing. Before you can trust a training run, an evaluation report, or a demo, you need Python code that can read those examples, check the shape of each row, compute one clear metric, and fail when the input is broken.

Python turns loose files into named data, named data into measurements, and measurements into something another person can reproduce. Build that first loop from the ground up: read a JSONL evaluation file, parse each row into a typed object, score exact-match accuracy, and protect the behavior with tests.^{[1]Reference 1The Python Tutorial.https://docs.python.org/3/tutorial/}

Git and Docker protected the file and runtime. Now we open the scorer and build the code those earlier chapters were carrying. The work also hardens that behavior into a reproducible contract with tests, snapshots, leakage checks, and CI.

The first useful loop

For this Python chapter, don't think about model APIs yet. Think about the smallest reliable AI workflow:

That sequence is the foundation for later work. RAG evaluations, fine-tuning datasets, judge results, and agent traces all need the same habit: make the data visible before turning it into a number.

The whole flow:

Start with the three-row contract you already own

The Git and Docker chapters already gave you a protected three-row evaluation file at eval/access_requests.jsonl. We reuse it here so the Python scorer operates on the exact same contract the previous lessons locked down.

Two predictions match. One doesn't.

The exact-match score is:

$$\frac{2}{3} = 0.667$$

That hand calculation still matters. The Python code must produce this number on this file because the earlier Git and Docker chapters locked the same contract. This is the same fixture the pre-commit gate, the Docker container, and the later CI workflow will protect.

The file lives in the eval directory the repo already tracks

JSONL (JSON Lines) is the format used throughout the curriculum because it streams cleanly, supports appending new eval rows, and stays human-inspectable with cat or head.

Inspect one line (the run-102 failure)

It carries the three fields the scorer contract requires:

The row is a valid failure case. The scorer must treat it as incorrect. A row missing prediction or carrying a non-string value must fail before any metric is computed.

Name the pieces before writing code

The script uses a few Python ideas. Keep them attached to the evaluation task.

A type hint helps humans and editors read your intent. It doesn't validate JSON from disk by itself. A static type checker such as mypy or pyright verifies hints before the code runs, and a linter and formatter such as ruff keeps the style consistent. None of them inspect the actual file at runtime, so the runtime checks below still matter because real files can contain missing fields, wrong types, blank lines, or invalid JSON.

Build the scorer

The eval/access_requests.jsonl file is already present from the Git chapter and already travels through the Docker setup. Create the Python module that reads it. Put this code in python_eval_demo.py.

If you're testing this block in a blank scratch directory, create the same three-row fixture first:

Path comes from the standard library module pathlib. It turns a file path into an object you can read, check, and pass around. Using Path instead of a plain string makes file operations explicit and behaves the same way on Linux, macOS, and Windows.

Read the code in five blocks:

1. Example names one row.
2. require_str checks one field.
3. parse_example turns text into a checked row.
4. load_examples reads the file and rejects empty input.
5. score_examples turns row checks into one metric.

Run it

The printed score matches the hand calculation. Use that as your first sanity check.

Trace the access request 102 row through the script

Read the program by following one row all the way through.

The word "averaged" is literal here. In Python, True behaves like 1 and False behaves like 0 when summed:

The article rounds that final value to three decimal places, so it prints 0.667.

Read the code by boundaries

Good AI scripts make each boundary explicit. If the final number looks wrong, you can walk backward through the pipeline and inspect the smallest suspicious step.

That structure is more important than exact-match accuracy itself. Exact match is a simple first metric. Later chapters will use richer metrics, tensors, loss functions, eval sets, and model outputs. The boundary habit stays the same.

When debugging, print one value at each boundary:

Don't debug the aggregate score first. Debug one row first.

Make bad input fail loudly

Now break the input on purpose. Run this in a small scratch file or REPL after saving python_eval_demo.py in the same folder:

Both failures are useful. The first row is missing a required field. The second row has the right field name but the wrong value type. In both cases, the script stops before it prints a misleading metric.

Treat this as a production habit. Silent bad data can make an evaluation dashboard look healthy while the input is nonsense.

Turn the script into a command

The first version hardcodes the protected eval/access_requests.jsonl path so the example stays easy to run on the curriculum fixture. The next version should accept the file path from the command line so the same module can score any later eval set.

Add import sys near the top of the file, then replace main():

sys.argv is a list that holds the words you type after the script name. sys.argv[1] is the first argument, which we expect to be the JSONL file path.

Run it on the curriculum fixture

That small change makes the script reusable across any JSONL file that follows the same three-field schema. A teammate (or your future self on a fresh clone) can point it at the protected fixture or any later eval set and get a trustworthy number. The earlier fixture tests still pass because the behavior on this exact file is locked.

The input contract is now easy to state:

Add tests before the script grows

Once python_eval_demo.py is a module, tests can import its functions directly. Save the test file as tests/test_python_eval_demo.py so the project uses a standard pytest layout.

tmp_path is a pytest fixture. pytest automatically creates a temporary directory and passes it as the tmp_path argument, so the test doesn't leave files behind after it finishes.

Run the tests

Run pytest through an isolated tool command instead of installing packages into your global Python environment.

Expected output

These tests protect the three behaviors most likely to break first:

Don't write tests for volume. Test the boundaries that would make a metric lie.

Notice that the tests don't call main(). They test smaller functions, so failures point at the boundary that broke: parse_example for parsing and exact_match for scoring.

The same checks are easy to run directly while learning:

Make the tests strong enough for ML work

Three unit tests catch parser mistakes. Real ML evaluation code usually needs four more protections: prompt snapshots, simple property checks, train/eval leakage detection, and seeded CI runs.^{[2]Reference 2pytest Documentationhttps://docs.pytest.org/[3]Reference 3Hypothesis Documentationhttps://hypothesis.works/}

Snapshot the prompt template

Prompt drift is one of the fastest ways to make a metric move without anybody noticing why. Lock the exact prompt string with a small snapshot-style test:

If one word changes, the test fails before the scorer or model call hides the reason.

Add a property check and a leakage detector

ML tests should protect invariants rather than one fixed example. Accuracy must equal matches / n with matches in 0..n, which is stronger than checking only that it falls in [0, 1]. The eval set should never share exact prompts with training. Exact-prompt intersection is a first detector only: it won't catch near-duplicates, normalized IDs, or label joins, so treat a clean result as "no identical prompt reuse," not "no leakage possible."

This turns a suspiciously high score into a reproducible failing test.^{[4]Reference 4The Machine Learning Reproducibility Checklisthttps://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf}

Seed test runs and let CI enforce them

Notebook state and random seeds can make generated test cases change between runs. Put the seed in one pytest fixture, then use the same command locally and in CI:

PYTHONHASHSEED stabilizes Python's hash randomization, but it can't replace random.seed or later framework seeds such as torch.manual_seed. Set hash seed before Python starts; seed pseudorandom generators inside the test process.

Run scripts/eval-gate.sh in a clean clone and from whichever CI system owns the repository.

Now the contract is stronger:

Common bugs and how to find them

Beginner Python evaluation scripts usually fail in predictable ways.

Notebook-only code becomes fragile for the same reason. A notebook can hide state in old cells, local paths, and copied snippets. A small module with tests is easier to rerun and review.

Practice on the same file

Try these in order. Predict the result before running code.

1. Add a fourth row where prediction equals expected.
2. Predict the new exact-match score.
3. Add extra spaces around one correct prediction.
4. Confirm exact_match still treats it as correct.
5. Replace one prediction with the number 7.
6. Confirm parse_example rejects that row.
7. Create an empty file and confirm load_examples rejects it.
8. Change main() so the file path comes from sys.argv[1].

Use these checks after you try:

Name the input, checked fields, metric, and failure behavior rather than stopping at "the code works."

The reusable pattern

Don't memorize every Python feature. Practice one engineering pattern:

Read a data file, check one row, turn it into a named object, compute one metric, and protect the behavior with a test.

You'll reuse that pattern constantly. NumPy turns rows into arrays, PyTorch turns arrays into tensors and gradients, retrieval systems turn documents into ranked candidates, and agent systems turn tool calls into traces. Each later step is larger, but the first question stays familiar: what enters, what shape should it have, what operation happens, and what fails before the result can be trusted?^{[5]Reference 5Array programming with NumPy.https://www.nature.com/articles/s41586-020-2649-2[6]Reference 6PyTorch: An Imperative Style, High-Performance Deep Learning Library.https://arxiv.org/abs/1912.01703}

Python engineering checklist

Before sharing the scorer, run it against the committed JSONL, confirm malformed, empty, and wrong-typed rows fail before scoring, run the focused tests, and invoke the CLI from a clean working directory. The receipt should name input row count and metric so a teammate can distinguish code drift from fixture drift.