Capstone: Fine-Tuned Classifier

The evaluation dashboard in the previous capstone made one rule visible: a release is blocked by the failures that matter, even when its aggregate score looks good. Now apply that rule to a learned model.

Your product already has a document QA service that answers policy questions with permitted citations. A later production-agent capstone will draft an access response and request approval. This capstone builds the intake classifier that protects that future agent:

Given a new access ticket, should it enter human_review_now, or may it enter the guarded agent workflow?

Consider this ticket:

Missing this exception isn't the same as sending one routine key-rotation question to a human. The classifier therefore does routing, not authorization. It never decides access eligibility, sends a reply, or changes permissions. It protects the agent entry point.

The finished artifact has:

• a versioned label guide and split rule
• a baseline whose failure justifies learning contextual representations
• a small PyTorch encoder training loop that exposes the mechanics
• a practical Hugging Face fine-tuning recipe for a pretrained encoder
• held-out threshold and slice gates
• a serving bundle the production-agent capstone can consume

BERT popularized the pattern of adapting a pretrained bidirectional encoder to a downstream classification task with a small output layer.^{[1]Reference 1BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} Hugging Face exposes the same production-oriented path through sequence-classification models and training utilities.^{[2]Reference 2Text Classification.https://huggingface.co/docs/transformers/tasks/sequence_classification} The important discipline here isn't selecting a fashionable checkpoint. It's proving that a trained classifier satisfies a routing contract.

Define the routing contract

Use one binary label:

That second row matters. Label 0 does not mean "approve automatically." It means "safe to enter an automation path with its own controls."

The model produces a score. The pinned threshold turns that score into an intake route. Either route preserves the agent's downstream evidence and approval boundaries.

Connect the portfolio artifacts

Start with labeled fixtures that can be reviewed before any model is trained.

A label guide should also name ambiguous cases. Frustrated tone alone isn't a reason for immediate human routing, while a calm message requesting an incident-freeze exception may be. Have support operations adjudicate disagreements and store that policy version alongside any dataset generated from it.

Build an honest dataset

The lab uses a hypothetical teaching dataset, not measured production traffic:

In a real data card, replace every teaching assumption with provenance, consent and retention policy, labeling instructions, class balance, disagreement rate, and known coverage gaps.

Split before learning

Suppose multiple messages from the same workspace account share service names, incident-freeze wording, or a prior access incident. Randomly putting one thread into training and another into validation makes the classifier look better than it will on a new account. Group by account first, then hold out later traffic.

Use validation data to pick features, checkpoints, and threshold. Open the test set once for the final report after choices are frozen. A small, curated fixture set can reveal specific failures, but it can't establish broad production performance by itself.

Establish a cheap baseline

A transformer has to earn its complexity. Start with a deterministic rule or a bag-of-words logistic-regression baseline. If a simple baseline satisfies the routing gate on representative held-out data, shipping a neural model may add maintenance without adding safety.

Here the rule catches an explicit access-override phrase and misses an indirect incident-freeze exception. That failure gives a contextual model a job worth doing.

A TF-IDF plus logistic-regression baseline is a useful next step when the dataset grows: it learns weighted text features while remaining cheap to inspect and deploy.^{[3]Reference 3The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/} Don't claim a fine-tuned encoder is cheaper, faster, or more accurate than prompting, or than this baseline, until you measure those candidates on the same task and hardware.

From logits to a route

An encoder classifier emits logits: one unnormalized score for each class. Softmax converts two logits into class scores that sum to one. A separate decision threshold converts the positive-class score into the business route.

Use numerically stable softmax by subtracting the largest logit before exponentiating.

Call 0.95 a score until held-out reliability evidence supports calling it a probability. A bounded output isn't proof of calibration.

Train the mechanics in PyTorch

Before reaching for a downloaded checkpoint, make the trainable path concrete. The sandbox below builds a tiny word-embedding encoder, averages token vectors, attaches a classification head, and updates every parameter with cross-entropy loss. This is training a text classifier from scratch, not fine-tuning BERT. It exists so you can inspect the mechanics before replacing random embeddings with pretrained representations.

The sandbox can memorize six tiny training examples. That's not deployment evidence. The wiring is what matters here: text becomes tokens, tokens become pooled representations, a head produces logits, and gradients update those weights. Held-out evaluation remains the release evidence.

Fine-tune a pretrained encoder

For the capstone artifact, replace random embeddings with a pretrained encoder such as DistilBERT and fine-tune it for two labels. BERT-style encoders read context on both sides of a token and are designed to be adapted to tasks such as sequence classification.^{[1]Reference 1BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} In the Transformers task interface, AutoModelForSequenceClassification supplies the classification head while Trainer runs optimization and evaluation.^{[2]Reference 2Text Classification.https://huggingface.co/docs/transformers/tasks/sequence_classification}

This is a project script rather than a marked copy-runnable cell: it requires your local JSONL splits, a downloaded checkpoint, and Python packages such as transformers, datasets, evaluate, accelerate, and numpy. Each JSON Lines (JSONL) row needs text and integer label fields, where 0 means guarded_agent and 1 means human_review_now.

Two boundaries keep this honest:

1. Fine-tuning doesn't replace label review. It fits the guide it receives, including guide mistakes.
2. Highest validation F1 doesn't automatically ship. Prefer a checkpoint metric aligned with the product rule (for example validation recall or required-slice FN count), and still require the exact-receipt gate before shadow traffic.

Select a threshold on a selection set, not the freeze gate

Don't accept a library default threshold without checking its error tradeoff. Choose the threshold on a selection set, then freeze a separate gate receipt for release. Tuning threshold until required FNs disappear on the same six rows you later treat as independent evidence contaminates the gate.

Suppose encoder_v1 exports these selection-set scores for human_review_now:

On the selection set, threshold 0.35 is the only candidate shown that avoids a missed escalation. It sends one angry but routine key-rotation ticket to human review. That tradeoff is acceptable only if queue capacity allows it. After you lock 0.35, score a frozen gate receipt (held-out fixtures that were not used to choose the threshold) and run the executable gate below. Nested cross-validation is the production version of the same split.

Don't report 1.00 recall from three positive selection fixtures as proof of production reliability. It's evidence for this selection slice, not for unseen traffic.

Make the hard gate executable

The dashboard capstone compared runs only when dataset, grader, corpus, and fixture set matched. Carry that discipline here. A classifier comparison receipt pins its dataset, label guide, split manifest, model, and exact gate fixture IDs before scores are aggregated. Gate fixtures must not be the set used to choose the threshold.

intake_bundle_v1 is blocked because its score for the held-out freeze_window_ticket falls below the initially proposed threshold. intake_bundle_v2 keeps the same trained encoder, pins the reviewed threshold, and passes the separate access-intake-gate-v1 receipt before it becomes eligible for shadow traffic, not immediate autonomous routing. A threshold change versions the serving bundle, not the model weights.

Lowering the threshold isn't automatically the right fix. If false positives overwhelm reviewers, the executable gate holds on max_false_positives rather than only on required-slice FNs. For this teaching receipt, the important moves are: separate threshold selection from the freeze gate, refuse to hide a required-slice miss inside an average, and refuse to pad the comparison with easy rows.

Audit scores without overclaiming calibration

A threshold can be useful even when the score isn't a well-calibrated probability. Calibration asks whether tickets scored near 0.70, across enough held-out examples, need human review about 70 percent of the time. Modern neural classifiers can be miscalibrated; temperature scaling is one common post-training adjustment evaluated on held-out data.^{[4]Reference 4On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599}

Six fixtures are enough to demonstrate a calculation, not to justify operator-facing percentages:

The high bucket exposes why calibrated language matters: an average score of 0.80 with one positive among two tickets isn't evidence that 0.80 means an 80 percent escalation rate. Keep displaying a routing action until a larger retained set supports probability claims.

Package the serving contract

The shipped product isn't only model weights. It's a pinned bundle:

The endpoint should fail closed. A blank ticket, unavailable model, or malformed score is a reason to request human review rather than send uncertain traffic into the agent.

Hand off to the production agent

Only the guarded_agent route from the pinned bundle reaches the later production-agent capstone. That agent will still retrieve permitted policy evidence, draft rather than execute an access restore, and ask for approval before any consequential action. The classifier gives it an intake boundary, not extra authority.

This contract creates a portfolio story with traceable boundaries:

1. Document QA supplies evidence-bound policy answers.
2. Exact-receipt evaluation rows and gates expose failures before release.
3. The classifier blocks high-risk intake from automation.
4. The agent uses only admitted routine intake and still needs approval for consequential actions.

Release checklist

Don't ship a classifier project as a notebook screenshot. Keep these files or equivalent artifacts together:

Use shadow traffic before changing live routing. Record scores and proposed routes, retain human decisions, and compare missed-escalation rate by required slice. Roll back to manual review when a required slice misses its agreed floor; don't wait for average accuracy to fall.

Practice: break the intake contract

Run the relevant cells again after each mutation. Revert one mutation before starting the next.

1. Inspect intake_bundle_incomplete. Why does it hold even though every remaining score clears threshold 0.35?
2. Inspect intake_bundle_padded, intake_bundle_duplicated, and intake_bundle_drifted. Why can't easy extras, duplicate rows, or a newer dataset silently change the release decision?
3. Raise intake_bundle_v2 threshold from 0.35 to 0.50. Which required fixture blocks shadow eligibility?
4. Inspect intake_bundle_fp_flood. Why does it hold even though required-slice positives still clear a low threshold?
5. Change fallback_route to guarded_agent. Which serving-fallback.py assertions fail, and why is that unsafe?
6. Remove the bundle-version comparison from admitted_by_pinned_bundle. Which stale route artifact now reaches the agent?

Next artifact boundary

The classifier artifact is consumable rather than merely discussable: intake_bundle_v2 deploys encoder_v1 with a reviewed threshold, admits routine tickets to a guarded workflow, and blocks risky or failed inputs to human review. The later agent capstone must respect that boundary while combining it with policy retrieval, approval gates, traces, and executable evaluation.