Capstone: Fine-Tuned Classifier

The evaluation dashboard in the previous capstone made one rule visible: a release is blocked by the failures that matter, even when its aggregate score looks good. Now apply that rule to a learned model.

Your product already has a document QA service that answers policy questions with permitted citations. The next product will be an agent that can draft a support response and request approval. This capstone builds the intake classifier in between:

Given a new refund-support ticket, should it enter human_review_now, or may it enter the guarded agent workflow?

Consider this ticket:

Missing this exception is not the same as sending one harmless package-status ticket to a human. The classifier therefore does routing, not authorization. It never decides refund eligibility, sends a reply, or moves money. It protects the agent entry point.

By the end, you will have:

• a versioned label guide and split rule
• a baseline whose failure justifies learning contextual representations
• a small PyTorch encoder training loop that exposes the mechanics
• a practical Hugging Face fine-tuning recipe for a pretrained encoder
• held-out threshold and slice gates
• a serving bundle the production-agent capstone can consume

BERT popularized the pattern of adapting a pretrained bidirectional encoder to a downstream classification task with a small output layer.^[1] Hugging Face exposes the same production-oriented path through sequence-classification models and training utilities.^[2] The important discipline here is not selecting a fashionable checkpoint. It is proving that a trained classifier satisfies a routing contract.

Define The Routing Contract

Use one binary label:

That second row matters. Label 0 does not mean "approve automatically." It means "safe to enter an automation path with its own controls."

The model produces a score. The pinned threshold turns that score into an intake route. Either route preserves the agent's downstream evidence and approval boundaries.

Connect The Portfolio Artifacts

Start with labeled fixtures that can be reviewed before any model is trained.

A label guide should also name ambiguous cases. A frustrated tone alone is not a reason for immediate human routing. A calm message requesting a return-window exception may be. Have support operations adjudicate disagreements and store that policy version alongside any dataset generated from it.

Build An Honest Dataset

This lesson uses a hypothetical teaching dataset, not measured production traffic:

In a real data card, replace every teaching assumption with provenance, consent and retention policy, labeling instructions, class balance, disagreement rate, and known coverage gaps.

Split Before Learning

Suppose multiple messages from the same customer account share product names, carrier wording, or a prior return incident. Randomly putting one thread into training and another into validation makes the classifier look better than it will on a new account. Group by account first, then hold out later traffic.

Use validation data to pick features, checkpoints, and threshold. Open the test set once for the final report after choices are frozen. A small, curated fixture set can reveal specific failures, but it can't establish broad production performance by itself.

Establish A Cheap Baseline

A transformer has to earn its complexity. Start with a deterministic rule or a bag-of-words logistic-regression baseline. If a simple baseline satisfies the routing gate on representative held-out data, shipping a neural model may add maintenance without adding safety.

Here the rule catches an explicit damaged-package phrase and misses an indirect return-window exception. That failure gives a contextual model a job worth doing.

A TF-IDF plus logistic-regression baseline is a useful next step when the dataset grows: it learns weighted text features while remaining cheap to inspect and deploy.^[3] Don't claim a fine-tuned encoder is cheaper, faster, or more accurate than prompting, or than this baseline, until you measure those candidates on the same task and hardware.

From Logits To A Route

An encoder classifier emits logits: one unnormalized score for each class. Softmax converts two logits into class scores that sum to one. A separate decision threshold converts the positive-class score into the business route.

Use numerically stable softmax by subtracting the largest logit before exponentiating.

Call 0.95 a score until held-out reliability evidence supports calling it a probability. A bounded output is not proof of calibration.

Train The Mechanics In PyTorch

Before reaching for a downloaded checkpoint, make the trainable path concrete. The sandbox below builds a tiny word-embedding encoder, averages token vectors, attaches a classification head, and updates every parameter with cross-entropy loss. This is training a text classifier from scratch, not fine-tuning BERT. It exists so you can inspect the mechanics before replacing random embeddings with pretrained representations.

The sandbox can memorize six tiny training examples. That is not deployment evidence. It proves that text becomes tokens, tokens become pooled representations, a head produces logits, and gradients update those weights. Held-out evaluation remains the release evidence.

Fine-Tune A Pretrained Encoder

For the capstone artifact, replace random embeddings with a pretrained encoder such as DistilBERT and fine-tune it for two labels. BERT-style encoders read context on both sides of a token and are designed to be adapted to tasks such as sequence classification.^[1] In the Transformers task interface, AutoModelForSequenceClassification supplies the classification head while Trainer runs optimization and evaluation.^[2]

The following is a project script rather than a marked copy-runnable cell: it requires your local JSONL splits, a downloaded checkpoint, and the transformers, datasets, and evaluate packages. Each JSON Lines (JSONL) row needs text and integer label fields, where 0 means guarded_agent and 1 means human_review_now.

Two boundaries keep this honest:

1. Fine-tuning doesn't replace label review. It fits the guide it receives, including guide mistakes.
2. Highest validation F1 doesn't automatically ship. The candidate still has to pass the high-risk slice gate and serving contract.

Select A Threshold On Validation Rows

Don't accept a library default threshold without checking its error tradeoff. Suppose encoder_v1 exports these validation scores for human_review_now:

On these fixtures, threshold 0.35 is the only candidate shown that avoids a missed escalation. It sends one angry but routine delivery-status ticket to human review. That tradeoff is acceptable only if queue capacity allows it and the same rule survives a larger held-out dataset.

Don't report 1.00 recall from three positive fixtures as proof of production reliability. It is evidence for this test slice, not for unseen traffic.

Make The Hard Gate Executable

The dashboard capstone compared runs only when dataset, grader, corpus, and fixture inventory matched. Carry that discipline here. A classifier comparison receipt pins its dataset, label guide, split manifest, model, and exact validation fixture IDs before scores are aggregated.

intake_bundle_v1 is blocked because its encoder_v1 score for return_window_exception falls below the initially proposed threshold. intake_bundle_v2 keeps the same trained encoder but pins a reviewed threshold, then becomes eligible for shadow traffic, not immediate autonomous routing. A threshold change versions the serving bundle, not the model weights.

Lowering the threshold is not automatically the right fix. If false positives overwhelm reviewers, improve labels or model behavior instead. For this teaching receipt, the important move is refusing to hide a required-slice miss inside an average or to pad the comparison with easy rows.

Audit Scores Without Overclaiming Calibration

A threshold can be useful even when the score is not a well-calibrated probability. Calibration asks whether tickets scored near 0.70, across enough held-out examples, need human review about 70 percent of the time. Modern neural classifiers can be miscalibrated; temperature scaling is one common post-training adjustment evaluated on held-out data.^[4]

Six fixtures are enough to demonstrate a calculation, not to justify customer-facing percentages:

The high bucket exposes why calibrated language matters: an average score of 0.80 with one positive among two tickets is not evidence that 0.80 means an 80 percent escalation rate. Keep displaying a routing action until a larger retained set supports probability claims.

Package The Serving Contract

The shipped product is not only model weights. It is a pinned bundle:

The endpoint should fail closed. A blank ticket, unavailable model, or malformed score is a reason to request human review rather than send uncertain traffic into the agent.

Hand Off To The Production Agent

Only the guarded_agent route from the pinned bundle reaches the next capstone. That agent will still retrieve permitted policy evidence, draft rather than execute a refund, and ask for approval before any consequential action. The classifier gives it an intake boundary, not extra authority.

This contract creates a portfolio story with traceable boundaries:

1. Document QA supplies evidence-bound policy answers.
2. Exact-receipt evaluation rows and gates expose failures before release.
3. The classifier blocks high-risk intake from automation.
4. The agent uses only admitted routine intake and still needs approval for consequential actions.

Release Checklist

Don't ship a classifier project as a notebook screenshot. Keep these files or equivalent artifacts together:

Use shadow traffic before changing live routing. Record scores and proposed routes, retain human decisions, and compare missed-escalation rate by required slice. Roll back to manual review when a required slice misses its agreed floor; don't wait for average accuracy to fall.

Practice: Break The Intake Contract

Run the relevant cells again after each mutation. Revert one mutation before starting the next.

1. Inspect intake_bundle_incomplete. Why does it hold even though every remaining score clears threshold 0.35?
2. Inspect intake_bundle_padded, intake_bundle_duplicated, and intake_bundle_drifted. Why can't easy extras, duplicate rows, or a newer dataset silently change the release decision?
3. Raise intake_bundle_v2 threshold from 0.35 to 0.50. Which required fixture blocks shadow eligibility?
4. Change fallback_route to guarded_agent. Which serving-fallback.py assertions fail, and why is that unsafe?
5. Remove the bundle-version comparison from admitted_by_pinned_bundle. Which stale route artifact now reaches the agent?

Mastery Check

Evaluation Rubric

• Beginner: Defines the label and explains why the classifier routes intake rather than authorizes action.
• Applied: Trains a baseline and encoder, chooses threshold from frozen validation rows, and ships tokenizer, threshold, and fallback together.
• Advanced: Versions evaluation rows, blocks required-slice false negatives, tests calibration cautiously, and supplies a shadow-monitor rollback plan.
• Research-ready: Challenges split representativeness, label ambiguity, threshold policy, and calibration evidence before claiming generalization.

Common Failure Modes

Key Concepts

• Label contract before checkpoint.
• Baseline before complexity.
• PyTorch training mechanics before pretrained fine-tuning tooling.
• Threshold chosen on frozen validation evidence.
• Calibration measured, never assumed from softmax output.
• Frozen receipt identity, exact fixture coverage, and required-slice failures block release.
• Model, tokenizer, threshold, and fallback form one serving bundle.
• Classifier route limits the downstream agent's intake, not its approval controls.

Where This Leads Next

You now have a classifier artifact that can be consumed rather than merely discussed: intake_bundle_v2 deploys encoder_v1 with a reviewed threshold, admits routine tickets to a guarded workflow, and blocks risky or failed inputs to human review. The next capstone must respect that boundary while combining it with policy retrieval, approval gates, traces, and executable evaluation.