Capstone: Fine-Tuned Classifier

Consider this ticket:

Given a new access ticket, should it enter human_review_now, or may it enter the guarded agent workflow?

That exception isn't the same as a routine key-rotation question. The intended route is immediate human review; routine questions can use a guarded workflow without consuming the same review capacity. The classifier does routing, not authorization. It doesn't decide access eligibility, send a reply, or change permissions. It can misclassify a ticket, so the agent's independent approval controls must remain in place.

Why use a dedicated, fine-tuned encoder rather than prompting a frontier generative LLM? In production intake systems, four engineering realities decide the choice:

1. Latency budget and webhook SLAs: Webhook ingress requires an immediate routing lane in 5 to 15 milliseconds. A full autoregressive LLM API roundtrip costs 400 to 1,500 milliseconds, introducing queue backlog and token-generation latency jitter at high ticket volume.
2. Deterministic logit geometry: A fine-tuned encoder classification head produces continuous, unnormalized logits through an exact affine projection $z = W h_{[\text{CLS}]} + b$. That exposes raw score distributions for threshold tuning, temperature calibration, and out-of-distribution audits. Prompting an LLM yields sampled tokens that can break JSON formatting, hallucinate unsupported keys, or surrender to prompt injections embedded in customer ticket bodies.
3. Operational cost and throughput: Serving an encoder like DeBERTa-v3 or DistilBERT on CPU or small GPU instances costs roughly $0.00001 per ticket, compared to $0.001 to $0.005 for commercial LLM calls. At hundreds of thousands of daily tickets, that's the difference between single-digit server bills and thousands of dollars in token fees.
4. Asymmetric risk and critical triage: Support triage has radically asymmetric costs. Routing a routine key-rotation question to a human reviewer (a false positive) wastes two minutes of an engineer's time ($1.50). Routing an emergency incident-freeze bypass, suspected account takeover, or unauthorized privilege escalation into the automated agent (a false negative) risks automated token reactivation during an active security breach. That asymmetry means we can't rely on symmetric averages like accuracy or macro-F1. High-risk security slices demand a hard release gate with 0% false-negative tolerance.

The evaluation dashboard in the previous capstone made one rule hard to dodge: a release is blocked by the failures that matter, even when the average looks fine. Bring that exact-receipt habit to a learned classifier. Document QA already answers policy questions with permitted citations. A later production-agent capstone will draft an access response and request approval. This intake classifier is the boundary in front of that agent.

By the end, one bundle will carry:

• a versioned label guide and split rule
• a fitted baseline that a contextual model must actually improve on
• a small PyTorch encoder training loop that exposes the mechanics
• a practical Hugging Face fine-tuning recipe for a pretrained encoder
• held-out threshold and slice gates
• a serving bundle the production-agent capstone can consume

BERT showed how a pretrained bidirectional encoder can adapt to a downstream classification task with a small output layer.^{[1]Reference 1BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} Hugging Face exposes that sequence-classification pattern through models and training utilities.^{[2]Reference 2Text Classification.https://huggingface.co/docs/transformers/tasks/sequence_classification} Modern encoders like DeBERTa-v3 advance this by disentangling content and relative position representations, while distilled options like DistilBERT provide lightweight alternatives running at double the speed. The discipline here isn't choosing a fashionable checkpoint. It's proving that a trained classifier satisfies a routing contract.

The executable examples have three evidence levels: a fitted miniature baseline, a six-row neural training demonstration, and hand-authored score receipts for testing routing logic. The separate DistilBERT project script needs your reviewed dataset and a checkpoint download. None of the printed gate scores comes from that pretrained model, and a passing demonstration doesn't authorize deployment or shadow traffic.

Write the routing contract before training

Before training, give the model one narrow decision to make:

Read the second row carefully. Label 0 doesn't mean "approve automatically." It means "safe to enter an automation path with its own controls." The classifier acts as an intake firewall. When a routine request enters guarded_agent, the downstream agent still has to retrieve permitted documentation and request human approval before executing any state change. But when an incident-freeze exception or account takeover enters the agent, the system risks unauthorized automated actions. That's why the routing boundary treats missed escalations as critical errors.

The model will produce a score, but the score isn't a route yet. A pinned threshold turns it into an intake lane, and either lane preserves the agent's downstream evidence and approval boundaries.

Carry the evidence forward

Write labeled fixtures before training. They make the route contract concrete and give support operations something to review before a model can hide an ambiguous case.

A label guide should name ambiguous cases too. Frustrated tone alone isn't a reason for immediate human routing, while a calm request for an incident-freeze exception may be. Support operations should adjudicate disagreements and store that policy version beside every dataset generated from it.

Now the labels have meaning. The next question is whether future tickets can reach the evaluation split without bringing their account history with them.

Build a dataset you can defend

These rows form a hypothetical teaching dataset, not measured production traffic. Their job is to make failure modes visible:

For a real data card, replace each teaching assumption with provenance, consent and retention policy, labeling instructions, class balance, disagreement rate, and known coverage gaps.

Keep related tickets together

Multiple messages from one workspace account can share service names or incident wording. Putting one thread in training and another in validation can inflate an estimate meant to describe new accounts. For this target, use non-overlapping accounts and later time windows. If an account spans windows, exclude its earlier rows from training when reserving it for evaluation. Group splits and time splits solve different problems; neither alone enforces both.^{[3]Reference 3Cross-validation: evaluating estimator performancehttps://scikit-learn.org/stable/modules/cross_validation.html}

Use development data to pick features and checkpoints, then a disjoint selection partition to choose the threshold. Keep the final gate/test partition untouched until those choices are frozen. The eight-row manifest illustrates group and time checks; it isn't the actual split behind the later synthetic score tables. In a real pipeline, ticket IDs, normalized text hashes, account groups, and timestamps must join to the same frozen dataset.

The split output is simple on purpose: each account appears in one partition, and later months stand in for future traffic. That gives every later metric a clearer question to answer.

Two messages from account acme describe the same access incident. Why is putting one in training and one in validation invalid even when the text differs?

Answer

The rows share incident and account context, so the model can learn wording or facts from the training thread that reappear in validation. Keep the whole account or incident group on one side of the split, and reserve the final test groups until model and threshold choices are frozen.

Compare a cheap baseline first

Before paying for a contextual transformer, give a cheap baseline a chance. Start with a deterministic rule or a bag-of-words logistic-regression model. If it satisfies the routing gate on representative held-out data, a neural model may add maintenance without adding safety.

The rule below looks for contiguous phrases such as key disabled. Before reading its output, predict what happens to "Production API key disabled..." and to "The production API key was disabled during the incident freeze." The second sentence contains the same idea, but was sits between key and disabled.

The missed phrase exposes a weakness in this rule, not a proof that a transformer is necessary. Fit TF-IDF plus logistic regression next. The vectorizer learns its vocabulary and inverse document frequencies from training rows only; a pipeline keeps that fit boundary explicit.^{[4]Reference 4Common Pitfalls and Recommended Practices: Data Leakagehttps://scikit-learn.org/stable/common_pitfalls.html#data-leakage[5]Reference 5LogisticRegressionhttps://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html}

This small comparison uses the same four held-out teaching tickets as the rule. It prints positive recall and false positives instead of rewarding an all-human classifier for perfect recall. The training rows are deliberately similar to the validation wording, so these results test the implementation, not realistic generalization.

Keep this baseline when it meets the actual acceptance criteria. Fine-tuning is a candidate experiment, not the required winner. Why do linear baselines eventually fall short on complex enterprise traffic? While TF-IDF handles straightforward vocabulary overlap, it fails on complex syntactic negation (such as "I am not requesting a key restore, just asking where the guide is"), clause embedding, and cross-sentence semantic dependencies. That's precisely where contextual encoders like DeBERTa-v3 and DistilBERT justify their compute footprint by capturing bidirectional semantic dependencies across the entire sequence.

Keep the comparison empirical. A fine-tuned encoder isn't cheaper, faster, or more accurate than prompting or this baseline until those candidates run on the same task and hardware.

Turn a score into a route

The model's raw output isn't a route yet. An encoder classifier emits one unnormalized logit for each class. Softmax turns the pair into class scores that sum to one. A separate decision threshold turns the positive-class score into the intake route.

The small implementation detail matters: subtract the largest logit before exponentiating so softmax stays numerically stable. With logits [-1.0, 2.0], predict which class should receive the larger score before running the function.

Call 0.95 a score until held-out reliability evidence supports calling it a probability. A bounded output between zero and one isn't proof of calibration. The classification head computes an affine projection $z = W h_{[\text{CLS}]} + b \in \mathbb{R}^2$. Softmax normalizes those logits so they sum to one, but overparameterized neural networks routinely output overconfident scores on out-of-distribution inputs. The number is a continuous routing score for threshold comparison, not an objective Bayesian probability, until you measure calibration on held-out test data.^{[1]Reference 1BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805}

Make the training mechanics visible

Before downloading a checkpoint, trace one trainable path end to end. The sandbox builds a tiny word-embedding encoder, averages token vectors, attaches a classification head, and updates every parameter with cross-entropy loss.

This is a classifier trained from scratch, not BERT fine-tuning. Its job is to make each moving part visible before random embeddings give way to pretrained representations.

The sandbox can memorize six training examples. The four new strings check inference after fitting, but their near-duplicate wording doesn't establish generalization to new accounts. Follow the wiring: text becomes tokens, pooled representations become logits, and gradients update weights. The restore check uses the same vocabulary and architecture; saving weights without those would be incomplete.

One architecture gap is deliberate. This sandbox averages token vectors, so it can't distinguish word order: swapping tokens leaves the pooled representation unchanged. BERT-style sequence classification uses contextual token representations and a leading [CLS] summary.^{[1]Reference 1BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} Cross-entropy expects unnormalized logits, so don't apply softmax before passing the output to CrossEntropyLoss.^{[6]Reference 6CrossEntropyLosshttps://docs.pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html}

Fine-tune a pretrained encoder

Now replace random embeddings with a pretrained encoder such as DistilBERT or DeBERTa-v3 and fine-tune it for two labels. Bidirectional encoders read context on both sides of each token simultaneously, avoiding the unidirectional attention blind spot of causal decoder models. DeBERTa-v3 improves classification accuracy by disentangling token content from relative position vectors, while DistilBERT distills the 12 transformer layers of BERT-base into 6 layers, running at double the inference speed. Both adapt to sequence classification by attaching a linear projection head to the pooled [CLS] summary vector.^{[1]Reference 1BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} Hugging Face exposes that sequence-classification workflow through AutoModelForSequenceClassification and Trainer.^{[2]Reference 2Text Classification.https://huggingface.co/docs/transformers/tasks/sequence_classification}

The script uses three epochs, a learning rate of $2 \times 10^{-5}$, and batch size 16 as starting settings, not tuned values for this dataset. While DistilBERT's default configuration supports 512 positions, the serving policy here enforces a 256-token ceiling including special tokens.^{[7]Reference 7DistilBERT base uncased configurationhttps://huggingface.co/distilbert/distilbert-base-uncased/resolve/12040accade4e8a0f71eabdb258fecc2e7e948be/config.json}

Our truncation rule is an explicit security control: never silently truncate inputs from the right. When an engineer pastes a long diagnostic log followed by a closing sentence like "The key failed during the incident freeze, please bypass", right-truncation chops off the critical risk signal at token 256. That would leave the classifier scoring only the routine preamble and mistakenly routing the ticket to automation. Instead, oversized tickets fail closed directly to human triage.

This project script isn't a marked, executed fine-tuning result. It needs your reviewed JSONL splits, the pinned checkpoint download, and torch, transformers, datasets, accelerate, and numpy. Each row needs a unique fixture_id, nonempty text, and integer label (0 for guarded_agent, 1 for human_review_now). Apply the account/time manifest before export. The script loads only training and checkpoint-development data; threshold-selection and final-gate rows stay outside training.

Minimum development loss chooses a checkpoint here; it doesn't approve a routing policy. load_best_model_at_end restores the checkpoint selected by that metric.^{[8]Reference 8Trainerhttps://huggingface.co/docs/transformers/main_classes/trainer} The reported argmax recall uses the implicit two-class 0.5 boundary, not the later 0.35 routing threshold (at an exact logit tie, NumPy's argmax chooses class 0). Selecting only maximum recall could favor sending everything to people. Compare candidates using required-slice misses and review capacity after threshold selection, and record exact dependency versions with the export.

The base revision identifies pretrained weights, while the export hashes identify your fine-tuned model and tokenizer. A human-readable name such as encoder_v1 is only a label. Reload the exported directory, verify its hashes against an independently retained manifest, and use model.eval() with torch.inference_mode() when scoring. Take softmax(logits, dim=-1)[:, 1]: a generic pipeline's confidence in its winning label is not always the human-review score.

After completing the project training run, this separate scorer reloads its local export and writes one score per evaluator-owned fixture. gate-fixtures.json is a list of objects containing fixture_id and text; the trusted evaluator retains labels separately. Hash checks detect mismatched files, not a malicious replacement of both the files and their manifest. Store the approved manifest outside the scorer's write boundary.

Run this only after freezing the model and threshold. Retain the scored fixture-file hash, export-manifest hash, tokenizer policy, and package versions with the output. The next sections exercise the evaluator with synthetic values instead of pretending this scoring run has happened.

Select a threshold on a selection set, not the freeze gate

Don't accept a library default threshold of 0.50 without auditing its asymmetric error tradeoff. A default 0.50 cutoff assumes symmetric loss, treating a missed security escalation as mathematically equivalent to sending a routine question to a person. In enterprise access triage, the cost matrix is heavily asymmetric: false negatives on critical risk categories (account takeover, incident-freeze bypass, privilege elevation) risk severe compromise, while false positives merely add two minutes of human review.

Under asymmetric costs, the Bayes-optimal threshold shifts downward:

$\tau^* \approx \frac{C_{\text{FP}}}{C_{\text{FP}} + C_{\text{FN}}}$

When $C_{\text{FN}} \gg C_{\text{FP}}$, $\tau^*$ is much lower than 0.50. Choose that operational cutoff on a dedicated selection set, then evaluate a separate frozen gate receipt for release. Tuning until required false negatives disappear on the same rows you later treat as release evidence contaminates the gate.

The next scores are hand-authored numbers for the threshold exercise, not exports from either trained example. Suppose a future, independently evaluated encoder_v1 produced them. Which cutoff retains required positives without flooding the human queue?

The sweep answers one narrow question. On this selection set, threshold 0.35 is the only candidate shown that avoids a missed escalation. It sends one angry but routine key-rotation ticket to human review, so the tradeoff is acceptable only if queue capacity allows it.

After locking a threshold, score a frozen gate receipt that wasn't used to choose it. Nested cross-validation is one way to separate model selection from evaluation when data is limited; it isn't a replacement for this application's group, time, and leakage checks.^{[3]Reference 3Cross-validation: evaluating estimator performancehttps://scikit-learn.org/stable/modules/cross_validation.html}

Don't report 1.00 recall from three positive selection fixtures as proof of production reliability. It's evidence for this selection slice, not for unseen traffic.

Threshold 0.35 has zero false negatives on the selection set, while 0.50 misses one required escalation. What must happen before 0.35 can control shadow traffic?

Answer

Freeze 0.35 as part of a versioned serving bundle, then score a disjoint gate receipt that wasn't used to choose it. The gate must still satisfy required-slice false-negative and queue-capacity limits; selection-set recall alone can't authorize release.

Freeze the evidence gate

The dashboard capstone compared runs only when dataset, grader, corpus, and fixture set matched. Carry that discipline here. A classifier receipt pins its dataset, label guide, split manifest, model, and exact gate fixture IDs before scores are aggregated. The evaluator owns the fixture manifest, including each fixture's gold route and required slice. A scoring worker may submit only a fixture ID and the model's score. It can't submit a replacement label or slice. Gate fixtures must not be the set used to choose the threshold.

In the simulation, intake_bundle_v1 uses threshold 0.50 and misses freeze_window_ticket at score 0.41. intake_bundle_v2 uses 0.35 and passes the arithmetic gate. The two comparisons illustrate a policy change, not permission to retune after seeing the final gate. Real gate feedback becomes development data once used for tuning, and the next candidate needs new independent evidence. A threshold change versions the serving bundle, not the model weights.

Here eligible_for_shadow is a demonstration return value. These synthetic scores can't establish eligibility for real shadow traffic. That decision also needs approved data handling, measured model outputs, sufficient slice coverage, and operational review.

The evaluator checks declared model identity, bounded scores, and exact fixture coverage. It owns gold routes, slice membership, and bundle thresholds; submitted replacements are rejected. These checks don't prove which model ran or which text it saw. A real harness must score evaluator-owned inputs and retain authenticated output provenance. Disjoint fixture IDs also don't prove disjoint text, accounts, or incidents; check the split manifest and duplicates before scoring.

The rejected label mutation never reaches the gate calculation: gold and slice aren't part of the scoring-worker schema, so the evaluator keeps its own values. A malformed score also holds before aggregation, and a report that changes the threshold can't replace the evaluator's pinned policy. Lowering the threshold isn't automatically the right fix. If false positives overwhelm reviewers, the executable gate holds on max_false_positives rather than only on required-slice FNs. This receipt therefore keeps threshold selection separate from the freeze gate, refuses to hide a required-slice miss inside an average, and rejects easy rows that pad the comparison.

Audit scores without overclaiming calibration

A threshold can be useful even when the score isn't a well-calibrated probability. Calibration asks whether tickets scored near 0.70, across enough held-out examples, need human review about 70 percent of the time. Modern neural classifiers can be miscalibrated; temperature scaling is one common post-training adjustment, fit on held-out data, that rescales logits without changing the argmax class.^{[9]Reference 9On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599}

That last detail matters for this product. Guo et al. showed temperature scaling leaves predicted labels unchanged because it doesn't change which logit is largest. This router doesn't use argmax. It uses score >= 0.35 on the positive class, so rescaling logits can move a ticket across the cutoff even when the predicted class stays the same. Fit temperature on held-out data, then re-run the gate receipt before that change enters the serving bundle.

Six fixtures are enough to demonstrate a calculation, not to justify operator-facing percentages:

The high bucket exposes why calibrated language matters: an average score of 0.80 with one positive among two tickets isn't evidence that 0.80 means an 80 percent escalation rate. These are the same synthetic selection scores, not a fresh calibration evaluation. If you fit a temperature, choose the threshold under that transformed score scale and evaluate both together on untouched data.

Ship the serving contract

A model file alone can't reproduce the route. Deploying a serialized weights file (model.safetensors) without its surrounding runtime environment creates silent production drift. If a serving container boots with a newer tokenizer release that splits domain tokens differently, or if an engineer leaves an implicit 0.50 cutoff in place, routing decisions break even though the weights hash matches.

The production release artifact is an immutable serving bundle:

The endpoint must fail closed. In Python, evaluating float("nan") >= 0.35 returns False. If your code naively checks if score >= threshold: return "human_review_now" followed by a default fallthrough, any NaN output from a floating-point overflow or model exception would slip directly into guarded_agent! That is a catastrophic fail-open vulnerability.

A defensive intake contract enforces strict verification order:

1. Validate input text: empty, whitespace-only, or malformed inputs immediately take fallback_route.
2. Verify token length: inputs exceeding 256 tokens bypass the model and take fallback_route.
3. Check model availability: timeouts or worker crashes take fallback_route.
4. Validate the score: non-numeric, None, or non-finite values (NaN, Inf) take fallback_route.
5. Only finite numeric scores strictly within $[0.0, 1.0]$ are evaluated against threshold 0.35.

The trusted tokenizer counts the full input before model inference; a client-supplied token count isn't acceptable. The function below tests the routing layer with supplied scores and token counts, not the tokenizer or model itself. Blank, malformed, oversized, and unavailable inputs stay out of automation.

Admit only pinned routes

A valid serving bundle still needs an explicit downstream handoff contract. In production deployments, multiple model versions run concurrently across canary environments, rolling updates, and blue-green clusters. If an intake payload generated by an experimental canary bundle or an outdated service instance (intake_bundle_v1) reaches the downstream production agent, admitting it risks executing automated actions under an unapproved routing policy.

The downstream agent therefore enforces three admission criteria:

1. Pinned bundle identity: The ticket's bundle_version must match the agent's pinned active configuration (intake_bundle_v2).
2. Route verification: The assigned route must strictly equal guarded_agent.
3. Producer provenance: The intake artifact must originate from the authenticated classifier service.

Matching a caller-supplied version string alone isn't cryptographic authentication; a production pipeline signs provenance tokens. Even after admission, the agent capstone enforces independent security boundaries: document QA retrieval grounding and human approval gates before executing state changes.

The handoff now has four traceable boundaries:

1. Document QA supplies evidence-bound policy answers.
2. Exact-receipt evaluation rows and gates expose failures before release.
3. The classifier proposes a route, with fail-closed handling for invalid inputs.
4. The agent uses only admitted routine intake and still needs approval for consequential actions.

Keep the release evidence together

A notebook screenshot can't answer the release questions. Keep these files or equivalent artifacts together:

Before changing live routing, run shadow traffic. Record scores and proposed routes, retain human decisions, and compare missed-escalation rate by required slice. Roll back to manual review when a required slice misses its agreed floor; don't wait for average accuracy to fall.

Practice: break the intake contract

Treat each mutation as a controlled failure. Predict which evidence should reject it, run the relevant cells, then revert that mutation before starting the next.

1. Inspect intake_bundle_incomplete. Why does it hold even though every remaining score clears threshold 0.35?
2. Inspect intake_bundle_padded, intake_bundle_duplicated, and intake_bundle_drifted. Why can't easy extras, duplicate rows, or a newer dataset silently change the release decision?
3. Re-version the serving policy at threshold 0.50 while freeze_window_ticket still scores 0.41. Which required fixture blocks shadow eligibility?
4. Inspect intake_bundle_fp_flood. Why does it hold even though required-slice positives still clear a low threshold?
5. Change fallback_route to guarded_agent. Which serving-fallback.py assertions fail, and why is that unsafe?
6. Remove the bundle-version comparison from admitted_by_pinned_bundle. Which stale route artifact now reaches the agent?
7. Inspect intake_bundle_tampered_labels and intake_bundle_invalid_score. Which submitted fields does the evaluator trust, and why must the other fields hold before any metric is computed?

What should each mutation prove?

Answer

1. Missing fixture evidence must fail closed; absence isn't a passing result. 2. The receipt rejects unexpected easy rows, duplicate rows, and dataset drift before aggregation. 3. freeze_window_ticket blocks the re-versioned threshold, because its held-out score 0.41 falls below 0.50. 4. max_false_positives holds a threshold that floods the review queue even when required FNs are zero. 5. Blank input, unavailable model, and malformed score must stay out of automation. 6. r-106 must stay blocked because a route string without pinned-bundle provenance is stale evidence. 7. Only fixture_id and a finite score belong in submitted rows. The evaluator owns fixture text, gold, and slice values, and it must reject added labels or non-finite scores before aggregation.

What the agent receives next

The demonstration has tested threshold arithmetic, exact fixture coverage, fail-closed routing, and version checks. It hasn't deployed or validated a pretrained classifier. To finish the project, replace synthetic scores with authenticated outputs from the hashed export, compare the baseline on the same untouched gate, and obtain operational approval. The later agent capstone must retain these boundaries while adding retrieval, approval gates, traces, and evaluation.