Logistic Regression and Metrics

Logistic regression turns features into a binary-routing score, then challenges you to justify treating that score as a probability.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/} In the previous lesson, access-change request REQ-10234 needed a numeric estimate: would retrieved evidence fit the response-latency budget? Now it needs a decision: may the workflow auto-process the permission change under policy line P-7, or must a human review it first?

You'll fit a model by hand on eight historical access-change requests, derive its gradient, implement each line in NumPy, and check it against scikit-learn. A separate eight-request validation slice then carries the decisions that matter: threshold cost, ranking, and calibration. Training output explains the mechanism. Held-out labels are where you begin to trust an operating policy.

You need the linear-regression chapter (design matrix, gradient descent loop, NumPy broadcasting) and comfort with basic probability. Each derivative gets worked with real numbers on paper before the code appears.

From latency estimates to access-review routing

In the previous chapter you predicted a number: "how many milliseconds will this evidence cost?" Now the target is binary: 1 means an access request needs human review before any permission change; 0 means policy evidence supports automatic handling. An incorrect automatic approval can cost $120 in permission rollback and investigation. An unnecessary review costs $18 of queue time.

Linear regression on a 0/1 label would happily predict 1.3 or −0.2. Those numbers are meaningless as probabilities. We need something that squeezes any real number into (0, 1).

This is also how you evaluate LLM-era classifiers. A prompt-injection guardrail, a toxicity filter, or an agent action router emits a score; a threshold turns that score into an intervention. Precision, recall, cost, and calibration belong in its model card and on-call review.

The sigmoid turns any score into a probability

The sigmoid (or logistic) function does that:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

It has four properties you'll feel in your fingers:

• As z → +∞, σ(z) → 1
• As z → −∞, σ(z) → 0
• At z = 0, σ(0) = 0.5
• Its derivative is σ'(z) = σ(z) · (1 − σ(z)), which is positive for finite z and peaks at 0.25 when z=0. This tidy form is why the gradient of log-loss later becomes so simple.

Compute five concrete values by hand. Do this on a napkin before reaching for code:

The curve saturates quickly. A score of +4 produces p = 0.982; whether that number represents an honest frequency still needs validation.

The model itself is still linear inside:

$$z = w_0 + w_1 x_1 + w_2 x_2$$

Then p = sigma(z). The positive class is needs human review. This one line is the whole logistic regression model.

The score z is also the logit, or log-odds:^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

$$z = \log \left(\frac{p}{1 - p}\right)$$

At z = 0, the odds are 1:1 and p = 0.5. Adding one to z multiplies the odds by e; it doesn't guarantee that the displayed probability is calibrated on production traffic.

Our running example: eight access-change requests

We continue the REQ-10234 access-control workflow. Each historical request has two scaled features produced from retrieved evidence:

• x1 = ambiguity score: conflicting audit logs, missing requester context, or unclear scope description
• x2 = auto-policy support score: strength of evidence that policy line P-7 permits automatic handling

The label y = 1 when the completed case required human review before a permission change, and y = 0 when automatic handling was acceptable.

This tiny dataset stays with the entire chapter:

We kept this training fixture small so you can verify each number. R3 needed review despite only moderate ambiguity. R8 had substantial ambiguity, but strong policy support made automatic processing acceptable. The resulting coefficient signs should now tell a coherent story: ambiguity raises review risk; supporting policy evidence lowers it.

Add the intercept column of ones and build the design matrix X (8 × 3) and the label vector y.

Log-loss (binary cross-entropy) is the right loss for probabilities

If the model outputs p = 0.93 for a request with y = 0, it made a confident mistake that deserves a large penalty. If it outputs p = 0.07 for that request, the loss is small. Negative log-likelihood encodes that asymmetry.

For one example the loss is the negative log-likelihood (also called binary cross-entropy).^{[2]Reference 2Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/[3]Reference 3Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html}

$$L = − \bigl[ y \log p + (1 − y) \log (1 − p) \bigr]$$

When y=1 the second term vanishes and we pay −log p (large when p is near 0). When y=0 we pay −log(1−p).

Why not squared error? Reusing MSE from linear regression makes the objective non-convex after composition with a sigmoid. For unregularized linear logistic regression, log loss is convex in the weights and yields the clean gradient below.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/} Convexity makes optimization behavior easier to reason about, but it never replaces evaluation on unseen data.

Compute by hand for R4 (x1=3.5, x2=6, y=1) with a starting guess w = [0, 0, 0]:

• z = 0 + 0·3.5 + 0·6 = 0
• p = σ(0) = 0.5
• L = − [1 · log(0.5) + 0] = − log(0.5) ≈ 0.693

Now suppose the model is slightly better and predicts p = 0.82 for the same row:

• L = − log(0.82) ≈ 0.198
• The loss dropped dramatically because the model became more confident in the correct direction, exactly the behavior we want.

For code, compute the same loss from logits rather than taking log(sigmoid(z)). The expression log(1 + exp(z)) - y*z is the same loss, but evaluating exp(z) directly can overflow. NumPy's logaddexp(0, z) - y*z evaluates that softplus term stably even when z is very large in either direction.

The log-loss gradient collapses to (p − y) · x

Derive this once, with real numbers, on R4. The same pattern will reappear in neural-network classifiers later.

We have L(p, y) and p = σ(z), z = w·x (including w₀).

• By chain rule: ∂L/∂wⱼ = (∂L/∂p) · (∂p/∂z) · (∂z/∂wⱼ)
• First, ∂L/∂p = − (y/p − (1−y)/(1−p) )
• Second, ∂p/∂z = p · (1 − p) (the sigmoid derivative)
• Third, ∂z/∂wⱼ = xⱼ because increasing weight wⱼ changes the score in proportion to feature xⱼ.

The algebra collapses more neatly than it looks. Look at the two label cases:

Both rows are the same formula: ∂L/∂z = p − y. Then the weight gradient is:

$$\frac{\partial L}{\partial w_j} = (p - y) \cdot x_j$$

This holds for each weight, including the intercept (x₀ = 1).^{[2]Reference 2Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/[3]Reference 3Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html}

Plug in numbers for the starting point on R4 (p=0.5, y=1, x=[1, 3.5, 6]):

• grad_w0 = (0.5 − 1) · 1 = −0.5
• grad_w1 = (0.5 − 1) · 3.5 = −1.75
• grad_w2 = (0.5 − 1) · 6 = −3.0

The gradient points uphill (direction of increasing loss). We subtract it (scaled by the learning rate) to descend.

That sign is the entire learning signal. If y=1 and p=0.5, p-y is negative, so subtracting the gradient increases weights attached to positive features and raises the future score for similar requests. If y=0 and p=0.8, p-y is positive, so subtracting the gradient lowers those weights and reduces the score.

Do the same arithmetic for each row, average the gradients, and you have one step of gradient descent for logistic regression. The code is close to the linear-regression GD loop you already wrote; the prediction and gradient formula change.

Implementing logistic regression from scratch in NumPy

This complete, self-contained implementation can be copied into a file and run.

The negative intercept starts low when both signals are absent. Ambiguity's positive coefficient raises review risk, while the negative policy-support coefficient lowers it, matching the feature definition. Each progress line reports loss and weights after the same update, so snapshots stay comparable. These eight rows teach model mechanics; they never justify a shipping threshold by themselves.

The code builds a working logistic regression model whose scoring, gradients, and threshold behavior are visible.

Select a threshold on held-out requests

The fitted model emits a score in (0, 1). A business action needs a threshold t: route to review when p >= t. The training confusion matrix at t = 0.5 is a useful code check (TP=3, FP=1, FN=1, TN=3), but selecting t on those same rows would reward overfitting.

Use a validation slice with cases the optimizer never saw. Here a false negative means automatic processing when review was required, costed at $120. A false positive means unnecessary human review, costed at $18.

On this validation slice, t = 0.20 costs least because avoiding missed reviews outweighs two extra queue items. In production, estimate these costs with stakeholders and re-evaluate thresholds as request mix shifts.

Precision, recall, F1, and class imbalance

For any threshold, fill the 2x2 confusion matrix:

Definitions:

• Precision = TP / (TP + FP): among requests routed to review, what fraction required it?
• Recall = TP / (TP + FN): among requests requiring review, what fraction reached it?
• F1 = 2 * precision * recall / (precision + recall): one balance of those rates.

Accuracy alone can be nearly useless when positive cases are rare:

ROC and precision-recall curves measure ranking

To draw an ROC (Receiver Operating Characteristic) curve, vary the threshold from high to low and plot false-positive rate against recall. AUC answers a ranking question: how often does a random positive score rank above a random negative score? Ties get half credit. AUC never chooses a business threshold for you.

For rare positives, ROC can look generous because many true negatives keep false-positive rate small. Plot a precision-recall curve as well: it exposes how much review load accompanies higher recall.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

Calibration: do scores represent frequencies?

A model can rank access-change requests well while producing over-confident or under-confident numbers. If requests scored near 0.80 require review only half the time, exposing 80% to an operator misstates risk.

A reliability diagram compares average score with observed positive frequency per bin. Expected Calibration Error (ECE) summarizes binned gaps:

$$\mathrm{ECE} = \sum_b \frac{|B_b|}{n}\left|\operatorname{avg}_{i\in B_b}(p_i) - \operatorname{avg}_{i\in B_b}(y_i)\right|$$

ECE depends on sample size, prevalence, and bin choices. It has no universal pass threshold. State the binning scheme, measure on held-out labels, and use a separate calibration split when fitting Platt scaling, isotonic regression, or temperature scaling. Temperature scaling is a common post-hoc method for neural classifiers.^{[4]Reference 4On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599}

Eight validation requests are enough for arithmetic, not a release gate. A single disputed label moves both metrics noticeably:

First audit action selection:

Then audit probability meaning:

Verify the NumPy fit with scikit-learn

A from-scratch implementation earns trust by matching a maintained implementation under equivalent settings.^{[5]Reference 5Scikit-learn: Machine Learning in Python.https://jmlr.org/papers/v12/pedregosa11a.html} Scikit-learn applies regularization by default; set C=np.inf here to compare against our unregularized gradient-descent fit.

The weight match validates the mechanics. The validation metrics validate only this small held-out exercise; larger and temporally later data is still required before deployment.

Tests that protect the implementation

Tests should protect stable numerics, the known fit, held-out threshold behavior, and parity settings:

Run with pytest tests/test_logistic.py -q --tb=line after placing implementation in logistic_scratch.py. Unlike an exact ECE assertion on eight labels, these tests guard code contracts and the selected validation scenario.

Symptom, cause, fix table (the debugging checklist)

Try it yourself

These exercises turn the chapter from reading into mastery. Use training rows for fitting and validation rows for policy selection.

1. Start with w = [0,0,0] and compute the gradient vector on the full batch by hand for the initial step (lr = 0.5). Verify it matches the initial printed line of your NumPy run.
2. Change the learning rate to 2.0 and watch the loss explode or oscillate. Explain why in one sentence.
3. Sweep thresholds [0.2, 0.35, 0.5, 0.65, 0.8] on validation and print precision, recall, F1, and cost. Which threshold minimizes the stated cost?
4. Bin validation probabilities into four equal-width bins, compute ECE by hand, and compare with code output. Repeat with eight bins and explain why the number changes.
5. Flip one validation label and measure F1 and ECE again. Write the argument for gathering a larger, later-in-time validation set before publishing probability claims.

If any answer feels hand-wavy, print the arrays, loss numbers, and concrete cells of the confusion matrix. Concrete numbers build classification skill.

What you have built and where it leads

The classifier can now:

• Turn a binary classification problem into a design matrix and fit logistic weights with gradient descent using NumPy alone.
• Derive and implement the (p − y)·x gradient yourself.
• Select a decision threshold on held-out requests with an explicit cost model.
• Compute classification metrics and distinguish threshold decisions from ranking quality.
• Draw a reliability diagram and compute ECE without inventing a universal pass line.
• Write deterministic tests for numeric stability, parity, and validation behavior.

These are the same habits behind production classifiers: fit the model, inspect the threshold tradeoff, measure calibration, and write tests that keep the implementation honest.

The validation slice improved the evaluation boundary, but eight requests remain far too few for a release claim. Next, decision trees and ensembles attack the same access-review routing task with nonlinear rules; the later cross-validation lesson will deepen the evidence required before shipping a model.

Mastery check

Key concepts

• sigmoid activation
• log loss / binary cross-entropy
• gradient of logistic loss
• decision threshold tuning
• confusion matrix (TP/FP/TN/FN)
• precision, recall, F1-score
• ROC curve and AUC
• reliability diagram / calibration plot
• Expected Calibration Error (ECE)
• NumPy from-scratch implementation
• scikit-learn comparison
• imbalanced classification pitfalls

Evaluation rubric

• Foundational: Computes sigmoid, log loss, and the (p - y) * x gradient by hand on an access-change request with concrete scores and loss values
• Intermediate: Implements NumPy logistic regression with stable logit loss; selects a cost-aware validation threshold; computes F1, AUC, and binned ECE; verifies unregularized parity with LogisticRegression
• Advanced: Separates fit, threshold selection, ranking audit, and calibration audit; explains label sensitivity and the evidence needed before exposing probability claims

Follow-up questions

Common pitfalls

• Symptom: dashboard accuracy looks fine, but cases needing review are auto-processed. Cause: threshold stayed at 0.5 on imbalanced data, so recall collapsed. Fix: sweep thresholds on a held-out set and pick by expected business cost or recall target, not habit.
• Symptom: operators distrust percentages like 0.87 even though AUC looks useful. Cause: ranking and calibration answer different questions. Fix: draw reliability diagrams on enough held-out labels, reserve calibration data, and withhold percentages until verified.
• Symptom: your NumPy weights disagree sharply with scikit-learn. Cause: intercept handling, regularization, scaling, or solver settings don't match. Fix: compare design matrices first, disable sklearn regularization when matching plain GD, and verify feature preprocessing is identical.
• Symptom: loss falls for a few steps, then spikes or oscillates. Cause: learning rate is too large or logits are hitting unstable numeric regions. Fix: lower the learning rate, clip logits inside sigmoid, and print loss every few hundred steps until the curve is stable.