Decision Trees, Forests, and Boosting

Logistic regression scored access-change request REQ-10234, then converted that score into a human-review decision using asymmetric costs. A tree answers the same routing question with conditional rules: high ambiguity may usually require review, unless evidence for automatic handling under policy P-7 changes the case.

That access-review setting stays with us. Eight historical requests form training data; eight later requests form validation data. That train, validation, and eventual test split matters because a tree can memorize historical exceptions while failing on the next batch.

One Access-Review Table, Now With Nonlinear Rules

Each access-change request has two scaled evidence features:

The fitting example uses these eight historical requests:

Four requests require review and four can be automatic. No single obvious feature perfectly separates them: R3 needs review despite modest ambiguity, while R8 remains safe for automation despite higher ambiguity because the policy evidence is strong.

Impurity measures mixed leaves

A decision tree repeatedly splits rows into leaves. A useful split makes each leaf more uniform in the target.

For a binary leaf with review proportion $p$, Gini impurity is:

$$G = 1 - p^2 - (1-p)^2$$

A pure leaf has $G=0$. A leaf evenly split between classes has $G=0.5$.

An alternative is entropy:

$$H = -p\log_2(p) - (1-p)\log_2(1-p)$$

Both metrics reward purer child leaves. A split's impurity decrease is the parent impurity minus weighted child impurity. When entropy is the impurity metric, this decrease is called information gain. Below we use Gini decrease, or Gini gain, because its arithmetic is compact.

At the root, four of eight rows need review:

$$G_{\text{root}} = 1 - (4/8)^2 - (4/8)^2 = 0.500$$

Suppose the tree tests ambiguity <= 3.35:

• Left leaf: labels [0, 0, 1, 0, 0], so $G_{\text{left}} = 0.320$.
• Right leaf: labels [1, 1, 1], so $G_{\text{right}} = 0.000$.
• Weighted impurity: $(5/8)(0.320) + (3/8)(0.000) = 0.200$.
• Gini gain: $0.500 - 0.200 = 0.300$.

That split is helpful, but it isn't perfect: R3 remains a required-review request inside the mostly automatic leaf.

A stump tries every midpoint

A decision stump is a tree with one split. For each numeric feature, the stump can place a threshold between adjacent observed values. The training algorithm scores each candidate using impurity decrease and keeps the best one.

Testing midpoints matters. A threshold such as 3.35 represents the boundary between observed values 3.2 and 3.5; it avoids inventing an unnecessary boundary through an actual training row.

The best stump rule is:

$$\texttt{ambiguity} \le 3.35$$

The left leaf predicts review probability $1/5 = 0.20$. The right leaf predicts $3/3 = 1.00$.

The tree produces a score before a decision. A score of 0.20 isn't automatically "safe": the routing threshold must reflect the harm of an automatic permission change that needed review.

Build the Stump in NumPy

This implementation searches every feature and midpoint, stores leaf probabilities, then scores the training rows. That's the core mechanic hidden inside a library tree.

The stump misses R3. A deeper tree can create another branch to repair that training mistake, but a repaired training label alone gives no evidence that the extra branch generalizes.

Validation Turns a Fitted Rule Into Evidence

We now freeze the fitted stump and score later requests that were not used to choose its split:

The operational costs match the classification lesson:

A 0.50 threshold minimizes neither cost nor risk by definition. We compare candidate thresholds on validation data, then would confirm a selected policy on a fresh test period before deployment.

Both thresholds achieve the same F1 score here. They differ sharply in business cost: the lower threshold sends more requests to review but avoids expensive false negatives on this validation batch.

Depth Can Memorize the Exception

A one-split tree is easy to audit but often underfits. An unconstrained tree can isolate every historical row. That becomes overfitting when the tree fits historical rows too specifically and transfers worse to new requests. The right question is whether extra depth improves held-out behavior, not whether it reaches perfect training accuracy.

In this small slice, the deep tree fixes every training mistake and performs worse on validation. Real projects tune constraints such as max_depth, min_samples_leaf, and pruning using validation or cross-validation rather than a single training score.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/[2]Reference 2Scikit-learn: Machine Learning in Python.https://jmlr.org/papers/v12/pedregosa11a.html}

Random forests average across perturbed trees

A random forest trains many trees on perturbed views of the training set. Each tree receives a bootstrap sample of rows and, at each split, a random subset of candidate features. Averaging their class-probability scores reduces the instability of one fully grown tree while preserving nonlinear splits.^{[3]Reference 3Random Forestshttps://doi.org/10.1023/A:1010933404324}

The forest emits more varied scores than the stump. As before, its threshold belongs to the routing policy and must be evaluated with the same costs and fresh evidence.

Boosting adds small corrections

Gradient boosting builds trees sequentially. Each new tree learns a correction to predictions made so far. For squared-error regression, that correction target is the residual $y - \hat{y}$; Friedman generalized the idea to differentiable losses through negative gradients.^{[4]Reference 4Greedy Function Approximation: A Gradient Boosting Machinehttps://projecteuclid.org/journals/annals-of-statistics/volume-29/issue-5/Greedy-Function-Approximation--A-Gradient-Boosting-Machine/10.1214/aos/1013203451.full}

Before returning to binary routing, consider a side task: predict review-handling minutes for three access changes. Their ambiguity values are [1, 2, 5], and actual handling times are [10, 20, 60] minutes.

Shrinkage reduces the impact of any single correction. More rounds can accumulate useful structure, but too many rounds or overly complex trees can still overfit.

Classification Boosting on Access Routing

For needs_review, a classification boosting model uses a classification loss rather than raw minute residuals. The operational comparison remains the same: score the held-out requests, apply a candidate routing threshold, and count costly mistakes.

On this validation slice, boosting ranks review cases better than the stump but still needs a cost-aware threshold. With eight rows, treat that result as a mechanics exercise and a reason to gather more evaluation data, not as a production winner.

Importance Is a Diagnostic, Not a Cause

Tree libraries often report mean decrease in impurity (MDI): how much fitted splits on each feature reduced impurity. MDI is fast, but it describes the trained forest and can over-credit features favored by its split search.

Permutation importance asks a held-out question: how much does the chosen metric drop when one feature is shuffled? It's often more useful for validation diagnostics, though correlated features can share or mask importance.^{[2]Reference 2Scikit-learn: Machine Learning in Python.https://jmlr.org/papers/v12/pedregosa11a.html}

The wide permutation uncertainty is useful information: eight validation rows are too few for confident claims about which evidence field drives general behavior. Here, scoring="f1" uses the forest's default class predictions. A deployment audit should also define a scorer for documented routing cost at the selected threshold.

A Local Explanation for a One-Feature Stump

SHAP represents a model prediction as a baseline value plus additive feature contributions.^{[5]Reference 5A Unified Approach to Interpreting Model Predictionshttps://arxiv.org/abs/1705.07874} For our one-feature stump, using the training rows as background, the calculation is exact without any library:

• Baseline review score across the training background: $0.50$.
• Left-leaf score: $0.20$, so ambiguity contributes $-0.30$.
• Right-leaf score: $1.00$, so ambiguity contributes $+0.50$.
• auto_policy_support contributes 0.00 because this stump never split on it.

R3 shows the limit. The explanation faithfully reports why the stump scored the request low; it doesn't prove that low ambiguity made automatic processing correct. Explanations audit model behavior, while labels, policy review, and held-out monitoring audit decision quality.

Choosing a Model on Later Requests

A tree model earns deployment consideration only after the same later-request evaluation used for simpler baselines. Start with a regularized tree or ensemble, compare it against logistic regression under identical splits and costs, and inspect whether the added flexibility repairs meaningful failures.

Trees give transparent conditional splits; forests reduce single-tree instability; boosting accumulates targeted corrections. None of those mechanisms replaces held-out evaluation.

Practice tasks

1. Change R8's target to 1. Recompute the best stump split and explain why policy support may become more useful.
2. Add a third feature, photo_damage_score, and extend fit_stump to consider it. Report whether validation cost improves.
3. Tune max_depth and min_samples_leaf for the forest using only the validation data above. Then write down what separate data you would reserve for an unbiased final check.
4. Modify the false-negative cost from $120 to $40. Recompute threshold choices for stump, forest, and boosting.
5. Implement one more boosting regression round with a smaller learning rate. Compare training MSE after four rounds.

Key concepts

• gini impurity
• entropy and information gain
• Gini gain / impurity decrease
• decision stump
• axis-aligned splits
• overfitting in trees
• bagging vs boosting
• gradient boosting residual fitting
• feature importance
• SHAP values
• validation-based model comparison

Evaluation rubric

You're ready to continue when you can:

• Compute Gini impurity and Gini gain for the access-review stump by hand, then explain how entropy-based information gain follows the same weighted-child pattern.
• Implement a numeric decision stump that searches feature midpoints.
• Explain why R3 exposes underfitting and why a perfect deep-tree training score isn't sufficient evidence.
• Compare a stump, forest, and boosted classifier with held-out F1, ranking quality, and business cost.
• Distinguish model attribution from causal or policy justification.

Follow-up questions

Common pitfalls

• Symptom: a tree reaches perfect training accuracy, yet later costly exceptions are missed. Cause: depth memorized historical rows. Fix: tune tree constraints using held-out cost and inspect leaf sample counts.
• Symptom: a forest or boosted model has acceptable F1, but permission-rollback cost remains high. Cause: threshold stayed at 0.50 despite asymmetric errors. Fix: sweep thresholds on validation data under documented cost assumptions.
• Symptom: a feature attribution is presented as policy evidence. Cause: model explanation was mistaken for a causal justification. Fix: pair attributions with labels, policy review, and later-request monitoring.