Decision Trees, Forests, and Boosting

One access-change request breaks the obvious rule. R3 has an ambiguity score of only 2.0, yet its label says it needed human review. Several higher-ambiguity requests also needed review, but R8 at 3.2 didn't. How can a model catch exceptions without sending every request to a human?

The previous lesson fitted a smooth review score over ambiguity and policy support, with a straight decision boundary. A decision tree asks yes/no questions instead. Each answer leads to another question or a leaf, which stores a prediction. Fit one split, then investigate whether extra branches or multiple trees handle R3 more reliably.

Keep the same fictional error costs: $120 for a missed required review and $18 for an unnecessary review. These aren't measured incident costs. Authorization rules still apply independently of the classifier. A tree's feature threshold chooses a branch; a separate operating threshold turns its final score into a routing action.

The same eight requests, now as rectangles

Keep the same two evidence scores from the logistic example. ambiguity rises when request text and retrieved evidence conflict. auto_policy_support rises when policy P-7 backs automatic handling. needs_review is 1 when a human had to approve the permission change.

In this small synthetic dataset, each row is one request. Which single feature cut would leave the fewest mixed labels?

Four requests needed review and four didn't. Ambiguity looks promising because R4, R5, and R6 are high, but R3 and R8 are counterexamples. Keep eight separate constructed requests frozen for validation, using the same train / validation split as logistic regression. They represent a later batch in the story; their results aren't evidence about real traffic.

A decision tree splits on one feature at a time using an orthogonal, axis-aligned cut ($x_j \le \theta$). In our two-feature plane, each split draws a strictly horizontal or vertical line, partitioning the feature space into hyper-rectangles.

This geometry has a huge advantage and a subtle trap:

• Ordering rather than distance: An exact numeric split searches gaps between distinct values. Strictly increasing transformations preserve those candidate training partitions, so ordinary feature standardization isn't required. They don't necessarily preserve predictions inside unseen gaps. Ties, quantization, finite precision, and approximate split-search implementations also need checking.
• Diagonal boundary failure: If the true policy boundary is diagonal (say, $\text{ambiguity} + \text{support} > 6$), a single linear model draws it with one equation. An axis-aligned tree must approximate that diagonal using a jagged staircase of many orthogonal cuts, requiring deeper trees and fragmenting training data into tiny leaves.

Impurity scores how mixed a leaf is

Four of the eight requests needed review. Before choosing a cut, we need a mathematical score that measures how mixed or pure a leaf is: all automatic, all review, or at least less mixed than the parent.

For a leaf with $K$ classes where class $k$ has proportion $p_k$, Gini impurity measures the probability of misclassifying a randomly chosen sample if it were labeled randomly according to the leaf's class distribution:

$$G = 1 - \sum_{k=1}^K p_k^2$$

For binary review routing with review proportion $p$, this simplifies to:

$$G = 1 - p^2 - (1-p)^2 = 2p(1-p)$$

A pure leaf ($p = 0$ or $p = 1$) has $G = 0$. A leaf split evenly between classes ($p = 0.5$) has $G = 0.50$. Gini is a fast, bounded measure of label disagreement.

An alternative criterion from information theory is Shannon entropy^{[1]Reference 1A Mathematical Theory of Communicationhttps://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf}:

$$H = -\sum_{k=1}^K p_k \log_2(p_k)$$

For our binary case with base-two logarithms, entropy measures uncertainty in bits:

$$H = -p\log_2(p) - (1-p)\log_2(1-p)$$

Treat a zero-probability term as zero using $\lim_{p\to0^+}p\log_2p=0$. An evenly mixed binary leaf has $H=1$ bit; a pure leaf has $H=0$. Both criteria prefer purer children, but their weighted rankings can differ. Entropy includes logarithms; actual fitting speed also depends on the implementation and data.

What if the target is continuous (such as handling duration in minutes) rather than binary? In regression trees, CART minimizes squared error through variance reduction:

$$\Delta \text{Var} = \text{Var}(S) - \left[\frac{|S_L|}{|S|}\text{Var}(S_L) + \frac{|S_R|}{|S|}\text{Var}(S_R)\right]$$

where each leaf predicts the regional mean $\bar{y}$. Minimizing child variance is mathematically equivalent to maximizing the reduction in Mean Squared Error (MSE).

For classification, a split's impurity decrease is parent impurity minus weighted child impurity. When entropy is the metric, that decrease is information gain. When Gini is the metric, it's Gini gain. We'll use Gini gain because the arithmetic stays clean and fast.

At our training root, four of eight rows need review:

$$G_{\text{root}} = 1 - (4/8)^2 - (4/8)^2 = 0.500$$

Now test the tempting cut ambiguity <= 3.35. Predict the outcome before checking the arithmetic: the three high-ambiguity rows (R4, R5, R6) are all reviews, so only the low side remains mixed.

• Left labels [0, 0, 1, 0, 0], so $p = 1/5 = 0.20$ and $G_{\text{left}} = 1 - 0.20^2 - 0.80^2 = 0.320$.
• Right labels [1, 1, 1], so $p = 3/3 = 1.00$ and $G_{\text{right}} = 0$.
• Weighted child impurity: $(5/8)(0.320) + (3/8)(0.000) = 0.200$.
• Gini gain: $0.500 - 0.200 = 0.300$.

The cut removes most label disagreement, but not all of it. R3 remains a required-review request inside the mostly automatic leaf.

Two criteria can prefer different cuts

Use a separate 24-row counterexample with twelve labels of each class. Candidate A isolates four negatives; candidate B produces two equally sized, moderately pure leaves. Which kind of purity does each criterion reward more?

Gini prefers B; entropy prefers A. Neither result declares a better model on unseen requests. Return to our eight-row table and search its actual feature gaps.

A stump tests midpoints

A decision stump is a tree with one split. To find its rule, sort one feature, place a candidate between each pair of adjacent values, and score each partition by impurity decrease. The winner is the feature-threshold pair that leaves the children most uniform.

For ambiguity, 3.35 sits between 3.2 and 3.5, separating R4-R6 from the lower-ambiguity rows. With a <= rule, any threshold from 3.2 inclusive to 3.5 exclusive makes the same training partition. A midpoint is a convenient representative, not a requirement for correctness. Only gaps between distinct values matter: equal values can't be separated by this feature.

The search selects ambiguity <= 3.35. Its left leaf reports the observed review rate, $1/5 = 0.20$; its right leaf reports $3/3 = 1.00$. The number in each leaf comes from labels that reached that leaf, not from a smooth curve.

A score of 0.20 isn't automatically "safe." The routing threshold still has to price a missed review against an extra queue item, as it did for logistic regression.

Fit the stump

Read the next cell as the stump's complete loop: scan every feature, try every midpoint, keep the largest gain, and store the two leaf rates. Then use that stored rule to score the training rows.

The returned scores make the tradeoff visible. At routing threshold 0.50, the stump misses R3. A deeper tree could repair this training row, but that repair is a hypothesis about later requests, not evidence that the branch will transfer. This minimal implementation assumes nonempty numeric data with at least one varying feature; a general estimator must also handle constant features, missing values, and invalid inputs.

The simple search rescans labels for every threshold. With $D$ features and up to $N-1$ cuts per feature, that costs roughly $O(DN^2)$ plus sorting. Efficient exact implementations sort and update cumulative class counts; histogram methods search a smaller set of bins. Don't extrapolate this eight-row trainer's performance to large datasets.

Multiplying ambiguity by ten preserves this stump's partitions and rescales its midpoint. A nonlinear transform gives a subtler result. Take two training values, 1 and 4. The original midpoint is 2.5; after squaring, it's 8.5. Which side receives an unseen value of 2.8?

The training partition is identical; the unseen route changes. This counterexample concerns the chosen midpoint, not a new learned relationship. Scikit-learn also converts tree inputs to float32 internally, so extreme scaling can collapse distinct representable values.^{[2]Reference 2Decision Trees: Missing Values Supporthttps://scikit-learn.org/stable/modules/tree.html}

Held-out cost still picks the threshold

Freeze the fitted stump before looking at the later labels. These eight requests didn't choose its split and therefore give the first check on transfer:

Use the same operational costs as logistic regression:

A 0.50 threshold doesn't minimize cost by definition. Before running the cell, predict which threshold wins when a missed review costs almost seven times an unnecessary review. Compare candidates on validation, then confirm the selected policy on a fresh test period before shipping.

Both thresholds tie on F1 here, but they make different mistakes. At 0.20, four false positives cost $72; at 0.50, two false negatives cost $240. The lower threshold buys recall with queue work, which is cheaper on this batch.

Extra depth can memorize R3

A one-split tree is easy to audit and can underfit. An unconstrained tree can isolate each of these distinct training inputs; conflicting labels at identical inputs couldn't be separated. Extra depth becomes overfitting when branches capture quirks that don't transfer. Before growing it, predict the two scores: training accuracy should rise, but V2 may inherit R3's exception leaf.

Grow the same Gini tree until every leaf is pure:

The extra splits isolate R3 between ambiguity 1.75 and 2.40. That fits one training exception; whether the interval captures a reusable rule requires more evidence.

V2 has ambiguity 2.1 and label 0, so it inherits the review leaf and incurs an $18 false-positive cost. V8 has ambiguity 2.8 and label 1, so it falls into the R7/R8 automatic leaf and incurs a $120 false-negative cost.

Training accuracy reaches 1.000. At the same operating threshold 0.50, validation cost actually falls from the stump's $240 to $138; perfect training fit doesn't automatically imply worse held-out performance. The deep tree still costs more than the stump's $72 at threshold 0.20. Greedy, hierarchical splits can be unstable under changes to training data. Tune depth, minimum leaf size, and pruning with representative validation or cross-validation, then verify the selected policy independently.^{[3]Reference 3The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/[4]Reference 4Scikit-learn: Machine Learning in Python.https://jmlr.org/papers/v12/pedregosa11a.html}

Forests average perturbed trees

Bagging (bootstrap aggregating) attacks that high variance by fitting many trees independently in parallel. Each tree receives a bootstrap sample: draw $N$ rows from our $N$-row table with replacement. Some requests repeat; others are left out entirely.

At a fixed input $x$, consider $B$ identically distributed tree scores across training-data and tree-building randomness. Assume each has variance $\sigma^2$ and every distinct pair has correlation $\rho$. Summing their variances and covariances gives:

$$\text{Var}(\bar{T}) = \rho \sigma^2 + \frac{1 - \rho}{B}\sigma^2$$

As $B$ grows, the second term shrinks; the shared variation remains for this assumed building procedure. If $\rho=1$, averaging doesn't reduce variance. This floor isn't irreducible outcome noise or a formula for classification error. Bootstrap randomizations can be independent conditional on one fixed training set while the fitted scores remain correlated across shared random training sets.

Random forests perturb both rows and features to encourage useful disagreement between members.^{[5]Reference 5Random Forestshttps://doi.org/10.1023/A:1010933404324}

1. Bootstrap sampling: Each tree sees a different perturbed slice of data.
2. Feature subsampling: At each split, restrict the candidate features. This can reduce dependence on a dominant predictor, but may also weaken individual trees; tune the tradeoff. In scikit-learn 1.9.1, RandomForestClassifier defaults to max_features="sqrt", while RandomForestRegressor defaults to 1.0, considering all features. The older $D/3$ regression rule is a heuristic, not that library's current default.^{[6]Reference 6Ensembles: Gradient Boosting, Random Forests, Bagging, Voting, Stackinghttps://scikit-learn.org/stable/modules/ensemble.html}

Bootstrap sampling also provides a built-in validation mechanism: Out-of-Bag (OOB) validation. For a dataset of size $N$, the probability that a specific row isn't selected in a bootstrap draw of size $N$ is:

$$\left(1 - \frac{1}{N}\right)^N \xrightarrow{N \to \infty} \frac{1}{e} \approx 0.368 \quad (36.8\%)$$

The 36.8% value is a large-$N$ limit; for our eight rows the expected fraction is about 34.4%. Score each row using only trees whose bootstrap omitted it. OOB evaluation can estimate same-distribution performance without a separate fitting holdout, but isn't automatically unbiased: each row uses fewer trees, and a small forest may leave some rows without any OOB prediction. Breiman explicitly discusses the finite-forest effect.^{[5]Reference 5Random Forestshttps://doi.org/10.1023/A:1010933404324} Ordinary row bootstrapping also doesn't isolate a future period or related requests. Tuning against OOB scores still requires independent verification of the selected policy.

Here we average leaf class-frequency scores, as scikit-learn's random-forest classifier does.^{[7]Reference 7RandomForestClassifierhttps://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html} Breiman's original classifier used hard majority voting. Averages preserve more score detail than hard votes; neither makes the probabilities calibrated automatically.

With two features, use one random feature per split (mtry = 1) to make that disagreement obvious. Predict R3's forest score before running the cell: if two trees never draw R3 and one tree over-samples it, the mean should land around 0.33, rather than becoming a hard majority vote of 0.

Tree 1 never draws R3, so its low-ambiguity leaf is pure automatic and scores R3 as 0.00. Tree 2 over-samples R3 and splits on policy support, scoring it as 1.00. Tree 3 also drops R3.

The mean 0.33 is the forest's review score for that exception: two trees never saw it, and one tree treated it as a review case. It's an average of tree scores, not a new label.

Our toy considers exactly the sampled features at each node and stops if none yields a valid partition. Library split search can inspect additional features to find a valid partition. This demonstrates row/feature perturbation and averaging; it isn't a full clone of RandomForestClassifier.

Validation check: On validation, threshold 0.20 costs $54. V1's score is 0.00, a true negative, while the other low-ambiguity rows go to review. That is a real improvement on this slice, not a production bake-off. Eight rows plus three trees demonstrate the mechanism.

R3's OOB score is 0.00, using only trees 1 and 3; tree 2 must be excluded because its bootstrap contains R3. The full-forest mean 0.33 isn't an OOB score. With eight rows and three trees, neither supports a release estimate.

More trees stabilize the average of this tree-building procedure; they don't repair a bad feature set or a biased dataset. Individual forest trees often fit training data closely, and their average can still generalize well. Select depth and leaf size using validation rather than assuming that deeper members necessarily worsen the ensemble.

The forest emits more varied scores than the stump. Its threshold still belongs to the routing policy and must be evaluated with the same costs.

Boosting fits the leftover error

Gradient boosting also combines trees, but its trees take turns sequentially. Instead of fitting independent trees on perturbed data in parallel, each new tree learns to correct the leftover mistakes of the existing ensemble.

In standard gradient descent, we update model parameter weights $w$ in the direction of negative parameter gradients: $w \leftarrow w - \eta \nabla_w L$. In gradient boosting, we perform gradient descent in function space^{[8]Reference 8Greedy Function Approximation: A Gradient Boosting Machinehttps://projecteuclid.org/journals/annals-of-statistics/volume-29/issue-5/Greedy-Function-Approximation--A-Gradient-Boosting-Machine/10.1214/aos/1013203451.full}. Our "parameters" are the predicted outputs $F(x_i)$ at each training example.

To minimize an arbitrary differentiable loss function $L(y, F(x))$, the steepest descent step is the negative gradient of the loss with respect to the current predictions:

$$r_{im} = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]_{F(x) = F_{m-1}(x)}$$

We call $r_{im}$ a pseudo-residual.

For squared-error regression $L(y, F) = \frac{1}{2}(y - F)^2$, the negative gradient is simply the familiar arithmetic error:

$$-\frac{\partial L}{\partial F} = y - F(x)$$

The target for each new tree is literally the residual error $y - \hat{y}$.

Before returning to binary review routing, predict one correction step on continuous handling time. For ambiguity values [1, 2, 5] and handling times [10, 20, 60], the initial prediction is their mean, 30. The first two rows need downward corrections, while the last needs a large upward correction.

Why scale the update by learning rate (also known as shrinkage) $\eta = 0.10$?

At $\eta=1$, this step gives [15, 15, 60]: it fits the last row exactly but still misses the first two. Shrinkage scales each added correction and changes the learning path; it isn't the explicit $L_2$ leaf penalty introduced below. Smaller rates often need more rounds, and don't guarantee diverse trees or prevent noise fitting. Tune the rate together with tree complexity and the stopping round on appropriate validation data.^{[6]Reference 6Ensembles: Gradient Boosting, Random Forests, Bagging, Voting, Stackinghttps://scikit-learn.org/stable/modules/ensemble.html}

The next cell repeats that update four times. Watch whether each residual stump keeps the same threshold and whether training MSE falls:

Classification boosting uses y minus p

For access routing, we need probabilities rather than handling minutes. Let $F$ be the current additive logit score, and let $p = \sigma(F) = \frac{1}{1 + e^{-F}}$ be the review probability.

For a single request with true label $y \in \{0, 1\}$, the binary cross-entropy loss is:

$$L(y, F) = -[y \ln \sigma(F) + (1-y)\ln(1-\sigma(F))]$$

Differentiating with respect to the logit $F$ using the chain rule ($\frac{d\sigma}{dF} = p(1-p)$):

$$\frac{\partial L}{\partial F} = p - y$$

The pseudo-residual (the negative gradient) is therefore:

$$r = -\frac{\partial L}{\partial F} = y - p$$

When $y = 1$ and the current probability is low (like $p = 0.20$), the residual is positive ($+0.80$), asking for a higher logit. When $y = 0$ and $p = 0.80$, the residual is negative ($-0.80$), asking for a lower logit.

For an approximate Newton leaf update, also compute the second derivative:

$$h = \frac{\partial^2 L}{\partial F^2} = \frac{\partial (p - y)}{\partial F} = p(1-p)$$

Our gradient-boosting fixture fits split regions by squared error on $y-p$, then takes one Newton step in each fixed leaf:^{[8]Reference 8Greedy Function Approximation: A Gradient Boosting Machinehttps://projecteuclid.org/journals/annals-of-statistics/volume-29/issue-5/Greedy-Function-Approximation--A-Gradient-Boosting-Machine/10.1214/aos/1013203451.full}

$$\gamma_j = \frac{\sum_{i \in I_j} (y_i - p_i)}{\sum_{i \in I_j} p_i(1 - p_i)}$$

Start at $F=0$, the best constant unpenalized logit for these balanced training labels. In general, a constant baseline uses the training positive fraction $\pi$ through $\log(\pi/(1-\pi))$ when $0<\pi<1$. Here $p=0.5$ and every curvature term is $0.25$; the residual stump splits at ambiguity <= 3.35.

• Left leaf ($5$ rows: R1, R2, R3, R7, R8): One review ($y=1$) and four automatic ($y=0$). The residual sum is $1 \times (1 - 0.5) + 4 \times (0 - 0.5) = 0.5 - 2.0 = -1.5$. $$\gamma_{\text{left}} = \frac{-1.5}{5 \times 0.25} = -1.20$$
• Right leaf ($3$ rows: R4, R5, R6): Three reviews ($y=1$). The residual sum is $3 \times (1 - 0.5) = +1.5$. $$\gamma_{\text{right}} = \frac{1.5}{3 \times 0.25} = +2.00$$

With learning rate $\eta = 0.10$, the updated logits become $0 + 0.10(-1.20) = -0.12$ on the left and $0 + 0.10(2.00) = +0.20$ on the right. The corresponding probabilities shift to $\sigma(-0.12) \approx 0.470$ and $\sigma(0.20) \approx 0.550$. Notice the contrast with an isolated decision stump: boosting doesn't overwrite predictions with extreme 0.20 and 1.00 probabilities. It nudges the ensemble in the direction of the gradient.

The Newton values aren't exact minimizers of the original log loss. Starting at zero, the left leaf's exact constant optimum would be $\operatorname{logit}(0.20)\approx-1.386$, not $-1.20$. The all-positive right leaf has no finite unpenalized optimum, despite its finite first Newton step of $+2$. A quadratic approximation, a step-size choice, and an exact optimum are different objects.

XGBoost integrates gradient and curvature statistics directly into its split score, rather than selecting the tree solely by residual squared error.^{[9]Reference 9XGBoost: A Scalable Tree Boosting System.https://www.kdd.org/kdd2016/papers/files/rfp0697-chenAemb.pdf[10]Reference 10Introduction to Boosted Treeshttps://xgboost.readthedocs.io/en/stable/tutorials/model.html} For this binary log-loss objective, its second-order approximation is:

$$\mathcal{L}^{(t)} \approx \sum_{i=1}^n \left[ g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i) \right] + \gamma_{\text{split}} T + \frac{1}{2}\lambda \sum_{j=1}^T w_j^2$$

where $g_i = p_i - y_i$ is the gradient, $h_i = p_i(1 - p_i)$ is the Hessian, $T$ is the number of leaves, and $\lambda$ is an $L_2$ leaf regularizer.

Minimizing that quadratic approximation for a fixed tree gives:

$$w_j^* = -\frac{\sum_{i \in I_j} g_i}{\sum_{i \in I_j} h_i + \lambda} = \frac{\sum_{i \in I_j} (y_i - p_i)}{\sum_{i \in I_j} p_i(1 - p_i) + \lambda}$$

With $\lambda=0$ and positive summed curvature, this is the same Newton step. Positive $\lambda$ shrinks its magnitude for a fixed numerator and curvature; it doesn't guarantee calibration or exclude extreme predictions. Define $G_L,G_R$ as summed gradients and $H_L,H_R$ as summed Hessians. The proposed split's improvement, including the extra-leaf charge, is:

$$\text{gain}=\frac12\left[\frac{G_L^2}{H_L+\lambda}+\frac{G_R^2}{H_R+\lambda}-\frac{(G_L+G_R)^2}{H_L+H_R+\lambda}\right]-\gamma_{\text{split}}$$

For our first stump with $\lambda=0$, the improvement before the charge is 2.4; a charge greater than 2.4 makes this proposal unfavorable. Once the charge has been subtracted, compare gain with zero, not with the charge again. Actual growth also obeys estimator constraints such as depth and minimum child weight.^{[10]Reference 10Introduction to Boosted Treeshttps://xgboost.readthedocs.io/en/stable/tutorials/model.html}

Predict the validation consequence before running the four-round boosting code. The probability loss should fall, but if V3 and V8 stay on the low-ambiguity side, threshold 0.50 will still miss them.

Validation log loss falls from 0.693 to 0.584; all four rounds retain the original split. Cost at 0.50 stays $240 because V3 and V8 remain on the low-ambiguity side. A better probability loss needn't improve a selected policy. Choose rounds and thresholds on validation, freeze both, then evaluate a locked test period.

The stress check computes loss from finite logits, preserving a confidently wrong loss of 1000 even when sigmoid rounds to 1. Rounded probabilities can also make curvature zero; the unpenalized helper reports that condition instead of silently returning a zero update. An L2 penalty can make this leaf's quadratic problem well defined, but changes the fitted objective. This small trainer assumes valid 0/1 labels, finite features, and a varying feature; it's a demonstration, not a general production estimator.

The default unweighted Gini and log-loss criteria don't price a false negative at $120 while fitting. When false negatives cost far more than false positives, per-row sample weights can make high-harm rows influence splits. Sweep the decision threshold on validation anyway: weights change the fit, while the threshold turns scores into actions.

Leaf frequencies and forest averages are ranking scores, not automatically honest probabilities. The logistic chapter measured expected calibration error on these same eight validation rows. Treat ensemble predict_proba as a score until a reliability audit on a reserved split says otherwise.

A tree prediction is constant outside the training support of each split. If production ambiguity climbs past every training value, the leaf still returns the training rate for that region. Monitor feature ranges and don't expect the smooth extrapolation a linear model might give.

What bias and variance can tell us

The familiar decomposition is for squared prediction error, not every classification metric. At a fixed input $x$, let $Y=f_*(x)+\epsilon$, with zero-mean noise independent of the training data and variance $\sigma_\epsilon^2$. Across random training sets and fitting seeds $D$:

$$\mathbb E_{D,\epsilon}[(Y-\hat f_D(x))^2] =\underbrace{(\mathbb E_D[\hat f_D(x)]-f_*(x))^2}_{\text{squared bias}} +\underbrace{\operatorname{Var}_D(\hat f_D(x))}_{\text{variance}} +\underbrace{\sigma_\epsilon^2}_{\text{outcome noise}}$$

Perfect training accuracy doesn't measure statistical bias. A singleton tree can fit all eight labels while producing systematically wrong scores elsewhere. A stump can also be unstable if small data changes switch its best feature or threshold.

Averaging identically distributed member predictions preserves their common expectation and can reduce variance through imperfect correlation. Changing bootstrap, feature, depth, or leaf-size rules changes the member distribution too, so a forest isn't guaranteed to have the same bias as an arbitrary deep tree.

Boosting expands an additive function to improve a chosen loss. It can reduce approximation error, but its fitted variance can also change. More rounds can overfit; eventual memorization isn't guaranteed, especially with restricted learners or contradictory labels at identical inputs. Depth, learning rate, sampling, regularization, and stopping interact. For binary review routing, compare the resulting validation costs and calibration directly.^{[6]Reference 6Ensembles: Gradient Boosting, Random Forests, Bagging, Voting, Stackinghttps://scikit-learn.org/stable/modules/ensemble.html}

MDI can crown a useless ID

Tree libraries often report mean decrease in impurity (MDI) as default feature importance. Each split contributes its impurity decrease weighted by the fraction of training samples reaching that node. Contributions are grouped by feature, normalized, and averaged across trees. It's fast, but high-cardinality features offer many candidate thresholds and more chances to fit accidental patterns. A numeric request ID can receive large importance without predicting later labels.^{[11]Reference 11Permutation Feature Importancehttps://scikit-learn.org/stable/modules/permutation_importance.html}

To isolate that failure, use a separate synthetic experiment. A binary feature agrees with the label 70% of the time; a continuous feature is independent uniform noise. Both training and validation come from the same process. Scikit-learn supplies the forest and importance calculations here, so we don't need a second implementation of the tree builder. Forty training rows leave plenty of opportunity to overfit; 2,000 held-out rows give a less noisy comparison.

Permutation importance asks how this fitted model's selected metric changes when one feature is shuffled across held-out rows. Marginal shuffling breaks that feature's relationship with labels and other features, potentially creating unusual combinations. It doesn't measure intrinsic feature quality or establish a causal effect.^{[11]Reference 11Permutation Feature Importancehttps://scikit-learn.org/stable/modules/permutation_importance.html}

In this seeded experiment, MDI assigns 0.920 to continuous noise, while its held-out permutation F1 drop is only 0.003. The binary predictor's drop is 0.097. This exposes a misleading training-importance ranking. The reported 0.007 shuffle SD describes repeated permutations on the fixed model and data; it isn't a confidence interval over new datasets, refits, or production traffic.

With strongly correlated features, another feature may retain redundant information, making individual permutation effects small.^{[12]Reference 12Permutation Importance with Multicollinear or Correlated Featureshttps://scikit-learn.org/stable/auto_examples/inspection/plot_permutation_importance_multicollinear.html} This isn't universal: if a fitted tree relies entirely on one copy, shuffling that copy can still hurt. Consider a joint permutation of a feature group or a justified conditional method, and state which question it answers.

An importance score doesn't certify model quality. For access routing, include permutation effects on the declared cost at the frozen threshold, alongside recall and queue capacity. F1 alone can't express the two error prices.

A local explanation isn't a verdict

Global importances tell you which features matter across an entire dataset. They don't explain why the model made a specific decision on request R3.

SHAP (SHapley Additive exPlanations) solves this by adapting Shapley values from cooperative game theory to machine learning models^{[13]Reference 13A Unified Approach to Interpreting Model Predictionshttps://arxiv.org/abs/1705.07874}. It decomposes an individual prediction $f(x)$ into an additive sum of a baseline expectation plus individual feature contributions:

$$f(x) = \phi_0 + \sum_{j=1}^M \phi_j(x)$$

For our chosen background game, $\phi_0=\mathbb E[f(X)]$. The original SHAP paper identifies three properties: local accuracy, missingness in the simplified feature-presence representation, and consistency. Consistency requires the feature's marginal contribution not to decrease for any coalition when the model changes; then its attribution must not decrease. The reconstruction above expresses local accuracy. These properties concern an attribution game; they don't validate labels or establish causal effects.^{[13]Reference 13A Unified Approach to Interpreting Model Predictionshttps://arxiv.org/abs/1705.07874}

The baseline and contributions depend on the background data and how absent features are replaced. Here we explain the stump's review probability using an interventional background: hold selected features at the request's values and fill the others from each training row, without conditioning on the held features.^{[14]Reference 14shap.TreeExplainerhttps://shap.readthedocs.io/en/latest/generated/shap.TreeExplainer.html}

With two features, a Shapley contribution averages the change from adding that feature first and adding it second. Our stump only uses ambiguity, so revealing policy support never changes its output under this replacement rule. That gives a clean, exact check:

• Baseline review score across the training background: $0.50$.
• Left-leaf score: $0.20$, so ambiguity contributes $-0.30$.
• Right-leaf score: $1.00$, so ambiguity contributes $+0.50$.
• auto_policy_support contributes 0.00 under the stated interventional calculation.

R3 exposes the divide between explanation and decision correctness. The attribution faithfully reports why the stump scored the request low: its ambiguity of 2.0 placed it in the low-ambiguity leaf, contributing $-0.30$. But that doesn't prove automatic processing was safe! R3's true label is $y = 1$ (it required human review). Local explanations audit what a model computed; they don't validate real-world ground truth.

If instead the explainer conditions on correlated features, revealing policy support can convey information about ambiguity even though the tree never splits on support. Attribution then answers a different question. Always document whether your SHAP values use interventional or conditional expectations rather than assuming one attribution is uniquely defined.

With TreeExplainer, also record model_output and feature_perturbation. Its default raw output depends on the estimator; for binary XGBoost, raw attributions reconstruct log odds, not a percentage. Current auto chooses interventional replacement when a background is supplied and tree-path-dependent replacement otherwise. Tree-path-dependent replacement isn't a general conditional-expectation calculation.^{[14]Reference 14shap.TreeExplainerhttps://shap.readthedocs.io/en/latest/generated/shap.TreeExplainer.html}

Missing values are a library contract

"Trees handle missing data automatically" is a half-truth that breaks pipelines. Before moving an estimator, check which component owns the missing-value route. The behavior depends on library, estimator, and version:

The following contracts were checked in scikit-learn 1.9.1, XGBoost 3.4.2, and LightGBM 4.7.0 documentation on September 21, 2026. Our hand-written trainers don't implement native missing-value routing.

The trap is assuming the behavior is uniform. Moving a pipeline between estimators or versions can turn native routing into an input error.

The quieter trap changes data without raising one. zero_as_missing and sparse-matrix conventions mean a literal 0 and a missing value can be treated identically or differently depending on the tool, which changes splits. Decide the missing-value contract explicitly: impute, encode a missingness flag, or rely on documented native behavior. A tree won't infer that contract for you.

Pick a model the same way you picked logistic

Use the same later-request contract for every candidate. Compare a regularized tree or ensemble with logistic regression under identical splits and costs, then inspect whether added flexibility repairs meaningful failures. Read the table below as a set of decisions, not a leaderboard: the cheapest row still rests on eight validation requests.

The forest improves on our stump, but it doesn't beat the logistic baseline on this batch. Greater flexibility isn't a reason to replace a working model. We used these rows to inspect and select policies, so even the logistic result needs a fresh verification set before making a performance claim.

Trees expose conditional splits. Forests reduce single-tree instability by averaging perturbed members. Boosting accumulates targeted corrections. Each mechanism changes a different failure mode, and none replaces held-out evaluation.

Change the example, then check your reasoning