Gradient Boosted Trees in Production

Training-job rows now have input size, heartbeat age, backlog, priority class, and a later label for whether the job missed its SLA (service-level agreement). Those rows still aren't a decision. You need a score for upcoming jobs and a published rule for when to intervene.

A gradient boosted tree is a strong default on this kind of table.^{[1]Reference 1Why do tree-based models still outperform deep learning on typical tabular data?https://arxiv.org/abs/2207.08815} You already saw the mechanics in Decision Trees, Forests, and Boosting: each new tree fits leftover error. A candidate ships only when it beats a declared baseline on later jobs, survives required slices, and travels with enough evidence to reproduce the call.

Freeze the clock before training

Suppose missed_sla = 1 means a training job missed its promised finish time. Before any fitting, publish a time-ordered split:

A random shuffle can put the same disruption into train and test. Calendar order is closer to facing tomorrow's jobs. It still doesn't erase every dependency. If one incident can contribute several rows, document the split unit and keep those rows together whenever splitting them would make holdout look easier than production.

scikit-learn's leakage page says the same thing: don't train on information that wouldn't be available at prediction time.^{[2]Reference 2Common Pitfalls and Recommended Practices: Data Leakagehttps://scikit-learn.org/stable/common_pitfalls.html#data-leakage} The point-in-time replay from Batch and Streaming Feature Pipelines is how this table stays honest. A row created after its decision timestamp is feature leakage and doesn't belong in training.

Build a deterministic fixture that follows that calendar. Labels depend on heartbeat age, backlog, service tier, a large-model flag derived from input size, and a little noise. May also gets a small drift term, so the test month isn't a copy of training.

The fixture is synthetic so you can rerun the whole lab. In a real job, X has to come from that point-in-time contract, not from a table snapshot taken after the decision.

Publish the cheapest baseline

Your first candidate can be a rule: predict an SLA miss when heartbeat_age_hours >= 18. It's weak, and that's the point. The boosted model has to beat something you'd actually run, not get credit for any nonzero score.

Define one cost policy before comparing models:

• A missed priority job costs 150.
• A missed standard job costs 60.
• A false alarm costs 8.

The exact numbers will differ by product. Publishing them matters because model quality is inseparable from the action the score triggers.

The rule already catches 48 of April's 58 SLA misses, at cost 1560. Extra trees have to improve that decision, not just emit a prettier probability.

Why publish the heartbeat-age rule before evaluating the boosted model?

Answer

The cheap rule establishes the decision quality already available without model complexity. The booster must beat that baseline under the same held-out slice and cost policy.

Rebuild one boosting step by hand

A shallow decision tree partitions rows into a few rules. Gradient boosting adds those trees one at a time: each new tree moves predictions toward errors left by the current ensemble. Friedman describes this as stage-wise function approximation using loss gradients.^{[3]Reference 3Greedy Function Approximation: A Gradient Boosting Machinehttps://projecteuclid.org/journals/annals-of-statistics/volume-29/issue-5/Greedy-Function-Approximation--A-Gradient-Boosting-Machine/10.1214/aos/1013203451.full}

For intuition, switch from the binary SLA label to delay hours on three jobs:

A small correction tree might add little for local jobs and more for long economy workloads. For squared-error regression, its training target is the residual actual - prediction. A learning rate applies only part of that correction, so one noisy tree can't dominate.

The tree proposed -1 hour for shorter workloads and 14 hours for the large-model job. Shrinkage kept only a quarter of that move, so the large-model prediction went from 6 to 9.5, not all the way to 20. One round lowers mean absolute error without pretending the first correction is finished.

Friedman writes that update as stage-wise addition with shrinkage. If $F_{m-1}$ is the current score and $h_m$ is the new tree,

$$F_m(x) = F_{m-1}(x) + \nu \, h_m(x)$$

where $\nu$ is the learning rate (here $0.25$).^{[3]Reference 3Greedy Function Approximation: A Gradient Boosting Machinehttps://projecteuclid.org/journals/annals-of-statistics/volume-29/issue-5/Greedy-Function-Approximation--A-Gradient-Boosting-Machine/10.1214/aos/1013203451.full}

Classification needs one extra translation. An SLA-risk classifier doesn't fit delay-hour residuals. With log loss, each new tree fits the negative gradient of that loss. At an initial probability of 0.50, that negative binary log-loss gradient is gold - probability. Equivalently, the score gradient $\partial L/\partial F$ is probability - gold.

Negative values push SLA-risk scores down. Positive values push them up. A real classification booster repeats this across many rows and trees.

When the table gets large or contains high-cardinality categoricals, production teams often choose between three battle-tested boosting systems:

• XGBoost: Chen and Guestrin built it with a regularized objective ($\gamma T + \frac{1}{2}\lambda \sum w_j^2$), sparsity-aware split finding that learns default directions for missing values, weighted quantile sketching for approximate splits, and cache-aware column blocks for parallel split finding.^{[4]Reference 4XGBoost: A Scalable Tree Boosting System.https://www.kdd.org/kdd2016/papers/files/rfp0697-chenAemb.pdf}
• LightGBM: Ke et al. optimized training throughput using histogram-based feature binning, leaf-wise (best-first) tree growth with a max-depth limit, and native integer categorical handling via optimal histogram partitioning without one-hot expansion.^{[5]Reference 5LightGBM: A Highly Efficient Gradient Boosting Decision Tree.https://papers.nips.cc/paper/2017/hash/6449f44a102fde848669bdd9eb6b76fa-Abstract.html}
• CatBoost: Prokhorenkova et al. implemented symmetric (oblivious) trees for fast inference, ordered boosting to prevent target leakage during target encoding, and automated feature combinations for high-cardinality categoricals.^{[6]Reference 6CatBoost: unbiased boosting with categorical features.https://arxiv.org/abs/1706.09516}

The evidence contract doesn't change with the library. When downstream decision policies depend on well-calibrated probabilities rather than rank order alone, evaluate calibration (reliability curves, Brier score) or apply post-hoc calibration like isotonic regression or Platt scaling on the validation split before freezing the threshold policy.^{[7]Reference 7Predicting Good Probabilities With Supervised Learning.https://dl.acm.org/doi/10.1145/1102351.1102430}

Fit a booster with an explicit stopping rule

This lab uses scikit-learn's GradientBoostingClassifier so the code stays small. It builds an additive model stage by stage and fits each tree to the negative gradient of the chosen loss, here log_loss. A larger job might use XGBoost. You'd still freeze rounds on April, not on training loss.

Fix tree depth and learning rate first. Train a generous maximum number of rounds on January through March, then inspect April loss after each stage.

Don't use sklearn's validation_fraction helper for this split. That flag randomly holds out rows from the training matrix. Here the validation slice is a later month, so you own the loop with staged_predict_proba on April.

Training loss keeps falling through round 160. April loss bottoms at round 50, then gets worse. Extra trees still helped the training objective. They stopped helping future jobs. That's the overfitting symptom early stopping is for.

Some libraries expose built-in early stopping flags. A production job still needs to record the exact validation slice, monitored metric, patience policy, and selected round count. Here the rule is visible: choose the stage with minimum April log loss, then fit the candidate with that frozen count.

The booster improves April AUC from 0.731 to 0.792. Area under the receiver operating characteristic curve (AUC) summarizes ranking quality across thresholds. It doesn't choose an operations policy.

Validation loss is lowest at round 50 and rises through round 160. Which round count should the frozen candidate use?

Answer

Use round 50, selected on the declared validation slice. Continuing to the training cap would keep trees that improved training fit but worsened held-out loss.

Measure threshold cost, not AUC alone

The classifier outputs an SLA-risk score. Operations still needs a choice: intervene, notify, or leave the job on its normal path. A missed SLA on priority jobs may be more expensive than an unnecessary proactive notification.

Search candidate thresholds on April only. The sort key minimizes declared cost and uses the lower threshold as a tie-breaker.

Threshold 0.15 beats the heartbeat-age rule on April (1316 vs 1560) and sits far below the common default of 0.50. Missing a priority SLA is expensive in this policy, so the model accepts more false alarms. If alert fatigue mattered more, the cost table would need another term.

Now freeze the round count, threshold, and cost policy. May can be opened once for final candidate evidence.

May can look better or worse than April. Here the frozen booster still beats the same heartbeat-age rule (728 vs 1044). You don't retune. The job of May is to confirm the frozen policy, not to pick a new one. Aggregate AUC and cost can still hide a serious workload failure.

Gate slices before promotion

Pick required slices before reading test results. A slice gate also needs enough positive cases to mean anything: perfect recall on one SLA-miss job is weak evidence, and an empty slice can't produce recall at all. This lab requires at least 10 SLA-miss jobs per slice, recall of at least 0.90 for priority jobs, and recall of at least 0.85 for large-model jobs.

Both required slices pass for the frozen candidate. Keep the gate anyway. A future retrain, feature drift, or threshold edit can break one segment while aggregate metrics still look acceptable.

To see the failure symptom, test a careless threshold edit from 0.15 to 0.50.

Nothing about the fitted trees changed. The threshold edit alone drops priority recall from 0.944 to 0.722. Versioning only the model file would miss the regression.

Why doesn't a higher AUC choose the production intervention threshold?

Answer

AUC measures ordering across thresholds. Operations still needs a threshold chosen against concrete false-alarm and missed-SLA costs, including critical slices such as priority jobs.

Ship a model that can be operated

The candidate should export:

Create a compact receipt for the candidate. The real deployment bundle would persist the estimator with a format you can defend. scikit-learn's current guidance is: use ONNX when you only need scores and don't need the Python object back; use skops.io when you do need the object and want to inspect untrusted types before loading; treat pickle, joblib, and cloudpickle as trusted-source-only because loading them can execute code. None of those Python formats support loading an estimator trained on a different scikit-learn version, so pin Python, scikit-learn, NumPy, SciPy, and the serializer together.^{[8]Reference 8Model persistencehttps://scikit-learn.org/stable/model_persistence.html}

The receipt binds feature semantics, fitted-tree choices, threshold policy, split dates, measured holdout policy, and support-aware slice evidence. Shadow traffic and production monitoring still come next. Don't mutate the live policy in place when retraining changes those values.