Forecasting and Anomaly Detection

Ranking and recommendation chose an ordered slate at one moment. Forecasting asks the next production question: how many batch jobs will run tomorrow, and which observed counts fall outside that expectation?

Forecasting predicts a value for a later time. Anomaly detection asks whether an observation is unusual enough to inspect. The two jobs fit together: first estimate expected batch-job volume, then compare the actual count with that expectation. Monitoring chapters later reuse the same residual signal as a live operational check.

Start with an ordered batch-job series

A time series is a sequence of observations indexed by time. This lesson uses daily outbound batch-job counts for one model-serving cluster. Each row has a date, a batch-job count, and a flag for a planned model launch.

The forecast horizon is how far ahead you predict. A one-day horizon answers tomorrow's staffing question. A seven-day horizon helps reserve capacity for the coming week. This lab predicts one day at a time, then evaluates a full future week.

Build four weeks of daily counts. The weekday pattern repeats, but small changes keep the series realistic. The final Friday launch event creates a larger jump.

The fixture stays inspectable. Production evaluation should report separate series or slices by model-serving cluster, service tier, and forecast horizon.

Keep future rows out of training

Ordinary tabular prediction often starts with a random train/test split. That's unsafe for forecasting. A shuffled training set can contain January 31 while its test set contains January 6. A model evaluated that way has learned from the future relative to some test rows.

Compare an invalid shuffled split with a chronological split. The audit prints whether the latest training date reaches or passes the earliest test date.

Hyndman and Athanasopoulos describe time-series cross-validation as repeated evaluation where each test observation occurs after the observations used for training.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/} You can expand the training history or keep a rolling window. Either way, the time direction stays intact.

Why isn't a random split acceptable just because every row belongs to the same cluster?

Answer

Cluster identity doesn't remove time order. A shuffled split can train on later demand, launch events, and seasonal behavior before evaluating earlier dates. Production only predicts later periods from earlier information.

Establish a same-weekday baseline

Batch-job volume has seasonality when a pattern repeats at a known calendar frequency. Saturday volume may stay lower than weekday volume. Predicting tomorrow from yesterday alone will overreact around each weekend.

A seasonal naive forecast reuses the observation from the matching previous season. For daily batch job data with a weekly pattern, forecast Monday from last Monday, Tuesday from last Tuesday, and so on.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/}

The Friday arithmetic is 160 - 132 = 28. Positive error means the observed count exceeded the forecast. Negative error means the forecast was too high.

Compute the complete holdout week. Each forecast reads the value exactly seven rows earlier.

Simple baselines matter. A complex model should beat the seasonal naive forecast on future windows before its extra features and operational cost are justified.

Measure forecast error in batch jobs

Mean absolute error (MAE) averages the size of each mistake without letting positive and negative errors cancel:

$$\text{MAE} = \frac{1}{n}\sum_{t=1}^{n}|y_t - \hat{y}_t|$$

Here $n$ is the number of evaluated days, $y_t$ is observed volume on day $t$, and $\hat{y}_t$ is the forecast for that day. For the holdout week, the absolute errors are 1, 3, 2, 1, 28, 1, 1. Their sum is 37, so MAE is 37 / 7 = 5.3 batch jobs.

Calculate overall MAE and normal-day MAE separately. The second metric excludes the known launch event day so you can see baseline behavior without hiding the operational spike.

MAE remains in batch jobs, which makes it understandable to operations. It doesn't tell you everything: underforecasting a capacity spike may cost more than overforecasting by the same amount. Record asymmetric business cost when the decision requires it.

Backtest more than one future window

A single future week is fragile evidence. Rolling-origin evaluation moves the forecast origin forward and measures several later windows. In this fixture, the first fold evaluates January 19 through January 25. The second evaluates January 26 through February 1.

Run both folds. The seasonal baseline can forecast each day because each fold starts after at least one full week of history.

Backtest by model-serving cluster, service tier, weekday, and launch-event status. A global average can hide one cluster that routinely underforecasts peak demand. The target must also match the decision: submitted job count helps GPU reservation, failed job count helps incident response, and queue-delay count helps operator notification.

Distinguish forecast errors from residuals

Two related quantities often get called residuals, but the distinction is useful:

Hyndman and Athanasopoulos make this distinction explicitly.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/} Our Friday +28 is a forecast error because Friday's observed volume wasn't available when the forecast was issued. An operations team may casually say "residual alert," but the stored artifact should name the measured quantity precisely.

Calibrate an error band from earlier days

A point forecast such as 132 batch jobs doesn't express uncertainty. A prediction interval pairs a forecast with a lower and upper bound intended to cover a stated proportion of future outcomes. Forecast intervals should be evaluated for coverage, not displayed beside the point estimate alone.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/}

For a small first-principles exercise, use the 95th percentile of earlier absolute one-day forecast errors. This produces a symmetric empirical error band. The code deliberately calibrates on January 12 through January 25, before the January 26 holdout starts, then measures coverage again on the later holdout week.

The ±3 band explains the mechanics, not a production guarantee. Fourteen calibration errors are too few for a stable 95% interval. Its in-sample calibration coverage is 1.000, while later holdout coverage is 6 / 7 = 0.857 because Friday falls outside the band. A production system should measure out-of-sample interval coverage across rolling windows, forecast horizons, model-serving clusters, and important demand slices.

Why doesn't the tiny ±3 band prove that future batch-job counts will land inside it 95% of the time?

Answer

It comes from only fourteen earlier errors. In-sample calibration coverage tells you how the band was chosen; later holdout coverage checks whether it transferred to unseen days. One holdout week still isn't enough. A production interval needs enough history plus out-of-sample coverage checks across rolling windows, forecast horizons, and important slices.

Turn an unusual error into a review artifact

An anomaly is an observation unusual enough to inspect. It isn't proof that a cluster, pipeline, or model failed. Friday's jump could come from a planned model launch, duplicated event ingestion, a large backfill, or genuine demand growth.

Apply the empirical band to the held-out week. The alert stores enough context to reproduce the comparison.

The baseline caught Friday because 160 falls outside [129, 135]. The launch-event flag doesn't erase the event. It changes the first investigation step.

Route alerts with context. A planned model launch still deserves capacity review, but it shouldn't automatically create an incident page.

Log the forecast version, training cutoff, interval policy version, observed value, error, known-event flags, owner, and eventual resolution. That receipt helps operations act now and helps future model reviews learn from the alert.

Evaluate alert usefulness separately

Low MAE doesn't guarantee useful alerts. An alert threshold can still page too often or miss disruptions. Once reviewers label whether each historical alert was actionable, measure alert precision and recall:

Compute both metrics on a five-event resolution fixture. The policy surfaced two useful events, sent one noisy alert, and missed one disruption.

Tune the policy against operational cost. A missed cluster disruption and a noisy review ticket don't have equal consequences. If a denominator is zero, report None: no evidence isn't the same as measured failure. Page only when urgency warrants interruption; otherwise queue a review with the same evidence.

Reproduce a data-quality anomaly

Not every spike represents demand. A duplicated telemetry event can inflate a count before forecasting code sees it. Reproduce that failure with three raw telemetry rows, two of which describe the same batch job event.

An anomaly review should inspect input quality before retraining a model. Retraining on duplicated counts would teach the model to copy a pipeline bug. This tiny dictionary deduper keeps one row because the repeated payloads match. If two rows reuse an event ID but disagree, quarantine the conflict instead of silently choosing one.

Why shouldn't the alert policy declare an incident as soon as a count crosses the error band?

Answer

The forecast error proves that the observation is unusual relative to the baseline. It doesn't identify the cause. Known launch events, duplicated ingestion, and genuine demand shifts require different responses.

Gate and publish a forecast candidate

A minimal release gate should prove time order, calibration order, normal-day accuracy, interval bookkeeping, contextualized alerts, and assigned ownership. These checks don't prove the baseline is production-ready, but they prevent avoidable mistakes from reaching a capacity workflow.

Run the gate.

Publish a receipt. The hash lets a later monitoring job identify the exact baseline, interval, and input policies it evaluates.

The receipt keeps calibration evidence separate from later holdout evidence and binds the policies that produced both. Passing these gates earns shadow evaluation, not a production rollout.

When the policy breaks