Forecasting and Anomaly Detection

Friday's cluster ran 160 outbound batch jobs. Last Friday's same-weekday copy said 132. Is tomorrow a staffing problem, or just Friday?

Ranking and recommendation ordered documents for one query at one moment. Capacity planning looks forward in time. Forecasting predicts a later value. Anomaly detection asks whether an observation is unusual enough to inspect. Start with expected volume, then compare the actual count with that expectation. The leftover is the operational signal: unusual enough to inspect, not yet a root cause.

Start with an ordered batch-job series

A time series is a sequence of observations indexed by time. The series here is daily outbound batch-job counts for one model-serving cluster. Each row has a date, a batch-job count, and a flag for a planned model launch. The Friday 160 versus 132 gap is one row in that series.

The forecast horizon is how far ahead you predict. A one-day horizon answers tomorrow's staffing question. A seven-day horizon helps reserve capacity for the coming week. With a weekly seasonal naive baseline, those two horizons use the same numbers: from Sunday's origin, each of the next seven weekdays copies last week's matching day.

Build four weeks of daily counts. The weekday pattern repeats, but small changes keep the series realistic. The final Friday launch event creates a larger jump.

The fixture stays inspectable. Production evaluation should report separate series or slices by model-serving cluster, service tier, and forecast horizon.

Keep future rows out of training

Ordinary tabular prediction often starts with a random train/test split. That's unsafe for forecasting. A shuffled training set can contain January 31 while its test set contains January 6. A model evaluated that way has learned from the future relative to some test rows.

Compare an invalid shuffled split with a chronological split. The audit prints whether the latest training date reaches or passes the earliest test date.

Hyndman and Athanasopoulos call this time-series cross-validation, or evaluation on a rolling forecasting origin: each test observation occurs after the observations used to make that forecast.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/} You can expand the training history or keep a rolling window. Either way, the time direction stays intact.

Why isn't a random split acceptable just because every row belongs to the same cluster?

Answer

Cluster identity doesn't remove time order. A shuffled split can train on later demand, launch events, and seasonal behavior before evaluating earlier dates. Production only predicts later periods from earlier information.

Establish a same-weekday baseline

Batch-job volume has seasonality when a pattern repeats at a known calendar frequency. Saturday volume stays lower than weekday volume. Copying yesterday into tomorrow treats that weekend lull as the new baseline.

Monday, January 26 actually ran 103 jobs. Sunday was 78. Last Monday was 104.

A seasonal naive forecast reuses the observation from the matching previous season. For daily batch-job data with a weekly pattern, forecast Monday from last Monday, Tuesday from last Tuesday, and so on.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/} Hyndman and Athanasopoulos treat this method as a benchmark: a fancier model has to beat it on later windows before the extra machinery is worth running.

The Friday arithmetic is 160 - 132 = 28. Positive error means the observed count exceeded the forecast. Negative error means the forecast was too high.

Compute the complete holdout week. Each forecast reads the value exactly seven rows earlier.

Saturday shows the same trap in the other direction. A naive one-step forecast would copy Friday's launch (160) into Saturday and miss by -76. The weekly lag copies last Saturday (83) and misses by +1.

Why is last Monday a better forecast for this Monday than yesterday's Sunday count?

Answer

Sunday is a weekend lull. Copying 78 into Monday treats that dip as the new baseline and misses by 25 jobs. Last Monday already lived in the weekday regime, so the error is 1 job.

Measure forecast error in batch jobs

Mean absolute error (MAE) averages the size of each mistake without letting positive and negative errors cancel:^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/}

$$\text{MAE} = \frac{1}{n}\sum_{t=1}^{n}|y_t - \hat{y}_t|$$

Here $n$ is the number of evaluated days, $y_t$ is observed volume on day $t$, and $\hat{y}_t$ is the forecast for that day. For the holdout week, the absolute weekly-lag errors are 1, 3, 2, 1, 28, 1, 1. Their sum is 37, so MAE is 37 / 7 = 5.3 batch jobs.

MAE stays in batch jobs, which operations can read. It also treats a 28-job miss and a 28-job surplus as equal. Underforecasting a capacity spike can cost more than overforecasting by the same amount, so record asymmetric business cost when the decision requires it.

Calculate overall MAE and normal-day MAE separately. The second metric excludes the known launch-event day so the weekday baseline isn't hidden by one planned spike. Then compare the same week against copying yesterday.

The weekly lag's holdout MAE is 5.3 batch jobs. Yesterday's lag is 23.6 on the same week. Hyndman and Athanasopoulos recommend this comparison before you spend complexity: if a candidate can't beat last week's same weekday on later windows, don't ship it.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/}

Backtest more than one future window

A single future week is fragile evidence. Rolling-origin evaluation moves the forecast origin forward and measures several later windows. In this fixture, the first fold evaluates January 19 through January 25. The second evaluates January 26 through February 1.

Run both folds. The seasonal baseline can forecast each day because each fold starts after at least one full week of history.

Backtest by model-serving cluster, service tier, weekday, and launch-event status. A global average can hide one cluster that routinely underforecasts peak demand. The target must also match the decision: submitted job count helps GPU reservation, failed job count helps incident response, and queue-delay count helps operator notification.

Distinguish forecast errors from residuals

Two related quantities often get called residuals. Hyndman and Athanasopoulos separate them on two axes: which rows they use, and how many steps ahead they look.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/}

Friday's +28 is a forecast error: Friday's observed volume wasn't available when the forecast was issued. This weekly-lag baseline has no fitted parameters, so a one-step training residual uses the same copy-last-week rule. What still changes is which rows you score: known history versus a later window. An operations team may casually say "residual alert," but the stored artifact should name the measured quantity.

Calibrate an error band from earlier days

A point forecast such as 132 batch jobs doesn't express uncertainty. A prediction interval pairs a forecast with a lower and upper bound intended to cover a stated proportion of future outcomes. Hyndman and Athanasopoulos treat the interval as part of the forecast, not a chart decoration, and they evaluate it on later data.^{[1]Reference 1Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/}

When one-step errors look roughly Gaussian, their textbook 95% interval is $\hat{y}_{t} \pm 1.96\,\hat{\sigma}$, where $\hat{\sigma}$ is the residual standard deviation. When that shape is doubtful, they fall back to resampling past errors. This lab uses a smaller cousin of that second idea: the 95th percentile of earlier absolute one-day errors, applied as a symmetric band.

Calibrate on January 12 through January 25, before the January 26 holdout starts. Then measure coverage again on the later holdout week. Coverage is the share of days whose absolute error stays inside the band. A band that always covers by growing infinitely wide isn't useful, so keep the width in batch jobs next to the coverage number.

The empirical band is +/-3 batch jobs. The normal-theory cousin is about +/-3.4. Both widths agree that Friday's +28 isn't ordinary noise. Neither width is a production 95% guarantee from fourteen calibration errors. In-sample coverage is 1.000 because the band was chosen on those same errors. Later holdout coverage is 6 / 7 = 0.857 because Friday falls outside. A production system should measure out-of-sample coverage across rolling windows, forecast horizons, model-serving clusters, and important demand slices, and it should keep the interval narrow enough to act on.

Why doesn't the tiny +/-3 band prove that future batch-job counts will land inside it 95% of the time?

Answer

It comes from only fourteen earlier errors. In-sample calibration coverage tells you how the band was chosen; later holdout coverage checks whether it transferred to unseen days. One holdout week still isn't enough. A production interval needs enough history plus out-of-sample coverage checks across rolling windows, forecast horizons, and important slices.

Turn an unusual error into a review artifact

An anomaly is an observation unusual enough to inspect. It isn't proof that a cluster, pipeline, or model failed. Friday's jump could come from a planned model launch, duplicated event ingestion, a large backfill, or genuine demand growth.

Apply the empirical band to the held-out week. The alert stores enough context to reproduce the comparison.

The baseline caught Friday because 160 falls outside [129, 135]. The launch-event flag doesn't erase the event. It changes the first investigation step.

Route alerts with context. A planned model launch still deserves capacity review, but it shouldn't automatically create an incident page.

Log the forecast version, training cutoff, interval policy version, observed value, error, known-event flags, owner, and eventual resolution. That receipt helps operations act now and helps future model reviews learn from the alert.

Evaluate alert usefulness separately

Low MAE doesn't guarantee useful alerts. An alert threshold can still page too often or miss disruptions. Once reviewers label whether each historical alert was actionable, measure alert precision and recall:

Compute both metrics on a five-event resolution fixture. The policy surfaced two useful events, sent one noisy alert, and missed one disruption.

Tune the policy against operational cost. A missed cluster disruption and a noisy review ticket don't have equal consequences. If a denominator is zero, report None: no evidence isn't the same as measured failure. Page only when urgency warrants interruption; otherwise queue a review with the same evidence.

Reproduce a data-quality anomaly

Not every spike represents demand. A duplicated telemetry event can inflate a count before forecasting code sees it. Reproduce that failure with three raw telemetry rows, two of which describe the same batch job event.

An anomaly review should inspect input quality before retraining a model. Retraining on duplicated counts would teach the model to copy a pipeline bug. This tiny dictionary deduper keeps one row because the repeated payloads match. If two rows reuse an event ID but disagree, quarantine the conflict instead of silently choosing one.

Why shouldn't the alert policy declare an incident as soon as a count crosses the error band?

Answer

The forecast error proves that the observation is unusual relative to the baseline. It doesn't identify the cause. Known launch events, duplicated ingestion, and genuine demand shifts require different responses.

Gate and publish a forecast candidate

A minimal release gate should prove time order, calibration order, normal-day accuracy, interval bookkeeping, contextualized alerts, and assigned ownership. These checks don't prove the baseline is production-ready, but they prevent avoidable mistakes from reaching a capacity workflow.

Run the gate.

Publish a receipt. The hash lets a later monitoring job identify the exact baseline, interval, and input policies it evaluates.

The receipt keeps calibration evidence separate from later holdout evidence and binds the policies that produced both. Passing these checks earns shadow evaluation, not a production rollout. The hash is what a later monitoring job should look up when it asks whether live inputs, delayed labels, or a newer candidate should replace this baseline.

When the policy breaks