LearnProduction ML SystemsForecasting and Anomaly Detection

📊MediumEvaluation & Benchmarks

Forecasting and Anomaly Detection

Forecast parcel demand with time-aware evaluation and turn large residuals into reviewable operational alerts.

7 min read

Learning path

Step 43 of 151 in the full curriculum

Ranking and Recommendation Systems Monitoring Predictive Models

A ranking model selects which products appear now. Warehouse planners face a future question: how many parcels will leave a fulfillment center tomorrow, and which observed counts signal an unexpected disruption?

Forecasting differs from ordinary random-split prediction because observations are ordered. Tomorrow isn't exchangeable with last month. A useful evaluation must train on the past and predict a later window, repeatedly if possible.

Warehouse volume forecast timeline showing past daily parcel counts, a weekly seasonal baseline, future forecast interval, and an observed spike sent to operations review. — A forecast uses only earlier counts to predict later capacity; anomaly alerts compare new observations with an expected range rather than with intuition.

Establish a Seasonal Baseline

Suppose Saturday parcel volume is normally lower than weekday volume. Predicting tomorrow using yesterday alone will overreact each Friday and Monday. A small baseline can predict each weekday from the same weekday last week.

Day	Last week parcels	This week parcels	Weekly-lag error
Monday	100	104	4
Tuesday	112	110	-2
Wednesday	115	119	4
Thursday	118	116	-2
Friday	132	160	28

The Friday miss might be an anomaly, or it might be a promotion planned outside the dataset. A monitoring system shouldn't announce a warehouse incident from one residual alone; it should surface the deviation with context and a review path.

Hyndman and Athanasopoulos emphasize that forecast accuracy must be evaluated on genuinely future observations, with time-series cross-validation expanding or rolling the training origin forward.^[1] That rule prevents tomorrow's demand from influencing the model that claims it could have predicted tomorrow.

Diagram showing Daily parcel counts ordered history, Seasonal baseline same weekday, Future forecast expected range, and Observed volume. — Daily parcel counts ordered history, Seasonal baseline same weekday, Future forecast expected range, and Observed volume.

Calculate a Forecast and Alert Threshold

Mean absolute error (MAE) averages absolute forecast mistakes:

\text{MAE} = \frac{1}{n}\sum_{t=1}^{n}|y_t - \hat{y}_t|

Here $y_t$ is observed volume and $\hat{y}_t$ is its forecast. MAE remains in parcels, which makes it understandable to operations.

weekly-lag-forecast-alert.py

history = [100, 112, 115, 118, 132, 82, 76]
observed = [104, 110, 119, 116, 160, 84, 78]

forecasts = history[:]  # same weekday last week
errors = [actual - forecast for actual, forecast in zip(observed, forecasts)]
mae = sum(abs(error) for error in errors) / len(errors)
alert_limit = 20
alerts = [
    (day, actual, forecast, error)
    for day, (actual, forecast, error) in enumerate(zip(observed, forecasts, errors), start=1)
    if abs(error) >= alert_limit
]

print("MAE parcels:", round(mae, 1))
for day, actual, forecast, error in alerts:
    print(f"alert day={day} observed={actual} forecast={forecast} residual={error:+d}")

Output

MAE parcels: 6.3
alert day=5 observed=160 forecast=132 residual=+28

The example gives a baseline and one reviewable alert. In a real system, an alert threshold should reflect historical residual variation and capacity cost. A prediction interval or empirically measured quantile of prior residuals is more defensible than picking 20 without evidence.

Split by Time, Then Backtest

A single future week is fragile evidence. Run several rolling-origin evaluations:

Training history	Validation window	Question
January	first week of February	works after initial launch?
January through February	first week of March	adapts after more data?
January through March	first week of April	survives seasonal shift?

Record MAE by fulfillment center, service tier, weekday, and promotion status. A national average can hide a warehouse that routinely underforecasts peak demand. For inventory decisions, also consider asymmetry: underpredicting demand may cost more than reserving excess capacity.

The target must match the decision. Forecasting shipped parcel count helps staffing; forecasting late parcels helps customer notification; detecting an unusual scan-drop rate helps incident response. Don't combine those labels into one ambiguous "operations score."

Anomalies Need Explanation and Ownership

An anomaly is a residual large enough to justify inspection, not proof that the model or warehouse failed. The Friday spike may correspond to a campaign, a bulk seller import, duplicated event ingestion, or genuine demand growth.

Log an alert artifact:

Field	Purpose
series and warehouse	route investigation
forecast version and training cutoff	reproduce expectation
observed value and residual	quantify deviation
interval or threshold version	explain alert policy
known promotion flag	reduce false alarms
owner and resolution	teach future models and policies

A model that silently swallows anomalies is hard to operate. A model that pages on every seasonal pattern is equally unhelpful. Forecast evaluation and alert evaluation are related but separate: one measures prediction error, the other measures whether the alert led to useful action.

Mastery Check

Evaluation rubric

Evidence	A production-ready answer demonstrates
temporal evaluation	uses rolling or future-window backtests rather than shuffled rows
forecast use	maps prediction intervals and error costs to inventory or capacity decisions
alert policy	detects actionable residual anomalies with escalation and review controls

Common Pitfalls

Symptom	Cause	Fix
Test results collapse at launch	random split leaked future behavior	use rolling-origin validation
Alerts fire every Monday	weekly seasonality absent	add seasonal baseline first
Operations ignores alerts	no ownership or context	log threshold, residual, and resolution

Next Step

Continue to Monitoring Predictive Models

You can now measure forecasts and surface surprising observations. Next you will monitor deployed models for input shift, delayed outcome degradation, and retraining decisions.

PreviousRanking and Recommendation Systems

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Forecasting: Principles and Practice, Third Edition.

Hyndman, R. J. & Athanasopoulos, G. · 2021

Back to Topics

LearnProduction ML SystemsForecasting and Anomaly Detection

📊MediumEvaluation & Benchmarks

Forecasting and Anomaly Detection

Forecast parcel demand with time-aware evaluation and turn large residuals into reviewable operational alerts.

7 min read

Learning path

Step 43 of 151 in the full curriculum

Ranking and Recommendation Systems Monitoring Predictive Models

Establish a Seasonal Baseline

Day	Last week parcels	This week parcels	Weekly-lag error
Monday	100	104	4
Tuesday	112	110	-2
Wednesday	115	119	4
Thursday	118	116	-2
Friday	132	160	28

Calculate a Forecast and Alert Threshold

Mean absolute error (MAE) averages absolute forecast mistakes:

\text{MAE} = \frac{1}{n}\sum_{t=1}^{n}|y_t - \hat{y}_t|

Here $y_t$ is observed volume and $\hat{y}_t$ is its forecast. MAE remains in parcels, which makes it understandable to operations.

weekly-lag-forecast-alert.py

history = [100, 112, 115, 118, 132, 82, 76]
observed = [104, 110, 119, 116, 160, 84, 78]

forecasts = history[:]  # same weekday last week
errors = [actual - forecast for actual, forecast in zip(observed, forecasts)]
mae = sum(abs(error) for error in errors) / len(errors)
alert_limit = 20
alerts = [
    (day, actual, forecast, error)
    for day, (actual, forecast, error) in enumerate(zip(observed, forecasts, errors), start=1)
    if abs(error) >= alert_limit
]

print("MAE parcels:", round(mae, 1))
for day, actual, forecast, error in alerts:
    print(f"alert day={day} observed={actual} forecast={forecast} residual={error:+d}")

Output

MAE parcels: 6.3
alert day=5 observed=160 forecast=132 residual=+28

Split by Time, Then Backtest

A single future week is fragile evidence. Run several rolling-origin evaluations:

Training history	Validation window	Question
January	first week of February	works after initial launch?
January through February	first week of March	adapts after more data?
January through March	first week of April	survives seasonal shift?

Anomalies Need Explanation and Ownership

Log an alert artifact:

Field	Purpose
series and warehouse	route investigation
forecast version and training cutoff	reproduce expectation
observed value and residual	quantify deviation
interval or threshold version	explain alert policy
known promotion flag	reduce false alarms
owner and resolution	teach future models and policies

Mastery Check

Evaluation rubric

Evidence	A production-ready answer demonstrates
temporal evaluation	uses rolling or future-window backtests rather than shuffled rows
forecast use	maps prediction intervals and error costs to inventory or capacity decisions
alert policy	detects actionable residual anomalies with escalation and review controls

Common Pitfalls

Symptom	Cause	Fix
Test results collapse at launch	random split leaked future behavior	use rolling-origin validation
Alerts fire every Monday	weekly seasonality absent	add seasonal baseline first
Operations ignores alerts	no ownership or context	log threshold, residual, and resolution

Next Step

Continue to Monitoring Predictive Models

You can now measure forecasts and surface surprising observations. Next you will monitor deployed models for input shift, delayed outcome degradation, and retraining decisions.

PreviousRanking and Recommendation Systems

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Forecasting: Principles and Practice, Third Edition.

Hyndman, R. J. & Athanasopoulos, G. · 2021

Forecasting and Anomaly Detection

Establish a Seasonal Baseline

Calculate a Forecast and Alert Threshold

Split by Time, Then Backtest

Anomalies Need Explanation and Ownership

Mastery Check

Why can't a forecasting dataset use a random train/test split by day?

What does a large positive residual mean in the parcel-volume example?

Why evaluate an anomaly policy separately from MAE?

Evaluation rubric

Common Pitfalls

Forecasting and Anomaly Detection

Establish a Seasonal Baseline

Calculate a Forecast and Alert Threshold

Split by Time, Then Backtest

Anomalies Need Explanation and Ownership

Mastery Check

Why can't a forecasting dataset use a random train/test split by day?

What does a large positive residual mean in the parcel-volume example?

Why evaluate an anomaly policy separately from MAE?

Evaluation rubric

Common Pitfalls