Final published version
Licence: CC BY: Creative Commons Attribution 4.0 International License
Research output: Contribution to Journal/Magazine › Journal article › peer-review
Research output: Contribution to Journal/Magazine › Journal article › peer-review
}
TY - JOUR
T1 - Online Detection of Forecast Model Inadequacies Using Forecast Errors
AU - Grundy, Thomas
AU - Killick, Rebecca
AU - Svetunkov, Ivan
PY - 2025/6/11
Y1 - 2025/6/11
N2 - In many organizations, accurate forecasts are essential for making informed decisions in a variety of applications, from inventory management to staffing optimization. Whatever forecasting model is used, changes in the underlying process can lead to inaccurate forecasts, which will be damaging to decision‐making. At the same time, models are becoming increasingly complex, and identifying change through direct modeling is problematic. We present a novel framework for online monitoring of forecasts to ensure they remain accurate. By utilizing sequential changepoint techniques on the forecast errors, our framework allows for the real‐time identification of potential changes in the process caused by various external factors. We show theoretically that some common changes in the underlying process will manifest in the forecast errors and can be identified faster by identifying shifts in the forecast errors than within the original modeling framework. Moreover, we demonstrate the effectiveness of this framework on numerous forecasting approaches through simulations and show its effectiveness over alternative approaches. Finally, we present two concrete examples, one from Royal Mail parcel delivery volumes and one from NHS A&E admissions relating to gallstones.
AB - In many organizations, accurate forecasts are essential for making informed decisions in a variety of applications, from inventory management to staffing optimization. Whatever forecasting model is used, changes in the underlying process can lead to inaccurate forecasts, which will be damaging to decision‐making. At the same time, models are becoming increasingly complex, and identifying change through direct modeling is problematic. We present a novel framework for online monitoring of forecasts to ensure they remain accurate. By utilizing sequential changepoint techniques on the forecast errors, our framework allows for the real‐time identification of potential changes in the process caused by various external factors. We show theoretically that some common changes in the underlying process will manifest in the forecast errors and can be identified faster by identifying shifts in the forecast errors than within the original modeling framework. Moreover, we demonstrate the effectiveness of this framework on numerous forecasting approaches through simulations and show its effectiveness over alternative approaches. Finally, we present two concrete examples, one from Royal Mail parcel delivery volumes and one from NHS A&E admissions relating to gallstones.
KW - concept drift
KW - structural break
KW - sequential change
KW - changepoint
U2 - 10.1111/jtsa.12843
DO - 10.1111/jtsa.12843
M3 - Journal article
JO - Journal of Time Series Analysis
JF - Journal of Time Series Analysis
SN - 0143-9782
ER -