Alternatives to the Chi-Square Test for Evaluating Rank Histograms from Ensemble Forecasts

• Published in: Weather and forecasting Vol. 20; no. 5; pp. 789 - 795
• Main Author: Elmore, Kimberly L.
• Format: Journal Article
• Language: English
• Published: Boston, MA American Meteorological Society 01.10.2005
• Subjects: Climate; Earth, ocean, space; Exact sciences and technology; External geophysics; Histograms; Instruments; Meteorology; Weather analysis and prediction; Weather forecasting; Performance evaluation; Rank statistic; Ensemble simulation; Weather forecast; histograms; algorithm performance; Goodness of fit test; Forecast skill; chi-square test; Numerical forecast; Comparative study; Cramer von Mises test
• ISSN: 0882-8156; 1520-0434; 1520-0434
• DOI: 10.1175/WAF884.1

Rank histograms are a commonly used tool for evaluating an ensemble forecasting system’s performance. Because the sample size is finite, the rank histogram is subject to statistical fluctuations, so a goodness-of-fit (GOF) test is employed to determine if the rank histogram is uniform to within some statistical certainty. Most often, the χ2 test is used to test whether the rank histogram is indistinguishable from a discrete uniform distribution. However, the χ2 test is insensitive to order and so suffers from troubling deficiencies that may render it unsuitable for rank histogram evaluation. As shown by examples in this paper, more powerful tests, suitable for small sample sizes, and very sensitive to the particular deficiencies that appear in rank histograms are available from the order-dependent Cramér–von Mises family of statistics, in particular, the Watson and Anderson–Darling statistics.