Use of Confidence Intervals for Verification Statistics

Tressa L. Fowler and Barbara G. Brown

National Center for Atmospheric Research

Frequently, the goal of a forecast verification study is to compare the quality of different forecasts or algorithms. Point estimates of forecast quality are inadequate for determining if one type of forecast or algorithm performs significantly better than another or whether forecasts have been improved. Confidence intervals provide an effective way to make these comparisons. Additionally, confidence intervals explicitly communicate the uncertainty associated with a measure of forecast quality in a way that point estimates cannot. However, these methods have been infrequently applied in forecast verification studies, due to the characteristics of the verification data and measures.

Classical statistical models, conditional models, and computer resampling methods can all be used to estimate confidence intervals for verification statistics. The assumptions, performance, and relative merits of each type of interval are presented.