Performance Measures: A Beginning Rather than an End
to Assessment of Warning and
Forecast Operations
Preston W. Leftwich, Jr.
Scientific Services Division
Performance measures are routinely computed by
the National Weather Service (NWS) as an aid in evaluation of its warnings and
forecasts. Ultimately, our goal is to make them a meaningful part of our
efforts in forecaster performance improvement.
With this in mind, performance measures can be a beginning, rather than an end, of steps to improve warning and forecast performance. That is, performance measures can be used to identify periods, such as seasons or meteorological episodes, for which warnings or forecasts were particularly good or particularly bad. Then, focus can be placed on associated meteorology and warning or forecast decisions.
As an example, a performance measure for Probability of Precipitation (PoP) forecasts is “Percent Improvement in Brier Score for Cases when the Difference between Model Output Statistics (MOS) Guidance and Local PoP Forecasts is At Least 20%.” Annual and seasonal values can be used to isolate periods when values of the metric indicated better or worse performance for local forecasts as compared to MOS guidance. Percent improvement in Brier Score is best applied over a period rather than for an individual forecast. However, review of forecaster decisions made for individual forecasts comprising a selected sample provides meaningful insight into the forecast process. For example, forecasts for which MOS guidance and local PoP values are on the “opposite side of 50%” identify forecasts for which a forecaster made a definite decision to differ from guidance. Associated meteorology, forecast guidance and forecaster reasoning are reviewed for these cases. Such assessment can suggest steps (e.g., additional training) likely to lead to forecast improvement.
Examples are discussed
for PoP forecasts. Additionally, ways
to implement similar procedures for warnings and other forecasts are
discussed.