Equitable Threat Score Applied to MCS Rainfall: Scale Dependence, Disagreement with Subjective Evaluations and Impact of Phase-Shifting

William A. Gallus, Jr., Isidora Jankov, and Steven Aves

Iowa State University

Evaluation of 10 km Eta model forecasts of MCS rainfall in 23 warm season events has offered insight into several problems with the use of equitable threat score (ETS). First, ETS has been found to increase as the verification of rainfall forecasts is performed on a coarser grid. This agrees with the findings of Tustison et al. that representativeness error will worsen for finer resolutions if point-to-area conversion is done (gauge data is averaged to the model's grid). Surprisingly, the increase in ETS not only occurs when the 10 km model output is averaged to and verified on a 30 km grid, but also occurs when the model is re-run with 30 km grid spacing. In fact, ETSs for the 30 km output are higher than those for the 10 km output averaged to 30 km, particularly when the Betts-Miller-Janjic (BMJ) convective scheme is used, as opposed to the Kain-Fritsch (KF). Second, it was found that a subjective evaluation examining shape and location of rainfall area, timing of initiation, and over prediction or under prediction of average rainfall amount disagrees with the trends noted in ETS. Whereas the BMJ scheme earns higher ETSs than the KF, even after adjusting for different Biases, the KF seems to be the better scheme in the subjective evaluation. Its location and shape of rainfall area better resembled observations, although it had a more pronounced problem with late initiation (likely related to our use of NCEP Eta output for initialization -- the BMJ scheme is used in NCEP's Eta assimilation system).

Finally, results of the use of a phase-shifted ETS applied to these MCS cases will be presented. This technique was originally developed by Kalnay and Miller and permits the forecasted rainfall to be shifted in one direction up to 8 model grid points to maximize ETS. The technique attempts to remove some of the penalty that might be present in ETS due to small spatial shifts in the forecasted rainfall fields compared to observations. For the MCS events, the phase-shifted technique tends to accentuate the difference between the BMJ and KF schemes, with the BMJ runs having a larger improvement in ETS after the spatial adjustment. The average shift was also found to be larger for the BMJ runs. On average, the phase shift was non-zero in the east-west direction but negligible in the north-south direction.