A disadvantage of traditional point-based verification is that it gives no credit to “near misses”. That is, forecasts that are close are considered to be just as wrong as forecasts that are quite far away from the observations. If the observation has some uncertainty associated with it, a “near miss” has some probability of actually being a “hit”. Using the same logic, a “hit” in the traditional view has some probability of being a “false alarm” when the observation is uncertain. Fuzzy verification attempts to take into account the uncertainties in the observations and/or the forecasts, to give partial credit to forecast success and failure instead of only 0’s and 1’s.
The underlying assumption of fuzzy verification is that an observation can be represented by a PDF, where the width of the PDF is a measure of uncertainty in the observation. Further, assume that the forecast can also be represented as a PDF, for example, obtained from an ensemble prediction system. If the forecast is good then the PDF of the forecast will overlap with the PDF of the observation. When computing categorical statistics in the traditional way, the observation and forecast are assigned to one or the other side of the yes/no threshold. Using fuzzy verification, when the threshold value intercepts the PDF then there is some likelihood of the forecast or observation falling on either side of the threshold. The entry in the contingency table, instead of a 1 in one quadrant and a 0 in the other three quadrants, would be a number between 0 and 1 in each of the four quadrants.
Another standard score that lends itself very well to this framework is
the Brier score. Unlike the traditional Brier score that uses only values of 0
or 1 to represent the probability of an observation exceeding a threshold, the
fuzzy version uses the PDF of the observation to specify that probability.
Fuzzy verification can also be used to weight the forecast errors
according to the uncertainty associated with the observation. When doing
spatial verification over a large domain, some regions are better observed than
others. For example, rain gauge distributions are generally better in heavily
populated areas than in less populated areas. The weight is a function of the
linear error in probability space, using the PDFs of the observation and
forecast to define the probability space for that forecast/observation pair.
When measuring the accuracy of a forecast, or especially when intercomparing
two or more forecast systems, this technique gives greater weight to the
estimated forecast errors in which there is greater confidence (i.e. in which
the observation error is smaller).
Fuzzy verification can be used even if only the observations have uncertainty, or only the forecast has uncertainty, that is, it is not necessary to have two PDFs. The most difficult aspect of this approach will be getting good estimates of the PDFs. The observation PDF could be estimated from (a) expected measurement (instrument) errors, (b) variances used in data assimilation, (c) kriging or some other objective analysis scheme, or (d) near neighbours in time or space. The forecast PDF could be estimated from (a) ensemble members, (b) statistical relationships derived from historical data, or (c) near neighbours in time or space.