Spatial Forecast Verification Methods Inter-Comparison Project

Subjective Evaluation Results

At the 20 February 2007 planning meeting, we looked at the nine initial test cases and performed a gut reaction subjective evaluation. The three versions of WRF were all shown together with the stage 2 analysis with the 24-hour forecasts randomly assigned A, B and C so that the evaluators did not know which forecast was for which model. The cases were shown for one minute each, and then we ran through the process a second time to account for burn-in. The results are given in the following two files: pdf, Excel

Agenda

printable pdf

Verification Method Intercomparison Meeting - Tuesday, February 20, 2007
    Session 1: Participant presentations of methods (Rm. FL2-1001)
8:30
8:40
9:00
9:30 		Welcome and introductions
Overall plan for ICP and meeting goals (Eric Gilleland) (pdf)
Subjective verification of cases (Eric Gilleland and Dave Ahijevych) (ppt)
Traditional statistics - results for two focus cases (John Halley Gotway) (ppt)
    Session 2: Participant presentations of methods (Rm. FL2-1001)
9:40
9:50
10:00

10:10

10:20
10:30
10:40
10:50
11:00
11:10
11:20
11:30
11:40
11:50



Bias-adjusted CSI (Fedor Mesinger) (ppt)
MODE (Dave Ahijevych) (pdf)
Baldwin Object-based approach (Mike Baldwin) (pdf)

Break

Cell Identification/Procrustes Shape Analysis (Steven Lack) (ppt)
Cluster analysis (Eric Gilleland for Caren Marzban and Scott Sandgathe) (pdf)
Intensity-Scale (Barbara Casati) (ppt)
Fuzzy Neighborhood Methods (Beth Ebert) (ppt)
Wernli and Keil presentations (Beth Ebert) (Wernli ppt, Keil pdf)
Version of Nigel Roberts FSS (Beth Ebert for Marion Mittermeier) (ppt)
Distributional method (Beth Ebert for Chiara Marsigli) (ppt)
Forecast Quality Index (Sukanta Basu) (ppt)
Composite Method (Jason Nachamkin) (ppt)
(modified) Ebert-McBride CRA (Beth Ebert) (ppt)

Lunch

    Session 3: Discussion and introduction to breakout session (Rm. FL2-1001)
1:00 Chris Davis and Barb Brown
    Session 4: Breakout groups (Rooms FL2-1001, FL2-1002, FL0-2512)
1:45 Randomly break into three groups
    Primary goal: Examine all of the methods as applied to the two example cases. Complete a table considering the characteristics of the methods and their differences. Consider additional attributes as they are discussed and add to the table.

Secondary goal: Consider plans for future efforts and meetings (idealized cases, additional cases, outstanding issues, etc.)

Breakout group leaders: Barbara Brown (FL2-1001), Chris Davis (FL0-2512) and Beth Ebert (FL2-1002)
3:30 Break
    Session 5: Discussion and future plans (Rm. FL2-1001)
4:00
4:30
5:30		 Discussion of results from breakouts
Discussion of future plans (additional cases, idealized cases, questions to be considered, sharing software, special issue of Weather and Forecasting)
End!
After our 20 February 2007 planning meeting, there was a semi-related workshop on forecast verification methods at which Beth Ebert gave an overview talk of the various spatial forecast verification methods presented above (pdf). Eric Gilleland also gave a talk about the meeting with a brief summary of the discussions (pdf, see above for results of subjective evaluations).

*Any information collected is used solely to determine the legitimacy of subscription requests (e.g., to protect against spam). Email addresses are added to a controlled list (only subscribers may send messages). The list is intended primarily for participants in the ICP, but others wishing to keep updated on the progress of the project may also subscribe. If you have any trouble subscribing (or unsubscribing) contact the webmaster (Eric Gilleland).