Hydrology Professional Development Series (PDS) 2 : Develop, Implement and Maintain Models and Tools

PCU 3: Evaluate the hydrological forecasts and services (Ongoing assessment of model performance, forecast validation & verification)

Producers:
Dave Streubel - Alaska-Pacific River Forecast Center, Anchorage, AK
Rick Koehler - Forecast Decision Training Branch, Boulder, CO
Laura Diamond - North Central RFC, Chanhassen, MN

Description of Job Competency to be Achieved :
Performing a periodic comparison between the forecasts and the conditions which occurred and reviewing forecast justification will enable forecasters to continuously improve the services they provide to our customers and partners.

Description of Need :
Accuracy and skill are two performance measures for any hydrologic forecast. To become a better forecaster, it is not enough to simply know that a forecast did not verify based on a specific standard. A forecaster must determine what went into a forecast and identify methods for improvement. This is done through forecaster development that allow assessment of the forecast relative to the inputs and operational environment under which the forecasts were issued and communicated.

In other words, we need to evaluate all aspects of the NWS hydrologic forecast process in order to improve services to the public and other partners.

Abilities/Performance Elements:

• Perform a periodic/seasonal review
• Review relevant statistics
• Identify forecast/verification errors
• Review forecast justification
• Create hypothesis to explain why forecast errors occurred
• Modify the forecasting steps that were involved in the forecast error
• Develop potential service assessment skills
• Identify relevant performance standards

Ability 1. Perform a periodic/seasonal review with input from user community. Skill 1.1. Review hydrologic forecasts beyond a routine verification calculation. Examine additional factors from a periodic/seasonal review to identify possible hypothesis and actionable items to include in the forecast process. IC Title: Gridded Forecast Verification and Bias Correction Present Type: Formal Training (in LMS) URL/location: https://doc.csod.com Description: To become a better forecaster, it is not enough to simply know that a forecast did not verify. One must determine what happened and identify methods for improvement through forecast verification. In this module, you will be given an example forecast and verifying analysis. Then you will assess what corrective actions are needed based on the mismatches that occurred. Length: 1 hour Vetted: Yes Skill 1.2. Interact with user community to solicit input from past season regarding product quality and usefulness. Other Hydro PDS cross referenced abilities and skills PDS 1, PCU 1 - Provide Hydrologic Services Ability 2 - Build and maintain positive relationships with our partners and users Skill 2.2 - Identify opportunities to enhance and improve NWS hydrologic services Ability 4 - Collaborate with NWS peers for effective forecast and warning services Skill 4.1 - Coordinate hydrologic events, programs, and non-operational activities Ability 5 - Collaborate with partners and users for effective planning, mitigation, forecast and warning services Skill 5.1 - Coordinate hydrologic programs and activities through establishing operational thresholds and event procedures PDS 1, PCU 2 - Monitor and assess the hydrologic and hydrometeorological environment to identify specific threat areas Ability 4 - Monitor data and conditions throughout the operational period to continuously re-evaluate products and services Skill 4.1 - Assess and maintain situational awareness of current and forecast river conditions PDS 1, PCU 4 - Provide hydrologic and hydrometeorological products and services Ability 5 - Prepare and provide hydrologic forecast verification information Skill 5.1 -Utilize available internal and external tools to provide and overview of forecast skill Skill 5.2 - Assess sources of forecast skill and error Skill 5.3 - Follow-up with stakeholders following an event to explain strengths and weaknesses of forecast services provided during the event.

Ability 2. Review relevant statistics for evaluating forecast accuracy and correct probabilistic values. Skill 2.1. Describe statistical metrics used in forecast verification and their respective shortfalls. IC Title: Introduction to Verification of Hydrologic Forecasts Present Type: Formal Training (in LMS) URL/location: https://doc.csod.com Description: This module offers a comprehensive description of a set of common verification measures for hydrologic forecasts, both deterministic and probabilistic. In addition to providing a measure of how well a forecast matches observations, verification measures can be used to help forecasters and users learn about the strengths and weaknesses of a forecast. Length: 2 hours Vetted: Yes IC Title: Techniques in Hydrologic Forecast Verification Present Type: Formal Training (in LMS) URL/location: https://doc.csod.com Description: This module demonstrates techniques for developing a hydrologic forecast Length: 2 hours Vetted: Yes

Ability 3. Identify elements present in forecast errors. Derive verification data relevant to the forecast event Skill 3.1. Compare the forecast to the verification data to identify differences present in the forecast. Skill 3.2. Choose verification statistic in context with performance question being evaluated. Correctly identify differences between forecast and verification metrics as compared to observed values. IC Title: Gridded Forecast Verification and Bias Correction Present Type: Formal Training (in LMS) URL/location: https://doc.csod.com Description: To become a better forecaster, it is not enough to simply know that a forecast did not verify. One must determine what happened and identify methods for improvement through forecast verification. In this module, you will be given an example forecast and verifying analysis. Then you will assess what corrective actions are needed based on the differences that occurred. Length: 1 hour Vetted: Yes Possible training gap - will be considered for further development.

Ability 4. Review forecast justification Analyze the parts of forecast reasoning related to the performance statistic of the differences. Review forecast procedures for problematic items. Skill 4.1. Review forecast justification and analyze the parts of forecaster reasoning related to the difference. Possible training gap - will be considered for further development. Skill 4.2. Critically review the forecast process and examine the logic used in the forecast process that may lead to differences. IC Title: Gridded Forecast Verification and Bias Correction Present Type: Formal Training (in LMS) URL/location: https://doc.csod.com Description: To become a better forecaster, it is not enough to simply know that a forecast did not verify. One must determine what happened and identify methods for improvement through forecast verification. In this module, you will be given an example forecast and verifying analysis. Then you will assess what corrective actions are needed based on the differences that occurred. Length: 1 hour Vetted: Yes Possible training gap - will be considered for further development.

Ability 5. Determine the model contribution to the forecast/verification difference and how it influenced forecaster reasoning. Skill 5.1. Generate plausible scenarios and reasons to explain why the forecast/verification difference occurred. IC Title: Gridded Forecast Verification and Bias Correction Present Type: Formal Training (in LMS) URL/location: https://doc.csod.com Description: To become a better forecaster, it is not enough to simply know that a forecast did not verify. One must determine what happened and identify methods for improvement through forecast verification. In this module, you will be given an example forecast and verifying analysis. Then you will assess what corrective actions are needed based on the mismatches that occurred. Length: 1 hour Vetted: Yes Possible training gap - will be considered for further development.

Ability 6. Modify the forecasting steps that were involved in the forecast error Use the plausible scenarios and reasons generated earlier to modify: modeling parameters, interpretation of model output, or steps in the forecast process. Skill 6.1. Identify steps to use in similar future forecast situations and monitor how well the forecast matches observed conditions. Possible training gap - will be considered for further development. Skill 6.2. Evaluate the steps needed to produce a forecast and analyze the process to locate where potential problems could occur. Propose modifications to address the issues identified. IC Title: Gridded Forecast Verification and Bias Correction Present Type: Formal Training (in LMS) URL/location: https://doc.csod.com Description: To become a better forecaster, it is not enough to simply know that a forecast did not verify. One must determine what happened and identify methods for improvement through forecast verification. In this module, you will be given an example forecast and verifying analysis. Then you will assess what corrective actions are needed based on the mismatches that occurred. Length: 1 hour Vetted: Yes Possible training gap - will be considered for further development.

Ability 7. Develop potential service assessment skills Identify the evaluation skills from hydrologic forecast reviews for use in service assessments. Skill 7.1. Review Model Calibration Reference PDS 2, PCU 1, Ability 3 (Conduct Calibration) Skill 7.2. Identify elements used in Service Assessments IC Title: National Service Assessment Team Training Type: Formal Training (in LMS) URL/location: https://doc.csod.com Description: This module describes the activities and responsibilities of a National Service Assessment Team. The training covers the following topics: (1) what should you expect as a team member, (2) preliminary activities, (3) duties at the scene of the event, (4) responsibilities after the fieldwork is completed and (5) tips on writing an assessment report. Length: 0.5 hour Vetted: Yes Possible training gap - will be considered for further development.

Ability 8. Identify relevant performance standards for hydrologic forecasts Skill 8.1. Identify office and agency standards applicable to forecast verification. Coordinate with program office to develop reference materials Possible training gap - will be considered for further development.