Evaluating Quality, Relevance, and Performance in our Program Office

Supplemental materials and self-guided content for our inaugural Program Review.

On this page you’ll find materials about the Weather Program Office’s structure, functions and activities that are intended to assist our reviewers in evaluating our quality, relevance and performance as a Program Office.

The National Oceanic and Atmospheric Administration (NOAA), Office of Oceanic and Atmospheric Research (OAR) conducts science reviews every five years to evaluate the quality, relevance, and performance of the research and activities its laboratories and programs conduct and support. These reviews are also intended to ensure that OAR research is linked to NOAA’s mission and priorities, is of high quality as judged by preeminence criteria, and is carried out with a high level of performance.

The purpose of the Weather Program Office (WPO) review, which was held virtually January 24–27, 2023, is to provide an independent, external assessment of the program, including its management structures, and to recommend strategic directions in key areas for WPO and OAR.

The review assesses the research and other science activities WPO funds, how WPO decides what types of research and science to support, and how WPO engages with the public and peer communities to communicate about its work.

This external review covers the programmatic activities and management of the Weather Program Office over the past five years (2017–2022) and is the first program office review for WPO. The programmatic themes/activity areas and related topics for the WPO review are below.

Activity Area 1.

Organizational Excellence

Activity Area 2.

Weather Research Models, Observations and Forecasting Tools

Activity Area 3.

Advancement and Transition of Weather Research

Activity Area 4.

Effective Communication and Coordination of Weather Research

WPO Response to Panel Review Recommendations

WPO strives to find, fund and foster innovative atmospheric research. As a result of the recommendations from the expert panel during WPO’s Program Review in January 2023, WPO is implementing new strategies to expand and achieve more efficient and effective operations and management of its research portfolio. In the following document, WPO outlines the panel’s recommendations, their alignment with WPO’s Strategic Plan, and the actions WPO is taking to address any areas for improvement. WPO appreciates the opportunity to reflect and gain insight from such an esteemed panel of experts, and is looking forward to integrating their suggestions into daily operations.

The Weather Program Office Response to the Panel Review’s Recommendations can be found here.

Evaluation Report from the Review Panel

In 2023, the Weather Program Office (WPO) completed its first Program Review, covering the years of 2017–22. The Review Panel gave WPO an overall rating of “Exceeds Expectations.’‘ Throughout the assessment, reviewers emphasized the exceptional quality of work by WPO.

The panel concluded all of WPO’s programs were in alignment with NOAA’s mission and the office’s relevance is indisputable. Reflecting on the breadth of WPO’s work, the Review Panel commented: “WPO is a necessary and needed organization, particularly as convergent research comes to the forefront and societal impacts from weather and climate continue to rise.” The office’s work not only saves lives but also substantially enhances the economy as a result of the products produced, technical advancements, and improved communications. By integrating social and physical sciences within a unified framework, WPO produces actionable solutions for society.

You can read the full Evaluation report from the reviewers here.

Charge to Reviewers

Details for Evaluation & Recommendations

NOAA guidance asks reviewers to consider the Quality, Relevance, and Performance of WPO and provide an overall rating for each activity area reviewed. For each area, each reviewer will provide one of the following overall ratings:

Highest Performance: WPO greatly exceeds the satisfactory level and is outstanding in almost all areas.
Exceeds Expectations: WPO goes beyond the satisfactory level and is outstanding in many areas.
Satisfactory: WPO meets expectations and the criteria for a satisfactory rating.
Needs Improvement: In general, WPO does not reach expectations and does not meet the criteria for a satisfactory rating. The reviewer will identify specific problem areas that need to be addressed.

Download Charge to Reviewers

Evaluation Criteria
Quality: Evaluate the quality of WPO’s research and development (R&D) portfolio. This refers to the merit of R&D that is funded by WPO and the resulting communication, outputs, and knowledge shared with the scientific community.
Relevance: Evaluate the degree to which WPO funded research and development (R&D) is relevant to NOAA’s mission and of value to the Nation. This primarily refers to the value of R&D to users beyond the scientific community. Relevance includes not only hypothetical value, but actual impact. Examples of ways the impact of R&D can be realized include the application of scientific knowledge to policy decisions, the improvement of operational capabilities at NOAA’s service lines and other collaborating institutions, or licensing of inventions for commercial use.
Performance: Assessing performance may include considerations of technical execution, finances, workforce, infrastructure, and leadership necessary to achieve WPO’s goals. This involves understanding the quality of management, including interaction with stakeholders, clear articulation of strategic direction, as well as the balance of WPO’s portfolio across time frames and intended applications.

A Diversity of Expertise

Our Review Panel

Program Review Agenda

Day 1: Organizational Excellence

Day 2: Weather Research Models, Observations and Forecasting Tools

Day 3: Advancement and Transition of Weather Research

Day 4: Effective Communication and Coordination of Weather Research

January 24th

Activity Area 1: Organizational Excellence

Download Day 1 Slides

Time (ET)	Topic	Speaker
10:00–10:15 am	Welcome and Introductions	Dorothy Koch, WPO Director Steve Thur, OAR AA
10:15–10:30 am	Overview of NOAA Research and Charge to Reviewers	Emily Menashes, OAR DAA, Programs and Administration
10:30–10:45 am	OAR Portfolios Overview	Wayne Higgins, Climate Portfolio Steward
10:45–11:00 am	Questions and Open Discussion
11:00–11:15 am	Break
11:15–12:00 pm	Director’s Overview	Dorothy Koch, WPO Director
11:15–12:00 pm	Activity Area 1: Overview and Introduction to WPO	John Ten Hoeve, WPO Deputy Director
12:00–12:30 pm	Questions and Open Discussion
12:30–1:30 pm	Lunch
1:30–3:00 pm	Policy and Budget Overview	Melissa Pratt-Zossoungbo, Administrative Officer
	Notice of Funding Opportunity (NOFO) Overview	Christine Bassett, S2S Program Coordinator
	Intern and Fellow Opportunities	Leah Dubots, Management and Program Analyst
	3:00–4:00 pm	Questions and Open Discussion
4:00–4:15 pm	Break
4:15–5:00 pm	Closed Session: OAR Representatives	Review Panel & OAR Reps
5:00 pm	End Day 1

January 25th

Activity Area 2: Weather Research Models, Observations and Forecasting Tools

Download Day 2 Slides

Time (ET)	Topic	Speaker
10:00–10:15 am	Activity Area 2: Overview and Introduction to Programs	Mark Olsen, S2S Deputy Program Manager
10:15–11:15 am	Observations Program and Phased Array Radar	Mark Vincent and Segayle Thompson, Observations Program
	Subseasonal to Seasonal (S2S) Program and Climate Testbed	Jessie Carman, S2S Program Manager
	Earth Prediction Innovation Center (EPIC) Program	Maoyi Huang, EPIC Program Manager
11:15–12:30 pm	Questions and Open Discussion
12:30–1:30 pm	Lunch
1:30–2:30 pm	Closed Session: External Stakeholders	Review Panel & Stakeholders
2:30–2:50 pm	Break
2:50–4:00 pm	Closed Session: Review Panel
4:00 pm	End Day 2

January 26th

Activity Area 3: Advancement and Transition of Weather Research

Download Day 3 Slides

Time (ET)	Topic	Speaker
10:00–10:15 am	Activity Area 3: Overview and Introduction to Programs	Aaron Pratt, JTTI Program Coordinator
10:15–11:30 am	Air Quality (AQ) Program and Fire Weather	Jordan Dale, AQ Program Manager
	Joint Technology Transfer Initiative (JTTI) Program	Chandra Kondragunta, JTTI Program Manager
	Weather Testbeds Program	Jordan Dale, Testbeds Program Manager
	Research to Applications (R2X) and Transitions	Matt Mahalik, Research Transition & Evaluation Lead
11:30–12:30 pm	Questions and Open Discussion
12:30–1:30 pm	Lunch
1:30–2:30 pm	Closed Session: Line Office Representatives	Review Panel & LO Reps
2:30–2:50 pm	Break
2:50–4:00 pm	Closed Session: Review Panel
4:00 pm	End Day 3

January 27th

Activity Area 4: Effective Communication and Coordination of Weather Research

Download Day 4 Slides

Time (ET)	Topic	Speaker
10:00–10:15 am	Activity Area 4: Overview and Introduction to Programs	Castle Williamsberg, Social Science R2X Coordinator
10:15–11:15 pm	Supplemental Appropriations Program	Mark Vincent, Ben Woods, Chris Spells and Renee (Richardson) Keller, Supplemental Program
	Social Science Program	Gina Eosco, Social Science Program Manager
	Policy, Partnerships and Communication	Tamara Battle, Policy & Partnerships Lead
11:15–12:30 pm	Questions and Open Discussion
12:30–1:30 pm	Lunch
1:30–2:15 pm 2:15–3:30 pm	Closed Session: Review Panel Final Discussion and Preliminary Feedback to OAR and WPO
3:30–3:45 pm	Wrap Up and Thank You	Steve Thur, Dorothy Koch, and John Ten Hoeve
3:45–4:30 pm	Closed Session: Review Panel
4:30 pm	End Day 4