National Weather Service Training Center
Quality Assurance and Quality Control
Introduction | Quality
Assurance versus Quality Control
QA/QC Program
Objectives | Standards and
Guidelines
Data Accuracy
and Credibility | Preventive
Quality Control
An Increased
Focus on Quality Assurance
Return to Main
Page of Web Module
Introduction
The terms Quality Assurance (QA) and Quality Control (QC), often used interchangeably, represent two distinctly different functions. An effective Quality Control Program performs both functions on all raw observational data. This module briefly describes these two processes.
With the automation of data handling within the meteorological community, it is very important that an effective QC program be in place at all levels of the National Weather Service (NWS). A good QA/QC program provides both management and operations with the "data confidence" necessary to create and maintain a high degree of credibility in NWS products and services.
Quality Assurance versus Quality Control
The primary difference between QA and QC is their modes. Quality Assurance, a passive function, leads to the active function of Quality Control.
NWS personnel perform Quality Assurance by reviewing incoming data to ensure that it meets, and is consistent with, the needs of the users, i.e., no suspicious data are part of the data flow. Standards dictate the level of quality assurance review needed to ensure that data are consistent with goals and objectives of the organization. QA inspection is a continuous process.
Using NWS cooperative data as an example, the QA review includes the checking of monthly forms, charts and tapes. Check monthly forms for errors and inconsistencies and ensure the forms conform to standards established by the NWS and theNationalClimaticDataCenter .
A Quality Assurance inspection not only precedes Quality Control, but also identifies what areas require attention to improve data quality. This leads to Quality Control.
Quality Control applies to observations over which NWS personnel have some level of control. Quality Control functions include the improvement of form quality and observational procedures. Both of these areas are a function of the training provided to the data provider. For example, the most effective QC technique to improve cooperative data quality is to provide training to cooperative observers on all aspects of taking, recording, and reporting of their observations.
The primary emphasis on improving data quality must focus on training. One measure of the effectiveness of training is the quality of the forms and reports coming into the office. The quality of the training is directly proportional to the knowledge and effectiveness of the trainer.
This statement directly addresses an often-overlooked aspect of QA/QC. For example, when training a cooperative weather observer or any other data provider, the trainer can only pass on the knowledge he or she individually possesses or has available from appropriate reference material. It is unrealistic to assume that the whole of the trainer’s knowledge level passes to the trainee. Although initial training is important, reinforcement is also needed.
Review Questions
Question 1
Quality assurance is a _____ function while quality control is a _____ function.
A. detailed,
overview
B. passive,
active
C. corrective,
review
D. corrective,
overview
Question 2
Label each of the following as either quality assurance or quality control.
A. Providing
instruction on reading a river gauge
B. Checking
punch tape submissions from cooperative observer
C. Reviewing
SAWRS observation forms
D. Outlining
how to complete a form to an observer
QA/QC Program Objective
A quality QA/QC program must have a set of goals and objectives. The objective of a comprehensive QA/QC Program within the NWS data acquisition process is to increase the quality and reliability of data used in NWS products and services.
These objectives should be defined in terms of individual incoming data sets and require a systematic analysis for each application of these data. Ask users the following questions:
- What are the uses for these data?
- What are the implications of poor quality, corrupted, or erroneous data?
- What are acceptable quality criteria for the data set?
Explore each of these questions in the following sections.
Question 1: What are the uses for these data?
Understanding how external customers utilize NWS data is extremely important. For example, NWS forecasters and hydrologists use incoming cooperative observer reports in various internal operations to generate a number of products and services. Less obvious, however, are the non-NWS users who use cooperative reports. Examples of external users include other Federal agencies, state and local government agencies, State Climatologists, Regional Climate enters , the National Climatic Data Center , private weather agencies, among others. Contact all users and determine how data use in their systems and how they use these raw data. NWS personnel should determine if external customers apply Quality Assurance/Quality Control procedures to incoming NWS data and if so, if the procedures could be applied internally to review NWS data.
Question 2: What are the implications of poor quality, corrupted, or erroneous data?
This question produces a wide range of answers. Each user evaluates the effect of poor data ingested into individual programs. Certain incoming data elements may seriously affect the user's ability to do their job. Others may have little or no impact. An erroneous cooperative rainfall report, for example, if not flagged and corrected, might prove devastating if allowed to impact NWS statements and/or warnings. Conversely, the same report may have little or no serious implications on daily climate summaries or general information statements.
Question 3: What are acceptable quality criteria for the data set?
Perhaps the most critical of all three questions is getting each data user to provide acceptable quality criteria for data sets being incorporated into their products. Acceptable quality levels provided by one user may be entirely different from those provided by another.
The question is even further complicated when separate data elements within a data set, such as a cooperative report, are weighted differently in different situations. In other words, a hydrologist views cooperative temperature reports as insignificant during routine operations. However, should there be a possibility of rapidly rising water levels resulting from a winter thawing trend, the temperature element of the cooperative report is dramatically more critical. The user must consider extreme situations when establishing acceptable data quality criteria. The process of determining these criteria logically leads to the development of standards to insure that incoming data satisfies all users' needs.
Review Questions
Question 3
It is important to consider how non-NWS customers use NWS observational data prior to establishing a QA/QC program.
Question 4
Poor quality, corrupted, or erroneous data could:
A. Lead to
incorrect statements or warnings
B. Provide
misleading forecast guidance
C. Reflect
poorly on the credibility of NWS data
D. All of
the above
Question 5
Who defines quality criteria for data sets?
A. National
Weather Service Headquarters
B. Data
users
C. The data
provider
D. Office of
the Federal Coordinator for Meteorology
Standards and Guidelines
Without detailed standards, it is nearly impossible to enforce quality controls. Standards and guidelines form the basis for developing control procedures. The initial process of establishing standards can be broken out into three (3) functions. They are:
- Develop goal-oriented standards, policies, and procedures based on user-defined data quality requirements.
- Confirm that draft standards are acceptable to all users.
- Ensure that developed standards conform to the primary goals of the organization.
One of the most difficult aspects of developing standards and guidelines is the process of arriving at a level of data quality that meets the needs of the users while conforming to available data acquisition hardware and software capabilities. All too often, a user's definition of an acceptable level of data quality is unattainable or unrealistic. Acceptable quality control standards and guidelines must be:
- Acceptable and consistent with the needs of internal and external users
- Meet the goals of the organization
- Attainable within available budget constraints and human resources
- Compatible and within hardware and software limitations
- Effective
- Efficient
After drafting quality standards and guidelines, confirm their effectiveness by consulting with each individual user group. It is the primary responsibility of Quality Assurance is to ensure that the users' needs have been satisfied. Even so, quality standards must keep the needs of the users in perspective within the goals, objectives, and limitations of the organization.
Review Questions
Question 6
Only the needs of the users should be taken into consideration when developing data quality standards.
Data Accuracy and Credibility
Inaccurate or unreliable data will be directly reflected in all products and services provided by the National Weather Service. An effective QA/QC program must minimize the flow of bad data making its way into the system. Erroneous data not subjected to a strict QA/QC program may go undetected and find its way into products. Because employees responsible for data acquisition cannot continuously monitor incoming data for precision and accuracy, a quality control program that employees need a mixture of both human and automated QA/QC methods.
Data precision is extremely important. Even so, data must be accurate. Accuracy extends beyond precision. One can painstakingly measure temperature to a thousandth of a degree, but if misread by 20 degrees, what good is it? As a result, accurate data must:
- Be uniquely and readily identifiable.
- Originate from consistently accurate equipment.
- Be acquired and transmitted in a timely manner.
- Possess a high degree of dependability and consistency.
The first two items relate primarily to the meta data of individual data collection points as well as the method of acquiring it. When a human observer obtains the data, the last two items relate to the accuracy of the observer. Ask the observer the following questions:
- Exactly where were these data obtained?
- What type of instrument reported these data? (NWS or other)
- Has the instrument reporting these data been calibrated and/or maintained?
- Is the exposure of the instrument acceptable?
- Did the observer, if any, read the instrument properly?
These factors directly affect data accuracy and credibility. Ask the observer these quality assurance questions:
- Were these data taken at a predetermined and scheduled time?
- Were the data transmitted in a timely manner?
- Does the observer have the capability to do his or her job accurately?
Review Questions
Question 7
Accurate data must:
A. Be as precise
as possible.
B. Be
independent of equipment considerations.
C. Be
received within 12 hours of its observing time.
D. Possess a
high degree of dependability and consistency.
Preventive Quality Control
Another important part of any general QC Program is the preventive QC effort. Look at how this works with an example from the Cooperative Observing Program.
In the Cooperative Observing Program, the most effective Preventative Quality Control is the one-on-one contact performed during scheduled and emergency visits. Review procedures relating to the proper use of instruments, completion of forms, or transmitting the data at each station visit. You must ensure that the Cooperative Observers have confidence in their abilities to carry out their responsibilities. Additional time spent in retraining or refreshing the Cooperative Observer on methods and procedures is time well spent. This type of preventative QC also helps observers who exhibit no immediate problems, but might benefit from the reassurance provided by your visit.
For the Cooperative Observing Program, preventative Quality Control procedures during station visits should include:
- Review current monthly form(s).
- Review instrument observational procedures.
- Review past forms to minimize reoccurring problems or patterns.
- Inspect all equipment, and repair and/or calibrate as necessary.
- Review requirements and expectations with observer.
- Confirm schedules and/or deadlines.
- Make observer feel a part of the NWS.
- Review exposure of the instrumentation.
A visit including these preventative Quality Control procedures will undoubtedly improve data quality.
Review Questions
Question 8
The old adage, "An ounce of prevention is worth a pound of cure" applies to data quality control.
An Increased Focus on Quality Assurance
In the NWS, an increased emphasis on the passive function of Quality Assurance will result in a decrease in the time devoted to the active function of Quality Control. With close attention to these issues and adherence to QA/QC standards, the NWS will be able to maintain the public confidence in its data, its products, and its services.
Updated 08/06/2007