Quality Assurance and Quality Control
Introduction | Quality
Assurance versus Quality Control
QA/QC Program Objectives | Standards and Guidelines
Data Accuracy and Credibility | Preventive Quality Control
An Increased Focus on Quality Assurance
Return to Main Page of Web Module
The terms Quality Assurance (QA) and Quality Control (QC), often used interchangeably, represent two distinctly different functions. An effective Quality Control Program performs both functions on all raw observational data. This module briefly describes these two processes.
With the automation of data handling within the meteorological community, it is very important that an effective QC program be in place at all levels of the National Weather Service (NWS). A good QA/QC program provides both management and operations with the "data confidence" necessary to create and maintain a high degree of credibility in NWS products and services.
The primary difference between QA and QC is their modes. Quality Assurance, a passive function, leads to the active function of Quality Control.
NWS personnel perform Quality Assurance by reviewing incoming data to ensure that it meets, and is consistent with, the needs of the users, i.e., no suspicious data are part of the data flow. Standards dictate the level of quality assurance review needed to ensure that data are consistent with goals and objectives of the organization. QA inspection is a continuous process.
Using NWS cooperative data as an example, the QA review includes the checking of monthly forms, charts and tapes. Check monthly forms for errors and inconsistencies and ensure the forms conform to standards established by the NWS and theNationalClimaticDataCenter .
A Quality Assurance inspection not only precedes Quality Control, but also identifies what areas require attention to improve data quality. This leads to Quality Control.
Quality Control applies to observations over which NWS personnel have some level of control. Quality Control functions include the improvement of form quality and observational procedures. Both of these areas are a function of the training provided to the data provider. For example, the most effective QC technique to improve cooperative data quality is to provide training to cooperative observers on all aspects of taking, recording, and reporting of their observations.
The primary emphasis on improving data quality must focus on training. One measure of the effectiveness of training is the quality of the forms and reports coming into the office. The quality of the training is directly proportional to the knowledge and effectiveness of the trainer.
This statement directly addresses an often-overlooked aspect of QA/QC. For example, when training a cooperative weather observer or any other data provider, the trainer can only pass on the knowledge he or she individually possesses or has available from appropriate reference material. It is unrealistic to assume that the whole of the trainer’s knowledge level passes to the trainee. Although initial training is important, reinforcement is also needed.
Quality assurance is a _____ function while quality control is a _____ function.
Label each of the following as either quality assurance or quality control.
A quality QA/QC program must have a set of goals and objectives. The objective of a comprehensive QA/QC Program within the NWS data acquisition process is to increase the quality and reliability of data used in NWS products and services.
These objectives should be defined in terms of individual incoming data sets and require a systematic analysis for each application of these data. Ask users the following questions:
Explore each of these questions in the following sections.
Understanding how external customers utilize NWS data is extremely important. For example, NWS forecasters and hydrologists use incoming cooperative observer reports in various internal operations to generate a number of products and services. Less obvious, however, are the non-NWS users who use cooperative reports. Examples of external users include other Federal agencies, state and local government agencies, State Climatologists, Regional Climate enters , the National Climatic Data Center , private weather agencies, among others. Contact all users and determine how data use in their systems and how they use these raw data. NWS personnel should determine if external customers apply Quality Assurance/Quality Control procedures to incoming NWS data and if so, if the procedures could be applied internally to review NWS data.
This question produces a wide range of answers. Each user evaluates the effect of poor data ingested into individual programs. Certain incoming data elements may seriously affect the user's ability to do their job. Others may have little or no impact. An erroneous cooperative rainfall report, for example, if not flagged and corrected, might prove devastating if allowed to impact NWS statements and/or warnings. Conversely, the same report may have little or no serious implications on daily climate summaries or general information statements.
Perhaps the most critical of all three questions is getting each data user to provide acceptable quality criteria for data sets being incorporated into their products. Acceptable quality levels provided by one user may be entirely different from those provided by another.
The question is even further complicated when separate data elements within a data set, such as a cooperative report, are weighted differently in different situations. In other words, a hydrologist views cooperative temperature reports as insignificant during routine operations. However, should there be a possibility of rapidly rising water levels resulting from a winter thawing trend, the temperature element of the cooperative report is dramatically more critical. The user must consider extreme situations when establishing acceptable data quality criteria. The process of determining these criteria logically leads to the development of standards to insure that incoming data satisfies all users' needs.
It is important to consider how non-NWS customers use NWS observational data prior to establishing a QA/QC program.
Poor quality, corrupted, or erroneous data could:
Who defines quality criteria for data sets?
Without detailed standards, it is nearly impossible to enforce quality controls. Standards and guidelines form the basis for developing control procedures. The initial process of establishing standards can be broken out into three (3) functions. They are:
One of the most difficult aspects of developing standards and guidelines is the process of arriving at a level of data quality that meets the needs of the users while conforming to available data acquisition hardware and software capabilities. All too often, a user's definition of an acceptable level of data quality is unattainable or unrealistic. Acceptable quality control standards and guidelines must be:
After drafting quality standards and guidelines, confirm their effectiveness by consulting with each individual user group. It is the primary responsibility of Quality Assurance is to ensure that the users' needs have been satisfied. Even so, quality standards must keep the needs of the users in perspective within the goals, objectives, and limitations of the organization.
Only the needs of the users should be taken into consideration when developing data quality standards.
Inaccurate or unreliable data will be directly reflected in all products and services provided by the National Weather Service. An effective QA/QC program must minimize the flow of bad data making its way into the system. Erroneous data not subjected to a strict QA/QC program may go undetected and find its way into products. Because employees responsible for data acquisition cannot continuously monitor incoming data for precision and accuracy, a quality control program that employees need a mixture of both human and automated QA/QC methods.
Data precision is extremely important. Even so, data must be accurate. Accuracy extends beyond precision. One can painstakingly measure temperature to a thousandth of a degree, but if misread by 20 degrees, what good is it? As a result, accurate data must:
The first two items relate primarily to the meta data of individual data collection points as well as the method of acquiring it. When a human observer obtains the data, the last two items relate to the accuracy of the observer. Ask the observer the following questions:
These factors directly affect data accuracy and credibility. Ask the observer these quality assurance questions:
Accurate data must:
Another important part of any general QC Program is the preventive QC effort. Look at how this works with an example from the Cooperative Observing Program.
In the Cooperative Observing Program, the most effective Preventative Quality Control is the one-on-one contact performed during scheduled and emergency visits. Review procedures relating to the proper use of instruments, completion of forms, or transmitting the data at each station visit. You must ensure that the Cooperative Observers have confidence in their abilities to carry out their responsibilities. Additional time spent in retraining or refreshing the Cooperative Observer on methods and procedures is time well spent. This type of preventative QC also helps observers who exhibit no immediate problems, but might benefit from the reassurance provided by your visit.
For the Cooperative Observing Program, preventative Quality Control procedures during station visits should include:
A visit including these preventative Quality Control procedures will undoubtedly improve data quality.
The old adage, "An ounce of prevention is worth a pound of cure" applies to data quality control.
In the NWS, an increased emphasis on the passive function of Quality Assurance will result in a decrease in the time devoted to the active function of Quality Control. With close attention to these issues and adherence to QA/QC standards, the NWS will be able to maintain the public confidence in its data, its products, and its services.