Why is testing important?
Testing helps people to evaluate, communicate, and ultimately improve the performance and quality of cookstoves and fuels. Without testing, consumers, retailers, humanitarian agencies, and governments would have no quantitative, rigorous, or consistent basis to evaluate technology options. Testing provides an evidence-based understanding of performance and quality that can drive innovation to improve technologies over time, thus achieving greater impact and adoption.
What does testing measure?
There are many protocols to test stoves and fuels, and many different metrics. Fuel consumption, efficiency, emissions, safety, and durability are evaluated in many test protocols. Most test protocols test stoves and fuels in combination.
- Fuel efficiency metrics provide data to understand household impacts related to fuel expenditure and/or fuel collection time, and environmental impacts related to forest degradation and biomass use.
- Emissions metrics provide data to understand climate impacts through long-lived and short-lived climate pollutants, as well as the health impacts through air pollutants, such as particulate matter. Commonly measured emissions include carbon monoxide, carbon dioxide, particulate matter, methane, as well as black carbon and organic carbon.
- Other metrics, such as firepower, cooking power, and turndown ratio, help us to understand the cooking experience, and thus impacts on adoption and consumer preference.
Who performs stove testing?
Any organization can perform stove testing, including manufacturers, research labs, and testing centers. However, testing can require extensive equipment and training, especially for emissions testing. Although any organization can conduct cookstove testing, care should be taken to ensure that the organization conducting the testing has the experience, knowledge, procedures, and equipment needed to collect and analyze accurate and quality data. A list of organizations that provide testing services is available on the Alliance website. Inclusion in this list is not an endorsement from the Alliance. Each organization has different capabilities, services, and prices, and should be contacted directly for more information.
What is the value of lab testing? Of field testing?
Both laboratory and field testing are essential components of the testing process, and each have tradeoffs. Lab testing provides results under controlled conditions, which can be used for comparing different models or as the basis for certification. Lab results can also be used to evaluate the impact of design modifications. Lab results, however, often differ from field results. To characterize potential impacts, stoves must be tested under actual, real-use conditions. Field testing evaluates performance under a range of context specific conditions, including variations in cooking styles and local cuisines. These tests can also evaluate whether users’ needs are addressed, giving an indication of consumer preferences. Lab and field testing are complementary and together can provide a comprehensive assessment.
How can the validity of testing results be determined?
Individual protocols have different uses and limitations. Protocols should include guidance on appropriate application, interpretation of results, and limitations. Evaluation of stoves and fuels requires careful consideration of available protocols to ensure that the appropriate one is selected, and results are interpreted correctly. You should ask the questions “Is this protocol appropriate for my testing goals?”, “Can this test answer my questions?”, and “Is this testing center capable of conducting these tests?”. Testers should include these considerations when reporting testing results.
Determining the validity of testing results requires an understanding of sample size and statistical analysis methods. For accurate and thorough reporting of results, reports should include relevant statistics such as standard deviation, coefficient of variation, confidence interval, etc., in addition to averages or tier levels.
What causes variability in testing results?
There are many factors that can influence performance, including: fuel type and quality, cooking vessels, user behavior, stove operation, equipment maintenance and calibration, data analysis, testing protocol and interpretation, testing environment, and others. Some of these factors cannot be controlled (e.g., there is inherent variability of biomass combustion), and some of these factors must be appropriately accounted for in the data analysis (e.g., local boiling point).
Different protocols require different levels of controlling the cooking system, testing environment, and operation, depending on whether the goal is to compare stoves or to assess the potential impacts. Many lab testing protocols require a higher degree of control for results to be valid and comparable. If these factors are not controlled for, either within a test or between tests, the same stove can appear to have very different performance. Regardless of which protocol is being used, testers should measure critical factors and report them along with the test metrics to provide context for a more accurate assessment.
What are the IWA Guidelines for Evaluating Cookstove Performance?
International Workshop Agreements are a streamlined process within the International Organization for Standardization (ISO) that can be a first step towards formal ISO standards. In February 2012, more than 90 stakeholders from 23 countries met for an ISO meeting that resulted in unanimous support among meeting participants for IWA 11:2012 Guidelines for Evaluating Cookstove Performance. The IWA tiers are interim international guidelines for stove performance and safety. The goal is to provide common and easy-to-understand ratings for governments, donors, investors, and consumers to make decisions about technology options.
What do the tiers mean?
The IWA framework rates cookstoves on four indicators (efficiency, indoor emissions, total emissions, safety), each along 5 Tiers (0: lowest performing to 4: highest performing). For each indicator, the Tier boundaries are defined by quantitative values based on data from laboratory testing. The protocols that have been mapped to tiers are the Water Boiling Test 4.1.2 (WBT) and the Biomass Stove Safety Protocol 1.1, although the IWA framework was designed to accommodate other protocols.
What are the benefits of the WBT? What are the limitations?
The WBT is one of the more common lab tests used to evaluate cookstove performance. It is a useful tool during the research & development process to understand the effect of various design options, to compare different designs with each other, to help identify products for field assessments, and to ensure design specifications have been met. The WBT does NOT assess stove performance under field conditions, and therefore cannot quantify the precise impact the cookstove will have in any specific community. Because of the higher resource requirement for field testing, lab testing is sometimes used as the basis for modeling field performance. Such an approach should be taken cautiously and transparently and include clear statement of limitations and assumptions. These models should be updated with field data when it is available.
A number of specific questions about the WBT have been identified, including the applicability of the test sequence, the calculation of efficiency, the use of a lid on the pot, and the consideration of char. These and other issues are described in the WBT protocol, and should be considered when interpreting WBT test results. Similar limitations and issues should be considered when interpreting results from other existing test protocols.
Will the WBT be replaced?
The IWA included a recommendation that new protocols be developed and/or current protocols updated. The Alliance and many others in the sector recognize the need for improvement. Experts have been working continuously over the last few years, through the ISO Technical Committee 285 (TC 285), to develop improved methods and standards. Protocols will be published as soon as they are complete. The ISO process minimizes bias by including any interested countries or experts in the process. The ISO process is also consensus-driven. TC 285 currently includes experts from 45 countries and 8 liaison organizations. Information on how to participate is available here.
Does the Alliance only engage in lab testing?
The Alliance supports both lab and field testing as both are important, complementary assessment tools. The Alliance references lab testing and the IWA guidelines for our definitions of “clean” and “efficient”, as lab testing is generally the basis of standards. After updated lab testing protocols and tiers are published, the Alliance will reassess our definitions of “clean” and “efficient” to ensure they reflect the latest international consensus on stove and fuel performance.
What is the Alliance stance of specific stove types or fuel types?
The Alliance is technology and fuel neutral. We understand that there are a multitude of needs and preferences that vary from country to country, and even household to household. Our goal is to support the development of cookstoves and fuels markets comprised of cleaner and more efficient options that are accessible to everyone. We do not define entire categories of stoves and fuels as clean or efficient, as performance is variable even within categories. Instead, the Alliance focuses on the performance of each stove and/or fuel; our definitions of clean and efficient are based on proven emissions and efficiency. Any stove or fuel can be designated as clean or efficient as long as performance standards are met.