The AffiliateMarketIngtools of Sciences, Engineering and Medicine
Office of Congressional and government Affairs
At A Glance
: National Energy Policy: Research and Development
: 07/18/2001
Session: 107th Congress (First Session)
: Robert Fri

Director, National Museum of Natural History, , D.C. and Chair, Committee on Benefits of DoE R&D on Energy Efficiency and Fossil Energy, Board on Energy and Environmental Systems, Division on Engineering and Physical Sciences, National Research Council, The AffiliateMarketIngtools

: Senate
: Committee on Energy and Natural Resources






JULY 18, 2001

Good morning Mr. Chairman, Senator Murkowski, and members of the Committee. My name is Robert Fri. I am Director of the National Museum of Natural History and served as the Chair of the Committee on Benefits of DOE R&D on Energy Efficiency and Fossil Energy of the National Research Council. The Research Council is the operating arm of the AffiliateMarketIngtools of Sciences, AffiliateMarketIngtools of Engineering, and the Institute of Medicine, chartered by Congress in 1863 to advise the government on matters of science and technology. The committee I have chaired this last year was given the charge of assessing the benefits and costs of Department of Energy research and development in fossil energy and energy efficiency since 1978 by the U.S. Congress. The committee’s report was released yesterday afternoon. I appreciate this opportunity to summarize it for you and to respond to your questions about our assignment.

The executive summary of the report is attached to my written testimony, and both the summary and the full report describe the analytic approach we adopted to carry out our work. This background provides essential context for our conclusions. Although I will not dwell today on these methodological details, it is important to point out that:

1. We studied only the fossil energy and energy efficiency programs of the Department of Energy (DOE) because these programs fall within the jurisdiction of the House Interior Appropriations Subcommittee, which directed that the study be undertaken. Since 1978, DOE has spent about $22.3 billion on these programs, or about 26 percent of its total energy R&D expenditures.

2. Our assignment was to assess the benefits actually realized since 1978 as a result of DOE-sponsored research in these programs. We did not account for benefits that might occur in the future. This focus on outcomes distinguishes our study from most other evaluations of DOE’s research.

3. Time and human resources constrained us to analyze in depth thirty-nine of DOE’s research programs. We believe that this is a representative sample for purposes of this study, but falls well short of looking at all of the research conducted by DOE over the past two decades. This is particularly true for the energy efficiency area.

4. We had a dual assignment. The more obvious one was to assess the benefits and costs of energy R&D. The second but equally important task was to develop an analytic framework for conducting such assessments in the future.

This said, in the next few minutes I want to concentrate on the major messages of the report. In particular, I would like focus on four questions:

• What should we count as the benefits of energy research and development?

• Did the benefits of the programs we analyzed exceed their cost?

• What actions might improve the odds of successful energy R&D?

• How can the evaluation of benefits be improved in the future?

The most fundamental issue we addressed was how to define and systematically capture the diverse benefits that result from publicly funded research. To answer this question, we developed an analytic framework designed to capture two dimensions of such research: 1) that DOE research is expected to produce public benefits that the private economy cannot reap, and 2) that some benefits may be created even when a technology does not immediately enter the marketplace to a significant degree.

We identified the public benefits to be captured as those associated with DOE national energy mission:

• Economic benefits, measured by the change in the market value of goods and services resulting from the introduction of a technology stemming from DOE research.

• Environmental benefits, based on changes in the quality of the environment that have occurred as a result of DOE research.

• Security benefits, measured by changes in the probability or severity of abnormal energy-related events.

To characterize the uncertainty about whether research will in fact produce benefits that can be captured, we defined three categories of research outcomes:

• Realized benefits, which are benefits almost certain to be produced. An example is the cost saving resulting from the development of electronic ballasts for fluorescent lights.

• Option benefits, which are associated with technologies that are fully developed but for which economic and policy conditions are not yet favorable for commercialization. Integrated gasifier-combined cycle technology is an example of research that has produced an option benefit.

• Knowledge benefits include all other benefits that we identified, because all research produces some knowledge. We recognize that this is a catch-all category, and that a more refined analysis of knowledge benefits would improve our methodology.

Using these definitions, we created an accounting framework to provide a consistent, comprehensive assessment of the benefits and costs of the fossil energy and energy efficiency programs. The framework is a matrix, shown on the chart before you. We also defined a set of rules that provide a calculus for measuring the values to be entered in each of the cells. These rules are thoroughly documented in an appendix to our report.

We successfully applied this analytic framework to thirty-nine technologies funded by DOE since 1978. We found that these programs yielded significant realized benefits, important technological options for potential application in the future, and useful additions to the stock of engineering and scientific knowledge. Tables 2 and 3 in the executive summary show how each technology we studied produced benefits in one or more elements of the matrix.

Based on this analysis, we were able to address whether the benefits we identified exceed the cost of producing them. Our findings on this question are:

1. The estimated total net realized economic benefits associated with the energy efficiency program we reviewed were about $30 billion, substantially exceeding both the $1.6 billion cost of the representative sample of programs that we analyzed and the the $7 billion in DOE’s total research investment in energy efficiency since 1978. Most of these benefits are attributable to three relatively modest projects in the buildings sector carried on in the late 1970s and 1980s – more efficient refrigerators, electronic ballasts, and low-e glass.

2. The estimated realized economic benefits associated with the fossil energy program amounted to nearly $11 billion, approximately equal to the cost of DOE’s research investment. However, the benefits of fossil energy programs conducted from 1978 to 1986, which included several alternate fuels projects, produced benefits of $3.4 billion and cost $6 billion. From 1986 forward, the economic benefits of $7.4 billion exceed the costs of $ 4.5 billion.

3. Although quantifying environmental benefits is difficult, we estimate that both programs realized benefits of this type valued at between $60 billion and $90 billion. Fossil energy programs that reduced nitrogen oxide emission account for most of this benefit. Other environmental benefits came from reducing emissions through energy efficiency.

4. Both programs produce important technologies that are viable options for reasonable policy and economic conditions that are likely to exist in the future . Chief among these option benefits are the Partnership for a New Generation of Vehicles, the Integrated Gasifier Combined Cycle program, and the Advanced Turbine Program.

5. National security has been enhanced by a number of programs, notably fossil energy programs that increased domestic oil production and reserves additions and efficiency programs that reduced oil consumption. However, DOE’s research programs designed to reduce dependence on oil in the transportation sector have been disappointing so far.

Based on our analysis of these programs, we found that the benefits flowing from DOE’s R&D programs were influenced by the structure and management of the programs. Among the useful lessons learned that can improve the odds of conducting successful research are the following:

• The largest realized benefits accrued in areas where public funding would be expected to have considerable leverage. Thus, the buildings sector is fragmented and the prevailing incentive structure is not conducive to technological innovation. Similarly, the nitrogen oxide reduction achieved in fossil energy is an environmental benefit that the private markets cannot easily capture. We believe, therefore, that DOE’s research should focus on achieving the department’s national public good goals.

• Important but smaller benefits were achieved in fossil energy’s oil and gas programs and energy efficiency’s industry programs. We concluded that DOE participation in these areas took advantage of private sector activity to realize additional public benefits. The lesson is that a clearly defined DOE role is crucial to ensuring that public funding is likely to produce appropriate benefits.

• It is particularly important that DOE manage a balanced portfolio of research. Individual research projects may well fail to achieve their goals, but DOE and Congressional policymakers should not view these as symptoms of overall program failure. Even failures generate considerable knowledge and a well-designed R&D program will inevitably include such failures.

• Where DOE seeks to develop technologies for near term deployment (as in the industrial energy efficiency program, for example), success is more likely when technological goals are consistent with the economic incentives of users to adopt the technologies. Standards can also serve as an important incentive, and the committee saw cases of both success and failure. Our case studies include a number of instances that did not meet this condition and so failed to produce significant economic benefits.

• Our case studies highlighted the need for periodic reevaluation of goals against changes in the regulatory or policy environment, projected energy prices and availability, and the performance of alternative technologies. Similarly, DOE should develop clear performance targets and milestones for achieving program goals. To evaluate progress against goals, we recommend that DOE expand its reliance of regular, independent, peer reviews that enlist the participation of experts who are not otherwise involved in DOE’s programs.

Finally, we addressed the question of how the evaluation of benefits can be improved in the future. We reviewed many other evaluations of DOE programs and found no consistent methodology or framework estimating and evaluating the benefits of research. This inconsistency was often associated with an overstatement of economic benefits and/or a tendency to assign too much weight to realized economic benefits (only one of the nine boxes in our matrix).

On the other hand, we believe that the benefits matrix adopted for this study is a robust framework for evaluating program outcomes. Its application imposes a rigor on the evaluation process that clarifies the benefits achieved and the relationship among them. Accordingly, we recommend that DOE adopt an analytic framework similar to that used in our study as a uniform methodology for assessing the costs and benefits of its R&D programs. DOE should use this framework for reporting to Congress on its programs and goals under the terms of the Government Performance and Results Act.

We recognize, however, that the framework we developed for this study requires refinement. Among other things, DOE should improve the guidelines for benefits characterization and adopt consistent assumptions to be used across programs. As a first step, DOE should convene a workshop of analysts, decision-makers, and members of our committee to discuss the problems we encountered in the application of our framework. Longer term, DOE should seek to enhance the transparency of the process by, among other things, providing external peer review of the application of the framework.

That concludes my prepared remarks, Mr. Chairman. I want to thank the members of our committee for the extraordinary effort they put into this challenging assignment, and to express our collective appreciation to the DOE staff that so diligently worked to respond to our extensive requests for data and analysis. I would now be pleased to respond to any questions you may have.