Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in Their Judgments About Future Energy Technologies

42 Pages Posted: 25 Oct 2016

See all articles by Gregory F. Nemet

Gregory F. Nemet

University of Wisconsin - Madison - La Follette School of Public Affairs

Laura Diaz Anadon

Harvard University, Harvard Kennedy School (HKS), Belfer Center for Science and International Affairs (BCSIA) ; Harvard University - Harvard Kennedy School (HKS)

Elena Verdolini

Fondazione Eni Enrico Mattei (FEEM), CMCC - Euro Mediterranean Centre for Climate Change

Date Written: October 25, 2016

Abstract

Expert elicitations are frequently used to characterize future technology outcomes. However their usefulness is limited, in part because: estimates across studies are not easily comparable; choices in survey design and expert selection may bias results; and over-confidence is a persistent problem. We provide quantitative evidence of how these choices affect experts’ estimates of the costs of future energy technologies. We harmonize data from 19 elicitations, involving 215 experts, on the 2030 costs of 5 energy technologies: nuclear, biofuels, bioelectricity, solar, and carbon capture. We control for expert characteristics, survey design, and public R&D investment levels on which the elicited values are conditional. We find that, on average, when experts respond to elicitations in person, they ascribe lower confidence (larger uncertainty) to their estimates than when responding via mail or online. In-person interviews also produce more optimistic assessments of best-case (10th percentile) outcomes. The impacts of expert affiliation — government, private sector, or academic — and geography — US or EU — are also significant; academics and US experts have lower confidence than other types of experts. Higher R&D investment levels have no effect on the confidence of experts’ judgments. R&D reduces both the median and breakthrough (10th percentile) cost estimates, although the size of the effect varies across technologies. These results indicate the source, direction, and size of bias in energy technology elicitations. They also point to the technology specificity of some of the effects. These biases should be seriously considered, both in interpreting the results of existing elicitations and in designing new ones.

Keywords: Expert Elicitations, Uncertainty, Energy Technologies, Heuristic Biases, Survey Design

JEL Classification: O13, O14, Q4

Suggested Citation

Nemet, Gregory F. and Diaz Anadon, Laura and Verdolini, Elena, Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in Their Judgments About Future Energy Technologies (October 25, 2016). FEEM Working Paper No. 061.2016, Available at SSRN: https://ssrn.com/abstract=2858786 or http://dx.doi.org/10.2139/ssrn.2858786

Gregory F. Nemet (Contact Author)

University of Wisconsin - Madison - La Follette School of Public Affairs ( email )

1225 Observatory Drive
Madison, WI 53705
United States

Laura Diaz Anadon

Harvard University, Harvard Kennedy School (HKS), Belfer Center for Science and International Affairs (BCSIA) ( email )

79 JFK Street
Cambridge, MA 02138
United States

Harvard University - Harvard Kennedy School (HKS) ( email )

79 John F. Kennedy Street
Cambridge, MA 02138
United States

Elena Verdolini

Fondazione Eni Enrico Mattei (FEEM), CMCC - Euro Mediterranean Centre for Climate Change ( email )

C.so Magenta 63
Milano, 20123
Italy

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
24
Abstract Views
373
PlumX Metrics