The Importance of Covariation among Impact Estimates in Monte Carlo Simulations: An Illustration from the BOND Benefit-Cost Analysis

Abstract

The findings from most benefit-cost analyses (BCA) are subject to considerable uncertainty. In BCAs of social programs, much of this uncertainty arises from sampling error to which the impact estimates used in determining benefits and costs are subject. Such uncertainty in BCAs of social policies can be addressed by Monte Carlo analysis. A potential issue with Monte Carlo analysis is that the individual impact estimates may be correlated. In principle, the potential correlation among impacts should be explicitly treated in Monte Carlo studies—for example, by drawing from a joint distribution of parameters. Because this is often difficult to do and not done, it is useful to investigate how important it is to take account of covariation among impact estimates in conducting Monte Carlo analyses. To do this, we compare the results of a CBA Monte Carlo analysis that ignores any covariation between the impacts with one that takes full account of the covariation. We do this twice. First, we conduct a simple Monte Carlo analysis of simulated data containing only two impact estimates that are highly correlated. Second, we look at a recent Monte Carlo analysis of a complex BCA from a large social experiment.

Keywords: Benefit-Cost Analysis, Monte Carlo, confidence interval, covariation, BOND