Simulated Power Analyses for Observational Studies: An Application to the Affordable Care Act Medicaid Expansion

84 Pages Posted: 7 May 2019 Last revised: 1 Sep 2021

See all articles by Bernard S. Black

Bernard S. Black

Northwestern University

Alex Hollingsworth

Indiana University Bloomington - School of Public & Environmental Affairs (SPEA)

Leticia Nunes

Fundação Getulio Vargas

Kosali Ilayperuma Simon

Indiana University Bloomington - School of Public & Environmental Affairs (SPEA); National Bureau of Economic Research (NBER)

Multiple version iconThere are 2 versions of this paper

Date Written: March 8, 2021

Abstract

Power is an important factor in assessing the likely validity of a statistical estimate.
An analysis with low power is unlikely to produce convincing evidence of a
treatment effect even when one exists. Of greater concern, a statistically significant
estimate from a low-powered analysis is likely to overstate the magnitude of the true
effect size, often finding estimates of the wrong sign or that are several times too
large. Yet statistical power is rarely reported in published economics work. This is in
part because modern research designs are complex enough that power cannot always
be easily ascertained using simple formulae. Power can also be difficult to estimate
in observational settings where researchers may not know—and have no ability to
manipulate—the true treatment effect or other parameters of interest. Using an applied
example—the link between gaining health insurance and mortality—we conduct
a simulated power analysis to outline the importance of power and ways to estimate
power in complex research settings. We find that standard difference-in-differences
and triple differences analyses of Medicaid expansions using county or state mortality
data would need to induce reductions in population mortality of at least 2% to be well
powered. While there is no single, correct method for conducting a simulated power
analysis, our manuscript outlines decisions relevant for applied researchers interested
in conducting simulations appropriate to other settings.

Keywords: simulated power analysis, health insurance, mortality, Medicaid expansion

JEL Classification: C15, I13

Suggested Citation

Black, Bernard S. and Hollingsworth, Alex and Nunes, Leticia and Simon, Kosali Ilayperuma, Simulated Power Analyses for Observational Studies: An Application to the Affordable Care Act Medicaid Expansion (March 8, 2021). Available at SSRN: https://ssrn.com/abstract=3368187 or http://dx.doi.org/10.2139/ssrn.3368187

Bernard S. Black (Contact Author)

Northwestern University ( email )

2001 Sheridan Road
Evanston, IL 60208
United States

Alex Hollingsworth

Indiana University Bloomington - School of Public & Environmental Affairs (SPEA) ( email )

1315 East Tenth Street
Bloomington, IN 47405
United States

HOME PAGE: http://alexjhollingsworth.com

Leticia Nunes

Fundação Getulio Vargas ( email )

Brazil

Kosali Ilayperuma Simon

Indiana University Bloomington - School of Public & Environmental Affairs (SPEA) ( email )

1315 East Tenth Street
Bloomington, IN 47405
United States

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
92
Abstract Views
1,262
rank
252,159
PlumX Metrics