Variation in Performance of Commonly Used Statistical Methods for Estimating Effectiveness of State-Level Opioid Policies on Opioid-Related Mortality

35 Pages Posted: 1 May 2020 Last revised: 9 Nov 2022

See all articles by Beth Ann Griffin

Beth Ann Griffin

RAND Corporation

megan schuler

RAND Corporation

Elizabeth A. Stuart

Johns Hopkins University - Bloomberg School of Public Health

Stephen Patrick

Vanderbilt University - Medical Center

Elizabeth McNeer

Vanderbilt University - Medical Center

Rosanna Smart

RAND Corporation

David Powell

RAND Corporation

Bradley Stein

RAND Corporation - Pittsburgh PA Offices

Terry Schell

RAND Corporation

Rosalie Liccardo Pacula

University of Southern California - Schaeffer Center for Health Policy and Economics; National Bureau of Economic Research (NBER)

Date Written: April 2020

Abstract

Over the last two decades, there has been a surge of opioid-related overdose deaths resulting in a myriad of state policy responses. Researchers have evaluated the effectiveness of such policies using a wide-range of statistical models, each of which requires multiple design choices that can influence the accuracy and precision of the estimated policy effects. This simulation study used real-world data to compare model performance across a range of important statistical constructs to better understand which methods are appropriate for measuring the impacts of state-level opioid policies on opioid-related mortality. Our findings show that many commonly-used methods have very low statistical power to detect a significant policy effect (< 10%) when the policy effect size is small yet impactful (e.g., 5% reduction in opioid mortality). Many methods yielded high rates of Type I error, raising concerns of spurious conclusions about policy effectiveness. Finally, model performance was reduced when policy effectiveness had incremental, rather than instantaneous, onset. These findings highlight the limitations of existing statistical methods under scenarios that are likely to affect real-world policy studies. Given the necessity of identifying and implementing effective opioid-related policies, researchers and policymakers should be mindful of evaluation study statistical design.

Suggested Citation

Griffin, Beth Ann and schuler, megan and Stuart, Elizabeth A. and Patrick, Stephen and McNeer, Elizabeth and Smart, Rosanna and Powell, David and Stein, Bradley and Schell, Terry and Pacula, Rosalie Liccardo, Variation in Performance of Commonly Used Statistical Methods for Estimating Effectiveness of State-Level Opioid Policies on Opioid-Related Mortality (April 2020). NBER Working Paper No. w27029, Available at SSRN: https://ssrn.com/abstract=3588151

Beth Ann Griffin (Contact Author)

RAND Corporation

1776 Main Street
P.O. Box 2138
Santa Monica, CA 90407-2138
United States

Megan Schuler

RAND Corporation

Elizabeth A. Stuart

Johns Hopkins University - Bloomberg School of Public Health ( email )

615 North Wolfe Street
Baltimore, MD 21205
United States

HOME PAGE: http://www.biostat.jhsph.edu/~estuart

Stephen Patrick

Vanderbilt University - Medical Center ( email )

1211 Medical Center Dr
Nashville, TN 37232
United States

Elizabeth McNeer

Vanderbilt University - Medical Center ( email )

1211 Medical Center Dr
Nashville, TN 37232
United States

Rosanna Smart

RAND Corporation ( email )

1776 Main Street
P.O. Box 2138
Santa Monica, CA 90407-2138
United States

David Powell

RAND Corporation ( email )

1776 Main Street
P.O. Box 2138
Santa Monica, CA 90407-2138
United States

Bradley Stein

RAND Corporation - Pittsburgh PA Offices ( email )

Pittsburgh, PA
United States

Terry Schell

RAND Corporation

Rosalie Liccardo Pacula

University of Southern California - Schaeffer Center for Health Policy and Economics ( email )

635 Downey Way
Los Angeles, CA 90089-3333
United States

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
7
Abstract Views
215
PlumX Metrics