Assessing Nudge Scalability: Two Lessons from Large-scale RCTs

80 Pages Posted: 20 Jan 2022 Last revised: 7 Mar 2022

See all articles by Silvia Saccardo

Silvia Saccardo

Carnegie Mellon University, Department of Social and Decision Sciences

Hengchen Dai

University of California, Los Angeles (UCLA) - Anderson School of Management

Maria Han

University of California, Los Angeles (UCLA) - UCLA Health System

Naveen Raja

University of California, Los Angeles (UCLA) - UCLA Health System

Sitaram Vangala

University of California, Los Angeles (UCLA) - Division of General Internal Medicine

Daniel Croymans

University of California, Los Angeles (UCLA) - UCLA Health System

Date Written: March 5, 2022

Abstract

Field experimentation and behavioral science have the potential to inform policy. Yet, many initially promising ideas show substantially lower efficacy at scale, reflecting the broader issue of the instability of scientific findings. Here, we identify two important factors that can explain variation in estimated intervention efficacy across evaluations and help policymakers better predict behavioral responses to interventions in their settings. To do so, we leverage data from (1) two randomized controlled trials (RCTs; N=187,134 and 149,720) that we conducted to nudge COVID-19 vaccinations, and (2) 111 nudge RCTs involving approximately 22 million people that were conducted by either academics or a government agency. Across those datasets, we find that nudges' estimated efficacy is higher when outcomes are more narrowly (vs. broadly) defined and measured over a shorter (vs. longer) horizon, which can partially explain why nudges evaluated by academics show substantially larger effect sizes than nudges evaluated at scale by the government agency. Further, we show that nudges' impact is smaller among individuals with low baseline motivation to act---a finding that is masked when only focusing on average effects. Altogether, we highlight that considering how intervention effectiveness is measured and who is nudged is critical to reconciling differences in effect sizes across evaluations and assessing the scalability of empirical findings.

Note:
Funding: Funding support for this research was provided by UCLA Health.

Declaration of Interests: The authors declare no competing interests. The authors did not receive financial or non-financial benefits from UCLA Health or speaking/consulting fees related to any of the interventions presented here.

Ethics Approval Statement: Our randomized control trials (RCTs) that promoted COVID-19 vaccine uptake were reviewed and approved by the Institutional Review Board (IRB) at the University of California, Los Angeles, which determined that a waiver of informed consent was appropriate.

Trial Registration: clinicaltrials.gov numbers: NCT04800965 and NCT04801524.

Keywords: Randomized controlled trials, Science policy, Nudge, Scalability, COVID-19, Vaccinations

JEL Classification: C93, I12

Suggested Citation

Saccardo, Silvia and Dai, Hengchen and Han, Maria and Raja, Naveen and Vangala, Sitaram and Croymans, Daniel, Assessing Nudge Scalability: Two Lessons from Large-scale RCTs (March 5, 2022). Available at SSRN: https://ssrn.com/abstract=3971192 or http://dx.doi.org/10.2139/ssrn.3971192

Silvia Saccardo

Carnegie Mellon University, Department of Social and Decision Sciences ( email )

Pittsburgh, PA 15213-3890
United States

Hengchen Dai (Contact Author)

University of California, Los Angeles (UCLA) - Anderson School of Management ( email )

110 Westwood Plaza
Los Angeles, CA 90095-1481
United States

Maria Han

University of California, Los Angeles (UCLA) - UCLA Health System ( email )

10833 Le Conte Avenue
17-165 CHS
Los Angeles, CA 90095-1730
United States

Naveen Raja

University of California, Los Angeles (UCLA) - UCLA Health System ( email )

10833 Le Conte Avenue
17-165 CHS
Los Angeles, CA 90095-1730
United States

Sitaram Vangala

University of California, Los Angeles (UCLA) - Division of General Internal Medicine ( email )

Los Angeles, CA
United States

Daniel Croymans

University of California, Los Angeles (UCLA) - UCLA Health System ( email )

10833 Le Conte Avenue
17-165 CHS
Los Angeles, CA 90095-1730
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
236
Abstract Views
1,102
rank
190,598
PlumX Metrics