Exploration in Action: The Role of Randomized Control Trials in Online Demand Generation

37 Pages Posted: 22 Mar 2021

See all articles by Julian Runge

Julian Runge

Duke University (Visiting Scholar)

Harikesh Nair

Stanford University - Graduate School of Business

Date Written: February 27, 2021

Abstract

Randomized control trials (RCTs) are widely lauded as the “gold standard” for inference, providing unbiased estimates of the causal effect of firms’ policies and allowing for better decisions, e.g., in product development, pricing, promotion and advertising. The main thrust of this paper is to ask: Is the hype justified? To collect evidence towards answering this question, the authors study Facebook’s organically evolving advertising market and compare demand outcomes for firms that embrace RCTs for policy evaluation with those of firms that do not. Results show that RCT adopters appropriate substantially more demand at equal or higher levels of efficiency than non-adopters, providing indicative evidence that there are strong performance returns to the adoption of RCTs for policy evaluation in online settings. To the extent that the cost of conducting RCTs is not prohibitively high and that policy makers carefully extrapolate findings from RCTs, this finding may generalize.

Keywords: Firm experimentation, randomized control trials, firm learning, online advertising, digital commerce

JEL Classification: M31, M37

Suggested Citation

Runge, Julian and Nair, Harikesh, Exploration in Action: The Role of Randomized Control Trials in Online Demand Generation (February 27, 2021). Stanford University Graduate School of Business Research Paper, Available at SSRN: https://ssrn.com/abstract=3794028 or http://dx.doi.org/10.2139/ssrn.3794028

Julian Runge (Contact Author)

Duke University (Visiting Scholar) ( email )

Box 90120
Durham, NC 27708-0120
United States

Harikesh Nair

Stanford University - Graduate School of Business ( email )

655 Knight Way
Stanford, CA 94305-5015
United States
650-736-4256 (Phone)

HOME PAGE: http://faculty-gsb.stanford.edu/nair/index.html

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
48
Abstract Views
290
PlumX Metrics