Exploration in Action: The Role of Randomized Control Trials in Online Demand Generation
37 Pages Posted: 22 Mar 2021
Date Written: February 27, 2021
Randomized control trials (RCTs) are widely lauded as the “gold standard” for inference, providing unbiased estimates of the causal effect of firms’ policies and allowing for better decisions, e.g., in product development, pricing, promotion and advertising. The main thrust of this paper is to ask: Is the hype justified? To collect evidence towards answering this question, the authors study Facebook’s organically evolving advertising market and compare demand outcomes for firms that embrace RCTs for policy evaluation with those of firms that do not. Results show that RCT adopters appropriate substantially more demand at equal or higher levels of efficiency than non-adopters, providing indicative evidence that there are strong performance returns to the adoption of RCTs for policy evaluation in online settings. To the extent that the cost of conducting RCTs is not prohibitively high and that policy makers carefully extrapolate findings from RCTs, this finding may generalize.
Keywords: Firm experimentation, randomized control trials, firm learning, online advertising, digital commerce
JEL Classification: M31, M37
Suggested Citation: Suggested Citation