The Role of Randomized Control Trials in Online Demand Generation: Exploratory Evidence from Facebook

Abstract

Observing the organically evolving Facebook advertising marketplace, this paper sets out to study how the adoption of randomized control trials (RCTs) for policy evaluation impacts firms’ behavior and performance on the platform. Findings indicate strong positive effects from the adoption of RCTs: After or in coincidence with RCT adoption, firms increase their investment in demand generation, at stable or higher levels of performance, as measured by customer purchases achieved per advertising dollar spent. A human expert provided to firms by the platform is also able to induce increases in investment but smaller ones, and, importantly, performance gains appear to exclusively materialize after the adoption of RCTs for policy evaluation, and not through human expertise alone. By means of the unbiased causal estimates obtained through RCTs, firms may be able to uniquely reduce their uncertainty about the environment and identify effective policies to exploit it with certainty.

Keywords: Experimentation, firm learning, online advertising, randomized control trials, exploration-exploitation

JEL Classification: M31, M37