Digital experimentation and startup performance: Evidence from A/B testing

Abstract

Recent work argues that experimentation is the appropriate framework for entrepreneurial strategy. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of experimentally testing business ideas. This paper provides the first evidence on how digital experimentation affects a large sample of high-technology startups using data that tracks their growth, technology use, and product launches. Despite its prominence in the business press, relatively few firms have adopted A/B testing tools. However, among those that do, we find increased performance on several critical dimensions, including page views, code changes, and new product features. These results are robust to our use of instrumental variables and synthetic control models, where we leverage Google’s launch of a new A/B testing tool. Our results also show that the value of A/B testing increases over time and is positively related to tail outcomes, with adopt- ing firms both scaling and failing faster. Our results inform the emerging literature on entrepreneurial strategy and illustrate the growing importance of digitization and data-driven decision-making.

Keywords: Entrepreneurial Strategy, Startups, A/B Testing, Data-Driven Decision-Making, Experimentation