Experimentation and startup performance: Evidence from A/B testing
78 Pages Posted: 23 Aug 2019 Last revised: 18 May 2021
There are 2 versions of this paper
Experimentation and startup performance: Evidence from A/B testing
Experimentation and Startup Performance: Evidence from A/B Testing
Date Written: August 20, 2019
Abstract
Recent scholarship has argued that experimentation should be the organizing prin- ciple for entrepreneurial strategy. Experimentation leads to organizational learning, which drives improvements in firm performance. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of testing business ideas. Our results provide the first evidence on how digital experimentation affects a large sample of high-technology startups using data that tracks their growth, technology use, and products. While relatively few firms adopt A/B testing, among those that do, we find increased performance on several critical dimensions, including page views and product launches. These results are robust to the use of instrumental variables and synthetic control models, where we leverage Google’s launch of a new A/B testing tool. Our analysis suggests that firms are using A/B tests to experiment with significant changes, not merely incremental improvements. Consistent with these results, we also find that A/B testing is positively related to tail outcomes, with adopting firms both scaling and failing faster. These findings inform the literatures on entrepreneurial strategy, organizational learning, and data-driven decision-making.
Keywords: Entrepreneurial Strategy, Startups, A/B Testing, Data-Driven Decision-Making, Experimentation
Suggested Citation: Suggested Citation