Experimentation and startup performance: Evidence from A/B testing

78 Pages Posted: 23 Aug 2019 Last revised: 18 May 2021

See all articles by Rembrand Koning

Rembrand Koning

Harvard University - Business School (HBS)

Sharique Hasan

Duke University - Fuqua School of Business

Aaron Chatterji

Duke University - Fuqua School of Business

Multiple version iconThere are 2 versions of this paper

Date Written: August 20, 2019

Abstract

Recent scholarship has argued that experimentation should be the organizing prin- ciple for entrepreneurial strategy. Experimentation leads to organizational learning, which drives improvements in firm performance. We investigate this proposition by exploiting the time-varying adoption of A/B testing technology, which has drastically reduced the cost of testing business ideas. Our results provide the first evidence on how digital experimentation affects a large sample of high-technology startups using data that tracks their growth, technology use, and products. While relatively few firms adopt A/B testing, among those that do, we find increased performance on several critical dimensions, including page views and product launches. These results are robust to the use of instrumental variables and synthetic control models, where we leverage Google’s launch of a new A/B testing tool. Our analysis suggests that firms are using A/B tests to experiment with significant changes, not merely incremental improvements. Consistent with these results, we also find that A/B testing is positively related to tail outcomes, with adopting firms both scaling and failing faster. These findings inform the literatures on entrepreneurial strategy, organizational learning, and data-driven decision-making.

Keywords: Entrepreneurial Strategy, Startups, A/B Testing, Data-Driven Decision-Making, Experimentation

Suggested Citation

Koning, Rembrand and Hasan, Sharique and Chatterji, Aaron, Experimentation and startup performance: Evidence from A/B testing (August 20, 2019). Available at SSRN: https://ssrn.com/abstract=3440291 or http://dx.doi.org/10.2139/ssrn.3440291

Rembrand Koning (Contact Author)

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Boston, MA 02163
United States

Sharique Hasan

Duke University - Fuqua School of Business ( email )

Box 90120
Durham, NC 27708-0120
United States

Aaron Chatterji

Duke University - Fuqua School of Business ( email )

Box 90120
Durham, NC 27708-0120
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
322
Abstract Views
2,030
rank
111,668
PlumX Metrics