The Politics of Experimentation

Abstract

Many startups have embraced formal experimentation — i.e., A/B testing — to improve the performance of their products and services. Experimentation, some have argued, should democratize innovation inside organizations by creating a platform to rigorously test new ideas, regardless of origin. In this article, we argue that experimentation's promise hinges on having the right organizational decision-making process — one that encourages innovation but without the added risk of unanticipated failures. We study this question by developing a model of experimentation inside organizations where decisions to implement are either centralized or decentralized — a tension identified by practitioners and scholars alike. We find that startups with centralized mechanisms that rely too much on the input of other teams benefit least from experimentation, as do ones with completely decentralized ones. In contrast, startups with mostly decentralized decisions, with a single authority that sets consistent thresholds for implementation, achieve growth but with less downside risk. Thus, without considering the politics that guide organizational decision-making, the benefits of experimentation may be limited.

Keywords: Entrepreneurial Strategy, Startups, A/B Testing, Data-Driven Decision-Making, Experimentation