Sampling Bias in Entrepreneurial Experiments

99 Pages Posted: 30 Nov 2020 Last revised: 25 Sep 2022

See all articles by Sam (Ruiqing) Cao

Sam (Ruiqing) Cao

Stockholm School of Economics

Rembrand Koning

Harvard University - Business School (HBS)

Ramana Nanda

Imperial College Business School; Harvard University - Entrepreneurial Management Unit

Multiple version iconThere are 2 versions of this paper

Date Written: July 29, 2022

Abstract

Using data from a prominent online platform for launching new digital products, we document that the composition of the platform's `beta testers' on the day a new product is launched has a systematic and persistent impact on the venture's success. Specifically, we use word embedding methods to classify products on this platform as more or less focused on the needs of female customers. We then show that female-focused products launched on a typical day -- when nine in ten users on the platform are men -- experience 45% less growth in the year after launch. By isolating exogenous variation in the composition of beta testers unrelated to the characteristics of launched products on that day, we find that on days when there are unexpectedly more women, this gender-product gap shrinks towards zero. Further, consistent with a sampling bias mechanism, we find that the composition of beta testers appears to impact VC decision making and the entrepreneur's subsequent product development efforts. Overall, our findings suggest that the composition of early users can induce systematic biases in the signals of startup potential, with consequential effects -- including a shortage of innovations aimed at consumers who are underrepresented among early users.

Suggested Citation

Cao, Ruiqing and Koning, Rembrand and Nanda, Ramana, Sampling Bias in Entrepreneurial Experiments (July 29, 2022). Harvard Business School Entrepreneurial Management Working Paper No. 21-059, Harvard Business School Strategy Unit Working Paper No. 21-059, Available at SSRN: https://ssrn.com/abstract=3736119 or http://dx.doi.org/10.2139/ssrn.3736119

Ruiqing Cao

Stockholm School of Economics ( email )

PO Box 6501
Stockholm, 11383
Sweden

Rembrand Koning (Contact Author)

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Boston, MA 02163
United States

HOME PAGE: http://www.remkoning.com

Ramana Nanda

Imperial College Business School ( email )

South Kensington Campus
Exhibition Road
London SW7 2AZ, SW7 2AZ
United Kingdom

Harvard University - Entrepreneurial Management Unit ( email )

Boston, MA 02163
United States

HOME PAGE: http://www.people.hbs.edu/rnanda

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
274
Abstract Views
2,559
Rank
186,047
PlumX Metrics