Sampling Bias in Entrepreneurial Experiments

Abstract

Using data from a prominent online platform for launching new digital products, we document that the composition of the platform's `beta testers' on the day a new product is launched has a systematic and persistent impact on the venture's success. Specifically, we use word embedding methods to classify products on this platform as more or less focused on the needs of female customers. We then show that female-focused products launched on a typical day -- when nine in ten users on the platform are men -- experience 45% less growth in the year after launch. By isolating exogenous variation in the composition of beta testers unrelated to the characteristics of launched products on that day, we find that on days when there are unexpectedly more women, this gender-product gap shrinks towards zero. Further, consistent with a sampling bias mechanism, we find that the composition of beta testers appears to impact VC decision making and the entrepreneur's subsequent product development efforts. Overall, our findings suggest that the composition of early users can induce systematic biases in the signals of startup potential, with consequential effects -- including a shortage of innovations aimed at consumers who are underrepresented among early users.