Designing Controlled Experiments with a Small Sample and Strategic Agents

Abstract

Problem definition: Firms with chain outlets have strong needs for controlled experiments in which they test new outlet layouts, new menu designs, and so on. Three major challenges make it difficult for them to obtain accurate experiment results: outlet heterogeneity, a relatively small set of outlets, and a high experiment cost. We propose to invite the outlet manager in the treatment group to provide an estimate of the counterfactual outlet performance, which is then used by the firm to improve the estimation of the treatment effect. Given the manager's self-interest and higher cost of implementing the change compared to the firm, we study the manager's strategic behavior and the optimal experiment design. Methodology/results: Using a two-period, stylized game-theoretical model, wherein only one chain outlet is selected as the treatment group and the manager tries to bias the firm's estimation by lowering sales or service effort level (i.e., moral hazard) and overstating the counterfactual outlet performance (i.e., cheap talk), we derive two major findings. (i) Inviting the manager's assistance in estimating the counterfactual outlet performance allows the firm to improve estimation quality in most cases. (ii) In some case, running the experiment with a unique outlet that has no similar peers and relying solely on the manager's counterfactual estimate can lead to a better estimation of the treatment effect—even if there are outlets with high quality control groups. Managerial implications: First, it is preferable for firms to get the manager's help in the estimating process when the degree of incentive misalignment (i.e., the manager's personal cost associated with implementing the change) is either very small or very high. Second, when designing the experiment, firms should consider whether to make a control group available (by carefully selecting the treatment group).

Keywords: A/B Testing, Small Sample, Experiment Design, Moral Hazard, Cheap Talk