Experimental Design in Live-Interaction Platforms

Abstract

We investigate experimental design problems in live-interaction platforms, which are commonly found in the online gaming industry and anonymous social networks. These platforms involve users being matched with others for activities such as games or social interactions, resulting in an interdependence of users' metrics on the treatment assignments of their counterparts. We construct a stochastic market model and develop its mean field limit to analyze the experimental dynamics in such platforms. Our focus is on two randomization strategies: user randomization and match randomization. We demonstrate that under Markovian user behavior and homogeneous treatment effects, match randomization can provide unbiased estimations. However, significant biases may arise when these conditions are not met. On the other hand, user randomization is generally biased but exhibits greater resilience to model inaccuracies. We then propose a simple linear regression estimator under user randomization and demonstrate that this estimator consistently outperforms alternatives in various situations.

Keywords: design of experiments, live-interaction platforms, gaming platforms, anonymous social networks, user engagement