A Bias Correction Approach for Interference in Ranking Experiments

50 Pages Posted: 30 Jan 2022 Last revised: 15 Mar 2022

See all articles by Ali Goli

Ali Goli

University of Washington - Michael G. Foster School of Business

Anja Lambrecht

London Business School

Hema Yoganarasimhan

University of Washington

Date Written: January 29, 2022

Abstract

Online marketplaces use ranking algorithms to determine the rank-ordering of items sold on their websites. The standard practice is to determine the optimal algorithm using A/B tests. We present a theoretical framework to characterize the Total Average Treatment Effect (TATE) of a ranking algorithm in an A/B test and show that naive TATE estimates can be biased due to interference. We propose a bias-correction approach that can recover the TATE of a ranking algorithm based on past A/B tests, even if those tests suffer from a combination of interference issues. Our solution leverages data across multiple experiments and identifies observations in partial equilibrium in each experiment, i.e., items close to their positions under the true counterfactual equilibrium of interest. We apply our framework to data from a travel website and present comprehensive evidence for interference bias in this setting. Next, we use our solution concept to build a customized deep learning model to predict the true TATE of the main algorithm of interest in our data. Counterfactual estimates from our model show that naive TATE estimates of clicks and bookings can be biased by as much as 15% and 28%, respectively.

Keywords: Experiments, A/B tests, Treatment Effects, Digital platforms, Interference, Machine Learning, Bias Correction

JEL Classification: C93, M31

Suggested Citation

Goli, Ali and Lambrecht, Anja and Yoganarasimhan, Hema, A Bias Correction Approach for Interference in Ranking Experiments (January 29, 2022). Available at SSRN: https://ssrn.com/abstract=4021266 or http://dx.doi.org/10.2139/ssrn.4021266

Ali Goli

University of Washington - Michael G. Foster School of Business ( email )

Box 353200
Seattle, WA 98195-3200
United States

Anja Lambrecht

London Business School ( email )

Regent's Park
London, NW1 4SA
United Kingdom

Hema Yoganarasimhan (Contact Author)

University of Washington ( email )

481 Paccar Hall
Seattle, WA 98195
United States

HOME PAGE: http://faculty.washington.edu/hemay/

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
60
Abstract Views
207
rank
477,969
PlumX Metrics