Estimating Effects of Long-Term Treatments

45 Pages Posted: 13 Feb 2023 Last revised: 18 Feb 2024

See all articles by Shan Huang

Shan Huang

The University of Hong Kong

Chen Wang

The University of Hong Kong, Faculty of Business and Economics, Students

Yuan Yuan

Mitchell E. Daniels, Jr School of Business, Purdue University

Jinglong Zhao

Boston University - Questrom School of Business

Jingjing Zhang

Tencent Inc.

Date Written: February 9, 2023

Abstract

Estimating the effects of long-term treatments in A/B testing presents a significant challenge. Such treatments --- including updates to product functions, user interface designs, and recommendation algorithms --- are intended to remain in the system for a long period after their launches. On the other hand, given the constraints of conducting long-term experiments, practitioners often rely on short-term experimental results to make product launch decisions. It remains an open question how to accurately estimate the effects of long-term treatments using short-term experimental data. To address this question, we introduce a longitudinal surrogate framework. We show that, under standard assumptions, the effects of long-term treatments can be decomposed into a series of functions, which depend on the user attributes, the short-term intermediate metrics, and the treatment assignments. We describe the identification assumptions, the estimation strategies, and the inference technique under this framework. Empirically, we show that our approach outperforms existing solutions by leveraging two real-world experiments, each involving millions of users on WeChat, one of the world's largest social networking platforms.

Keywords: A/B testing, long-term treatments, surrogates, causal inference, product management

Suggested Citation

Huang, Shan and Wang, Chen and Yuan, Yuan and Zhao, Jinglong and Zhang, Jingjing, Estimating Effects of Long-Term Treatments (February 9, 2023). Available at SSRN: https://ssrn.com/abstract=4352459 or http://dx.doi.org/10.2139/ssrn.4352459

Shan Huang

The University of Hong Kong ( email )

Pokfulam Road
Hong Kong
China

Chen Wang (Contact Author)

The University of Hong Kong, Faculty of Business and Economics, Students ( email )

Pokfulam Road
Hong Kong
Hong Kong

Yuan Yuan

Mitchell E. Daniels, Jr School of Business, Purdue University ( email )

403 Mitch Daniels Blvd.
West Lafayette, IN 47907
United States

Jinglong Zhao

Boston University - Questrom School of Business ( email )

595 Commonwealth Avenue
Boston, MA MA 02215
United States

Jingjing Zhang

Tencent Inc. ( email )

Shenzhen
China

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
393
Abstract Views
1,472
Rank
138,802
PlumX Metrics