Design and Evaluation of New Product Category Recommendations: Evidence from a Randomized Field Experiment

Posted: 27 Aug 2023 Last revised: 3 Oct 2023

See all articles by Meizi Zhou

Meizi Zhou

Boston University - Questrom School of Business

Ravi Bapna

University of Minnesota - Minneapolis

Gediminas Adomavicius

University of Minnesota - Twin Cities - Carlson School of Management

Jonathan Hershaff

University of Michigan at Ann Arbor

Date Written: August 20, 2023

Abstract

Recommending new product categories to existing consumers (i.e., categories that they have not yet purchased) can be useful for increasing customer lifetime value as well as for reducing risks from category-specific supply shocks and category-specific competition. In this paper, we design category-introduction-oriented recommendation methods to increase customers' purchases from new product categories. We focus on application settings where the sales are highly concentrated, i.e., where the new category recommendation is particularly challenging. We use granular consumer journey data, employ comprehensive feature engineering and selection, and compare 15 recommendation models designed for new category introduction with robust offline evaluations. Then we estimate the causal economic impact of new category recommendations using a large-scale randomized controlled trial (RCT). We find that the new product category recommendation can increase the purchase probability by up to 35% compared with no recommendation. We also explore two dimensions, namely, (i) increasing the choice in recommended new categories and (ii) providing personalized (as opposed to non-personalized) recommendations. We find that increasing choices further increases the sales in the recommended categories by up to 9% as compared to recommending a single category, and personalized new category recommendation leads to 11% more purchases than recommending the most popular (non-personalized) new category. However, when recommending personalized new categories, more choices do not further increase sales as compared to recommending only one category. Finally, we go beyond standard average treatment effect analysis to discover customer heterogeneity. We find that the most recent visitors (who visit the platform within last a couple of days before the new category recommendation) are most responsive to multiple choices. In contrast, personalizing recommendations is more effective for not-so-recent customers, who visit the platform within three months before the treatment. A conditional average treatment effect treatment policy, which deploys the best treatment for different user segments, shows favorable lift in profit.

Keywords: new category recommendation, machine learning, randomized field experiment, causal forests

Suggested Citation

Zhou, Meizi and Bapna, Ravi and Adomavicius, Gediminas and Hershaff, Jonathan, Design and Evaluation of New Product Category Recommendations: Evidence from a Randomized Field Experiment (August 20, 2023). Available at SSRN: https://ssrn.com/abstract=4546277 or http://dx.doi.org/10.2139/ssrn.4546277

Meizi Zhou (Contact Author)

Boston University - Questrom School of Business ( email )

595 Commonwealth Avenue
Boston, MA MA 02215
United States

Ravi Bapna

University of Minnesota - Minneapolis ( email )

321 19th Ave S
Information and Decision Sciences
Minneapolis, MN 55455
United States

Gediminas Adomavicius

University of Minnesota - Twin Cities - Carlson School of Management ( email )

19th Avenue South
Minneapolis, MN 55455
United States

Jonathan Hershaff

University of Michigan at Ann Arbor ( email )

Ann Arbor, MI
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
422
PlumX Metrics