Design and Evaluation of New Product Category Recommendations: Evidence from a Randomized Field Experiment
Posted: 27 Aug 2023 Last revised: 3 Oct 2023
Date Written: August 20, 2023
Abstract
Recommending new product categories to existing consumers (i.e., categories that they have not yet purchased) can be useful for increasing customer lifetime value as well as for reducing risks from category-specific supply shocks and category-specific competition. In this paper, we design category-introduction-oriented recommendation methods to increase customers' purchases from new product categories. We focus on application settings where the sales are highly concentrated, i.e., where the new category recommendation is particularly challenging. We use granular consumer journey data, employ comprehensive feature engineering and selection, and compare 15 recommendation models designed for new category introduction with robust offline evaluations. Then we estimate the causal economic impact of new category recommendations using a large-scale randomized controlled trial (RCT). We find that the new product category recommendation can increase the purchase probability by up to 35% compared with no recommendation. We also explore two dimensions, namely, (i) increasing the choice in recommended new categories and (ii) providing personalized (as opposed to non-personalized) recommendations. We find that increasing choices further increases the sales in the recommended categories by up to 9% as compared to recommending a single category, and personalized new category recommendation leads to 11% more purchases than recommending the most popular (non-personalized) new category. However, when recommending personalized new categories, more choices do not further increase sales as compared to recommending only one category. Finally, we go beyond standard average treatment effect analysis to discover customer heterogeneity. We find that the most recent visitors (who visit the platform within last a couple of days before the new category recommendation) are most responsive to multiple choices. In contrast, personalizing recommendations is more effective for not-so-recent customers, who visit the platform within three months before the treatment. A conditional average treatment effect treatment policy, which deploys the best treatment for different user segments, shows favorable lift in profit.
Keywords: new category recommendation, machine learning, randomized field experiment, causal forests
Suggested Citation: Suggested Citation