Adaptive Data Acquisition for Personalized Recommender Systems with Optimality Guarantees on Short-Form Video Platforms

Abstract

The popularity of short-form video (SFV) has been exploding on digital platforms recently. The large number of videos and evolving topics on digital platforms pose technical challenges in making personalized recommendations. In this paper, we introduce a new pure exploration problem on SFV platforms for finding a $(K, \epsilon_H, \epsilon_L)$-optimal set comprising all recommendations that are within $\epsilon_L$-optimality gap and excluding all recommendations outside of the $\epsilon_H$-optimality gap, relative to the best arm with capacity at most K, for quality control. We propose an adaptive data acquisition method, called Adaptive Acquisition Tree (AAT), to jointly account for user preference heterogeneity and high-dimensional product characteristics. AAT adaptively segments users based on preference similarity and then learns a personalized transductive bandit policy that can be used on partially observed or even unobserved items to accommodate the dynamic trends on SFV platforms. We derive the sample complexity for identifying a $(K, \epsilon_H, \epsilon_L)$-optimal set for a single user and for all users, respectively. We further evaluate the algorithm via numerical experiments on data collected from the NetEase platform. Our result demonstrates that the proposed policy performs significantly better than several state-of-the-art benchmarks in four transductive scenarios for both spotlight recommendations (i.e., best-arm identifications) and $(K, \epsilon_H, \epsilon_L)$-optimal set recommendations. With the potential to improve the expected view time by 20-25%, our method produces both academic and practical value, given the increasing popularity and uniqueness of SFVs and, more broadly, user-generated content.

Keywords: short-form video, pure exploration, information acquisition, transductive learning, contextual bandits; interpretability; short-form video platforms; recommender systems