Adaptive Data Acquisition for Personalized Recommender Systems with Optimality Guarantees on Short-Form Video Platforms

Abstract

The recent surge in the popularity of short-form video (SFV) on digital platforms has led to massive numbers of videos and ever-evolving topics. As a result, the task of making personalized recommendations has become increasingly challenging. We introduce a new pure exploration problem on SFV platforms: finding a $(K, \epsilon^H, \epsilon^L)$-optimal set that includes all recommendations within the $\epsilon^L$-optimality gap and that excludes those beyond the $\epsilon^H$-optimality gap, relative to the best arm with a capacity limit of K. To solve this problem, we propose an algorithm called Adaptive Acquisition Tree (AAT). AAT jointly accounts for user preference heterogeneity and for high-dimensional product characteristics. It adaptively segments users and then learns a personalized transductive policy that can be used on partially observed or even unobserved card types to accommodate the dynamic trends on SFV platforms. We derive the sample complexity required to identify a $(K, \epsilon^H, \epsilon^L)$-optimal set. Our method's efficiency is validated through numerical tests using data from the NetEase platform. Our results reveal that the proposed policy performs significantly better than several state-of-the-art benchmarks across four transductive scenarios for both spotlight recommendations (i.e., best-arm identifications) and $(K, \epsilon^H, \epsilon^L)$-optimal set recommendations. Compared to the best benchmarks for the best card and $(K, \epsilon^H, \epsilon^L)$-optimal set recommendations, our approach can elevate the average reward (measured by view time) by 30% (to 100%) and 43% (to 56%), respectively. Given the increasing popularity and uniqueness of SFVs and, more broadly, user-generated content (UGC), our method offers significant academic and practical merit.

Keywords: pure exploration, information acquisition, transductive learning, short-form video platforms, recommender systems