LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments

48 Pages Posted: 5 Jun 2024

See all articles by Zikun Ye

Zikun Ye

University of Washington - Michael G. Foster School of Business

Hema Yoganarasimhan

University of Washington

Yufeng Zheng

University of Toronto, Toronto, Canada

Date Written: June 02, 2024

Abstract

In the rapidly evolving digital content landscape, media firms and news publishers require automated and efficient methods to enhance user engagement. This paper introduces the LLM-Assisted Online Learning Algorithm (LOLA), a novel framework that integrates Large Language Models (LLMs) with adaptive experimentation to optimize content delivery. Leveraging a large-scale dataset from Upworthy, which includes 17,681 headline A/B tests aimed at evaluating the performance of various headlines associated with the same article content, we first investigate three broad pure-LLM approaches: prompt-based methods, embedding-based classification models, and fine-tuned open-source LLMs. Our findings indicate that prompt-based approaches perform poorly, achieving no more than 65% accuracy in identifying the catchier headline among two options. In contrast, OpenAI-embedding-based classification models and fine-tuned Llama-3-8b models achieve comparable accuracy, around 82-84%, though still falling short of the performance of experimentation with sufficient traffic. We then introduce LOLA, which combines the best pure-LLM approach with the Upper Confidence Bound algorithm to adaptively allocate traffic and maximize clicks. Our numerical experiments on Upworthy data show that LOLA outperforms the standard A/B testing method (the current status quo at Upworthy), pure bandit algorithms, and pure-LLM approaches, particularly in scenarios with limited experimental traffic or numerous arms. Our approach is both scalable and broadly applicable to content experiments across a variety of digital settings where firms seek to optimize user engagement, including digital advertising and social media recommendations.

Keywords: LLMs, Content Experiments, Bandits, News

Suggested Citation

Ye, Zikun and Yoganarasimhan, Hema and Zheng, Yufeng, LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments (June 02, 2024). Available at SSRN: https://ssrn.com/abstract=4851778 or http://dx.doi.org/10.2139/ssrn.4851778

Zikun Ye (Contact Author)

University of Washington - Michael G. Foster School of Business ( email )

Seattle, WA 98195
United States

Hema Yoganarasimhan

University of Washington ( email )

481 Paccar Hall
Seattle, WA 98195
United States

HOME PAGE: http://faculty.washington.edu/hemay/

Yufeng Zheng

University of Toronto, Toronto, Canada ( email )

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
91
Abstract Views
254
Rank
535,142
PlumX Metrics