Using Contextual Embeddings to Predict the Effectiveness of Novel Heterogeneous Treatments

58 Pages Posted: 30 May 2024

See all articles by Paul B. Ellickson

Paul B. Ellickson

University of Rochester - Simon Business School

Wreetabrata Kar

Purdue University

James C. Reeder, III

University of Kansas - School of Business

Guang Zeng

Simon Business School, University of Rochester

Date Written: May 28, 2024

Abstract

Our study demonstrates the power of contextual embeddings for predicting the performance of novel heterogeneous treatments. Our proposed framework leverages four key benefits of machine learning: prediction, enhanced causal estimation, estimation of heterogeneous effects, and generative capability. To test our framework, we exploit a targeted marketing setting in which 34 email promotions were sent to 1.3 million customers over a 45-day period. Using these emails as treatments, we start by estimating the doubly robust scores of customer-level purchase amounts to serve as our target variable. We incorporate customer-level demographics and contextual embeddings, which capture the context of the latent states, to estimate the response function of these emails. Using a series of leave-one-out exercises, we show how our approach can accurately extrapolate the average performance, heterogeneous performance, and recommended targeting policies of novel promotions. We find that our framework recovers 78.6% of the variation of the aggregate treatment effects, an average of 65.36% of the variation in the heterogeneous treatment effects of each novel treatment, and matches 82% of policy recommendations made using the true signals. We conclude our study with an example of leveraging Generative AI to create novel treatments and then evaluate their performance with our framework.

Keywords: Contextual Embeddings, Digital Marketing, Double Machine Learning, Multi-Treatment Heterogeneity, Large Language Models, ChatGPT.

Suggested Citation

Ellickson, Paul B. and Kar, Wreetabrata and Reeder, III, James C. and Zeng, Guang, Using Contextual Embeddings to Predict the Effectiveness of Novel Heterogeneous Treatments (May 28, 2024). Available at SSRN: https://ssrn.com/abstract=4845956 or http://dx.doi.org/10.2139/ssrn.4845956

Paul B. Ellickson

University of Rochester - Simon Business School ( email )

Rochester, NY 14627
United States

Wreetabrata Kar

Purdue University ( email )

James C. Reeder, III (Contact Author)

University of Kansas - School of Business ( email )

1300 Sunnyside Avenue
Lawrence, KS 66045
United States

Guang Zeng

Simon Business School, University of Rochester ( email )

Rochester, NY 14627
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
45
Abstract Views
257
PlumX Metrics