Using Algorithmic Scores to Measure the Impacts of Targeting Promotional Messages

Abstract

Nowadays, managers' targeting promotion decisions are often facilitated by machine learning (ML) algorithms. These algorithms often aggregate a large amount of customer demographic and behavioral information into several personalized scores, based on which managers then algorithmically make targeting or pricing decisions. We propose matching on these ML-generated scores used in targeting decisions to measure the effectiveness of targeting promotions. Our strategy can effectively address the issue of selection bias introduced by personalized targeting, and it could overcome the curse of dimensionality in matching when these ML systems use a large amount of consumer information. In order to test our proposed approach, we conducted a large field experiment on targeting promotions with a large retailing platform and analyzed customer behaviors. In our empirical application, customers are divided into two groups: the experimental group, where customers are randomly assigned to the treatment and control for receiving the targeting promotion, and the non-experimental (i.e., observational) group, where the algorithm will decide who receives the promotion. We first demonstrate that receiving the targeting promotions successfully increases store visits and purchases but also increases customers' likelihood of unsubscribing from the promotional message service. We then compare the results from experimental data and those by our matching approach using the non-experimental data. We showed that our proposed matching approach could effectively recover the same effects as the experimental data. By contrast, the estimated effects without matching are seriously over-biased, and traditional matching methods cannot mitigate such biases due to the curse of dimensionality.