Efficiently Evaluating Targeting Policies: Improving Upon Champion vs. Challenger Experiments

26 Pages Posted: 12 Aug 2017 Last revised: 15 May 2019

See all articles by Duncan Simester

Duncan Simester

Massachusetts Institute of Technology (MIT) - Sloan School of Management

Artem Timoshenko

Kellogg School of Management, Northwestern University

Spyros Zoumpoulis

INSEAD - Decision Sciences

Date Written: April 8, 2019

Abstract

Champion versus challenger field experiments are widely used to compare the performance of different targeting policies. These experiments randomly assign customers to receive marketing actions recommended by either the existing (champion) policy or the new (challenger) policy, and then compare the aggregate outcomes. We recommend an alternative experimental design and propose an alternative estimation approach to improve the evaluation of targeting policies.

The recommended experimental design randomly assigns customers to marketing actions. This allows evaluation of any targeting policy without requiring an additional experiment, including policies designed after the experiment is implemented. The proposed estimation approach identifies customers for whom different policies recommend the same action and recognizes that for these customers there is no difference in performance. This allows for a more precise comparison of the policies.

We illustrate the advantages of the experimental design and estimation approach using data from an actual field experiment. We also demonstrate that the grouping of customers, which is the foundation of our estimation approach, can help to improve the training of new targeting policies.

Keywords: targeting, field experiments, counterfactual policy logging, standard errors

JEL Classification: C12, C93

Suggested Citation

Simester, Duncan and Timoshenko, Artem and Zoumpoulis, Spyros, Efficiently Evaluating Targeting Policies: Improving Upon Champion vs. Challenger Experiments (April 8, 2019). INSEAD Working Paper No. 2017/63/DSC, Available at SSRN: https://ssrn.com/abstract=3017384 or http://dx.doi.org/10.2139/ssrn.3017384

Duncan Simester (Contact Author)

Massachusetts Institute of Technology (MIT) - Sloan School of Management ( email )

Management Science
Cambridge, MA 02142
United States
617-258-0679 (Phone)
617-258-7597 (Fax)

Artem Timoshenko

Kellogg School of Management, Northwestern University ( email )

2001 Sheridan Road
Evanston, IL 60208
United States

Spyros Zoumpoulis

INSEAD - Decision Sciences ( email )

France

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
250
Abstract Views
1,303
rank
143,334
PlumX Metrics