A/B Testing at Vungle

7 Pages Posted: 18 Jul 2018

See all articles by Yael Grushka-Cockayne

Yael Grushka-Cockayne

University of Virginia - Darden School of Business; Harvard University - Business School (HBS)

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business

Bert De Reyck

University of Virginia - Darden School of Business

Ioannis Fragkos

University College London

Abstract

Two recently graduated MBA students are tasked with developing an ad-serving learning algorithm for a mobile ad-serving company. The case illustrates the way in which hypotheses can be tested in an A/B format or "horse race" in order to establish customer preferences and superior profitability. The case was written for a course elective covering hypothesis testing.

Excerpt

UVA-QA-0821

Rev. Mar. 7, 2017

A/B Testing at Vungle

Andrew Kritzer and Hammond Guerin stared at the screen and then at each other. It was June 30, 2014—six weeks since they had graduated from the Darden School of Business. The ad-serving algorithm Kritzer and Guerin had spent six months developing for Vungle, a mobile advertising company, seemed to be outperforming the company's current algorithm. But they did not want to start celebrating too soon. Could their algorithm really deliver the type of improvement they had promised Vungle's CEO? Would install rates of advertised apps really increase? Would Vungle see an increase in ad-serving efficiency as a result?

Neither Kritzer nor Guerin could afford for the algorithm to disappoint. Now that he had graduated, Kritzer was headed to LinkedIn, having left a legend among MBA students for his appreciation of data science, tech, and media and raising expectations for what Darden students knew and could learn about data science, analytics, and the ever-growing world of big data. His work on the Vungle project during his second year had received a lot of attention, and he was looking forward to having the results support the effort.

Guerin's data science capabilities were also legendary among his MBA peers. He won every school forecasting competition, and his data mining algorithms even beat those of the professional consultants who did classroom visits. Late in his second year, Guerin decided to turn down a generous offer from a well-known consulting firm in favor of an offer from Vungle for an annual salary of $ 100,000 and stock options to serve as the head of Vungle's brand new data science team out in the company's San Francisco headquarters. The job was a dream for the computer scientist turned MBA. He and his wife were already house hunting in the Bay Area, looking for the right place to raise their baby daughter.

. . .

Keywords: hypothesis testing, A/B test, horse race, paired t-test, parallel run, algorithm, eRPM, conversion rate

Suggested Citation

Grushka-Cockayne, Yael and Lichtendahl, Kenneth C. and De Reyck, Bert and Fragkos, Ioannis, A/B Testing at Vungle. Darden Case No. UVA-QA-0821. Available at SSRN: https://ssrn.com/abstract=3213752

Yael Grushka-Cockayne (Contact Author)

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Harvard University - Business School (HBS) ( email )

Soldiers Field Road
Morgan 270C
Boston, MA 02163
United States

HOME PAGE: http://https://www.hbs.edu/faculty/Pages/profile.aspx?facId=263650

Kenneth C. Lichtendahl

University of Virginia - Darden School of Business ( email )

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Bert De Reyck

University of Virginia - Darden School of Business

P.O. Box 6550
Charlottesville, VA 22906-6550
United States

Ioannis Fragkos

University College London

Gower Street
London, WC1E 6BT
United Kingdom

Register to save articles to
your library

Register

Paper statistics

Downloads
4
Abstract Views
228
PlumX Metrics