AI-Driven Search A/B Testing: The Next Evolutionary Leap

Posted: 30 Jan 2020

Date Written: January 29, 2020

Abstract

A/B testing is one of the most powerful techniques in the data science toolbox, enabling practitioners to attribute causality to individual product changes. This technique has enabled companies as diverse as Amazon, Facebook, and Google to grow into market-straddling colossi. However, A/B testing does have challenges, especially for smaller players in the market. One challenging problem is local maxima – never quite knowing if sequential A/B tests have reached the apogee of their potential, or if they are residing on a lower peak. The traditional answer to this challenge has been multi-variate testing (MVT) – armed with sampling and statistical heuristics to find the best peak by testing many variants at once. But multivariate testing has its own set of challenges due to combinatorial explosion in terms of the sheer number of variations that require testing. To solve the problem of local maxima in a scalable manner, we have developed a testing strategy we call “Neo-Darwinian testing”. Why does Neo-Darwinian testing represent a leap forward? By using genetic algorithms, we’ve found a way to intelligently look at a population of tests, and rapidly breed new generations of variants until the ultimate ‘genome’ wins the test – each generation shaped by the external pressures of real customer behaviours. The power of this technique enabled us to run an MVT test so large that it would normally take years to produce a statistically valid result, and get results in 30 days. We will discuss how we are enhancing the capabilities of our in-house A/B testing to conduct Neo-Darwinian search A/B testing and also share initial results of our first pilot.

Keywords: search testing, search engine, genetic algorithms, evolutionary computing, AB testing, multivariate testing, local maxima, machine learning, artificial intelligence

Suggested Citation

Khamisani, Vaqar and Rubinstein, James and Costantino, Mark and Corney, Steve, AI-Driven Search A/B Testing: The Next Evolutionary Leap (January 29, 2020). Proceedings of the 3rd Annual RELX Search Summit, Available at SSRN: https://ssrn.com/abstract=3527356

Vaqar Khamisani (Contact Author)

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

James Rubinstein

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Mark Costantino

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Steve Corney

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
219
PlumX Metrics