AI-Driven Search A/B Testing: The Next Evolutionary Leap
Posted: 30 Jan 2020
Date Written: January 29, 2020
A/B testing is one of the most powerful techniques in the data science toolbox, enabling practitioners to attribute causality to individual product changes. This technique has enabled companies as diverse as Amazon, Facebook, and Google to grow into market-straddling colossi. However, A/B testing does have challenges, especially for smaller players in the market. One challenging problem is local maxima – never quite knowing if sequential A/B tests have reached the apogee of their potential, or if they are residing on a lower peak. The traditional answer to this challenge has been multi-variate testing (MVT) – armed with sampling and statistical heuristics to find the best peak by testing many variants at once. But multivariate testing has its own set of challenges due to combinatorial explosion in terms of the sheer number of variations that require testing. To solve the problem of local maxima in a scalable manner, we have developed a testing strategy we call “Neo-Darwinian testing”. Why does Neo-Darwinian testing represent a leap forward? By using genetic algorithms, we’ve found a way to intelligently look at a population of tests, and rapidly breed new generations of variants until the ultimate ‘genome’ wins the test – each generation shaped by the external pressures of real customer behaviours. The power of this technique enabled us to run an MVT test so large that it would normally take years to produce a statistically valid result, and get results in 30 days. We will discuss how we are enhancing the capabilities of our in-house A/B testing to conduct Neo-Darwinian search A/B testing and also share initial results of our first pilot.
Keywords: search testing, search engine, genetic algorithms, evolutionary computing, AB testing, multivariate testing, local maxima, machine learning, artificial intelligence
Suggested Citation: Suggested Citation