Automated Regression Testing of Search using Search Test Framework

Posted: 15 Nov 2021

Date Written: September 21, 2021

Abstract

Historically, chronologic baseline tracking of relevance precision was used as a metric to show search algorithm improvement or regression. Generating this data, based on consistent metrics and statistically significant data sets, has become a significant strategic advantage to LexisNexis by allowing data-driven improvement and decision making, and by tracking competitive relevance improvement over time.

Search Test Framework (STF) is the internal tool developed to support testing and analysis of search relevance precision and recall results based on various methods and metrics. STF Release 4 included a new, extensive set of tools to automate the regression process already embedded in the tool. The release functionality includes full capabilities for automatic chronological testing of various Discounted Cumulative Gain (DCG), Precision, and Recall relevance metrics for Document Human Relevance Testing (HRT4Doc).

STF Users can easily set up automated tests using their own query and rating result sets simply and quickly, and at whatever time intervals as desired. Aperiodic tests (e.g. based on builds or other events) can also be added to the automated regression test. Regression tests may be run against individual metric tests such as human sourced Discounted Cumulative Gain (hDCG), various relevance recall metrics, Precision(k), MRR, MAP, and ERR, both at the individual test level and at the test comparison level. Results partitioned by automatic query classification are also included along with visualizations, data cues and result analysis tools.

This paper will discuss the various uses for automated regression testing, methods to set up, monitor, and analyze automated relevance regressions, examine several cases studies, and discuss the technical and business benefits of automated regression relevance testing.

Keywords: Automated Regression Testing, Human Relevance Testing, DCG, Recall, Search Test Framework

Suggested Citation

Rosenoff, Doug and Diedrichsen, Tara and Kandappanthodi, Deepa and Kottapuzhackal, Sreenath, Automated Regression Testing of Search using Search Test Framework (September 21, 2021). Proceedings of the 5th Annual RELX Search Summit, Available at SSRN: https://ssrn.com/abstract=3963258

Doug Rosenoff (Contact Author)

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Tara Diedrichsen

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Deepa Kandappanthodi

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Sreenath Kottapuzhackal

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
77
PlumX Metrics