Using Search Test Framework and Human Relevance Testing to Evaluate Search Engine Changes for Impact and Relevance Improvements

Posted: 16 Dec 2019

Date Written: December 16, 2019

Abstract

Search Test Framework (STF) is a state-of-the-art tool for quantifying and comparing search relevance results. In this work, we cover how to use STF, including logging into the tool, creating Singleton, Comparison and Regression tests, and viewing the test results. After covering the basics on how to use STF, we focus on how to use STF to see changes to impact and relevance using a baseline set of queries which have been evaluated in a Human Relevance Test (HRT) study with DCG-Human comparison test results. Finally, we walk through the process that was used to analyze the results of a successful algorithm update.

Keywords: relevance testing, search algorithm evaluation, search engine, STF, Search Test Framework

Suggested Citation

Chatelain, Edward and Vu, Khanh and Rosenoff, Doug and Diedrichsen, Tara, Using Search Test Framework and Human Relevance Testing to Evaluate Search Engine Changes for Impact and Relevance Improvements (December 16, 2019). Proceedings of the 3rd Annual RELX Search Summit, Available at SSRN: https://ssrn.com/abstract=3504649

Edward Chatelain (Contact Author)

LexisNexis ( email )

P. O. Box 933
Raleigh, NC 27606
United States

Khanh Vu

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Doug Rosenoff

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Tara Diedrichsen

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
192
PlumX Metrics