Search Engine Result Page Human Relevance Testing

Posted: 27 Jan 2021

Date Written: November 4, 2020

Abstract

Human Relevance Testing (HRT) of Documents is an established, well tested, and production level technique for measuring relevance precision. While a powerful method, it is expensive both in terms of time and human resources, even when done using the LexisNexis Search Test Framework tooling to automate the administration, display, and results computation. Because of the level of required resources, it is often not appropriate to use HRT Document testing for high level / reconnaissance / relevance analysis at the Search Engine Result Page (SERP) level, especially for mixed content type results.

The authors present a pair of methods for performing Human Relevance Testing of Search Engine Result Pages (SERP), as gestalt result views, that allow for rapid evaluation of overall SERP quality.

In both methods, a single set of queries is used to generate the SERP result(s). One method, based on examining a single pane view of a Search Engine Result Page (SERP) allows one or more Subject Matter Expert raters to evaluate the overall SERP relevance quality results against the query that generated the SERP result set. Various ratings schemes are provided, and a variety of metrics are generated based on the individual page and cumulative query set results. A second method, based on comparing control and test SERPs generated independently from different search engines, search algorithms, regressions, or other longitudinal groups is also presented. In that technique, both control and test SERPs are presented in a side-by-side comparison view. Independent ratings for each SERP are collected at the same time by one or more Human Subject Matter Expert raters. Various ratings methods and metrics are then computed to compile a set of individual and comparison metrics for SERP relevance.

This presentation discusses the motivations behind the SERP relevance quality evaluation, and, using some initial case studies, will illustrate both methods and how they are integrated into the LexisNexis Search Test Framework. We will also share some discussion of lessons learnt and future directions for exploration.

Keywords: Human Relevance Testing, Search Engine Result Pages, Subject Matter Expert Relevance Ratings, SERP Relevance Techniques

Suggested Citation

Rosenoff, Doug and Diedrichsen, Tara, Search Engine Result Page Human Relevance Testing (November 4, 2020). Proceedings of the 4th Annual RELX Search Summit, Available at SSRN: https://ssrn.com/abstract=3774368

Doug Rosenoff (Contact Author)

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Tara Diedrichsen

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
277
PlumX Metrics