An Introduction to Query Token Density (QTD) Search Algorithms

Posted: 16 Dec 2019

Date Written: December 16, 2019

Abstract

A new class of search algorithms and metrics based on an idea called Query Token Density (QTD) has been created and implemented in the Lexis Nexis search engine. QTD is based on the notion that dense, high-frequency co-occurrence of query terms within a document is a proxy for high user relevance. QTD has proven to be highly successful at improving Discounted Cumulative Gain (DCG) based relevance precision for multi-token Boolean and Natural Language queries, though results are significantly better for Natural Language searches.

We discuss the computation and testing of the QTD algorithm including the initial implementations, testing of the algorithm via human subject matter expert relevance testing completed on thousands of queries and results, end-customer facing A/B testing, and performance testing in a cloud-based environment. We also discuss some additional notes on implementation, possible extensions for various content types, algorithm implementation improvements, and some variants of the algorithm to be explored in future work.

Keywords: Search Engine Algorithm, Search Algorithms, Query Token Density, QTD, algorithm implementation, algorithm testing, Natural Language search, Boolean search

Suggested Citation

Rosenoff, Doug and Vu, Khanh, An Introduction to Query Token Density (QTD) Search Algorithms (December 16, 2019). Proceedings of the 3rd Annual RELX Search Summit, Available at SSRN: https://ssrn.com/abstract=3504653

Doug Rosenoff (Contact Author)

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Khanh Vu

LexisNexis ( email )

P. O. Box 933
Dayton, OH 45401
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
133
PlumX Metrics