header

An Overview of Semantic Search Evaluation Initiatives

30 Pages Posted: 23 Jun 2018 First Look: Accepted

See all articles by Khadija M. Elbedweihy

Khadija M. Elbedweihy

University of Sheffield - Department of Computer Science

Stuart N. Wrigley

University of Sheffield - Department of Computer Science

Paul Clough

University of Sheffield - Information School

Fabio Ciravegna

University of Sheffield - The OAK Group (Organisations, Information and Knowledge Group)

Abstract

Recent work on searching the Semantic Web has yielded a wide range of approaches with respect to the underlying search mechanisms, results management and presentation, and style of input. Each approach impacts upon the quality of the information retrieved and the user’s experience of the search process. However, despite the wealth of experience accumulated from evaluating Information Retrieval (IR) systems, the evaluation of Semantic Web search systems has largely been developed in isolation from mainstream IR evaluation with a far less unified approach to the design of evaluation activities. This has led to slow progress and low interest when compared to other established evaluation series, such as TREC for IR or OAEI for Ontology Matching. In this paper, we review existing approaches to IR evaluation and analyse evaluation activities for SemanticWeb search systems. Through a discussion of these, we identify their weaknesses and highlight the future need for a more comprehensive evaluation framework that addresses current limitations.

Keywords: semantic search, usability, evaluation, benchmarking, performance, information retrieval

Suggested Citation

Elbedweihy, Khadija M. and Wrigley, Stuart N. and Clough, Paul and Ciravegna, Fabio, An Overview of Semantic Search Evaluation Initiatives (2015). Journal of Web Semantics First Look 30_0_6. Available at SSRN: https://ssrn.com/abstract=3199177 or http://dx.doi.org/10.2139/ssrn.3199177

Khadija M. Elbedweihy (Contact Author)

University of Sheffield - Department of Computer Science ( email )

Regent Court, 211 Portobello
Sheffield
United Kingdom

Stuart N. Wrigley

University of Sheffield - Department of Computer Science

Regent Court, 211 Portobello
Sheffield
United Kingdom

Paul Clough

University of Sheffield - Information School ( email )

211 Regent Court
Sheffield
United Kingdom

Fabio Ciravegna

University of Sheffield - The OAK Group (Organisations, Information and Knowledge Group)

Regent Court, 211 Portobello
Sheffield, +44 114 222 1800
United Kingdom

Register to save articles to
your library

Register

Paper statistics

Abstract Views
143
Downloads
5