header

Evaluating and Comparing Ontology Alignment Systems: An Mcdm Approach

20 Pages Posted: 20 Jan 2021 Publication Status: Accepted

See all articles by Majid Mohammadi

Majid Mohammadi

Delft University of Technology - Faculty of Technology, Policy and Management

Jafar Rezaei

Faculty of Technology, Policy, and Management, Delft University of Technology

Abstract

Ontology alignment is vital in Semantic Web technologies with numerous applications in diverse disciplines. Due to diversity and abundance of ontology alignment systems, a proper evaluation can portray the evolution of ontology alignment and depicts the efficiency of a system for a particular domain. Evaluation can help system designers recognize the strength and shortcomings of their systems, and aid application developers to select a proper alignment system. This article presents a new evaluation and comparison methodology based on multiple performance met-rics that accommodates experts' preferences via a multi-criteria decision-making (MCDM) method, i.e., Bayesian best-worst method (BWM). First, the importance of a performance metric for a specific task/application is determined according to experts' preferences. The alignment systems are then evaluated based on proposed expert-based collective performance (ECP) that takes into account multiple metrics as well as their calibrated importance. For comparison , the alignment systems are ranked based on a probabilistic scheme, where it includes the extent to which one alignment system is preferred over another. The proposed methodology is applied to six tracks from ontology alignment evaluation initiative (OAEI), where the importance of performance metrics are calibrated by designing a survey and eliciting the preferences of ontology alignment experts. Accordingly, the participating alignment systems in the OAEI 2018 are evaluated and ranked. While the proposed methodology is applied to six OAEI tracks to demonstrate its applicability, it can also be applied to any benchmark or application of ontology alignment.

Keywords: Ontology alignment, ranking, evaluation, MCDM, Bayesian BWM

Suggested Citation

Mohammadi, Majid and Rezaei, Jafar, Evaluating and Comparing Ontology Alignment Systems: An Mcdm Approach. Journal of Web Semantics First Look , Available at SSRN: https://ssrn.com/abstract=3769873 or http://dx.doi.org/10.2139/ssrn.3769873

Majid Mohammadi (Contact Author)

Delft University of Technology - Faculty of Technology, Policy and Management ( email )

P.O. Box 5015
2600 GB Delft
Netherlands

Jafar Rezaei

Faculty of Technology, Policy, and Management, Delft University of Technology ( email )

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
82
Downloads
7
PlumX Metrics