Measuring Semantic Preservation in Machine Translation with HCOMET: Human Cognitive Metric for Evaluating Translation

75 Pages Posted: 29 Apr 2021

See all articles by João Marinotti

João Marinotti

Center for Law, Society and Culture, Indiana University Maurer School of Law; Information Society Project, Yale Law School; Center for Intellectual Property Research, Indiana University Maurer School of Law; The City University of New York - The Graduate Center, City University of New York

Date Written: August 20, 2014

Abstract

Human ranking of machine translation output is a commonly used method for comparing different innovations in machine translation research. Theoretically simple, the comparison of multiple translations is, in effect, cognitively complex, requiring judges to balance the weight of different types of translation errors in the context of the whole sentence. This cognitive complexity is made evident through low intra- and inter- annotator agreements, which call into question the reliability of such ranking metrics. HMEANT (Lo and Wu, 2011) attempted to decrease the complexity of ranking by dividing sentences into smaller semantic units whose translation alignments were more objective, rendering the task cognitively simpler. However, HMEANT does not discern how these semantic units are related and relies heavily on language-dependent verb frames – a significant problem for a translation metric. This project defines a new set of human metrics focusing on HCOMET (Human COgnitive Metric for Evaluating Translation). HCOMET, attempting to overcome the limitations of HMEANT, employed a new cognitively-informed annotation scheme and new scoring guidelines. While the inter-annotator agreement did not surpass that of HMEANT, the conceptual framework of HCOMET allows for a much more detailed analysis of semantic adequacy in machine translation.

Keywords: Machine Translation, NLP, Metric, Computational Linguistics

Suggested Citation

Marinotti, João, Measuring Semantic Preservation in Machine Translation with HCOMET: Human Cognitive Metric for Evaluating Translation (August 20, 2014). Available at SSRN: https://ssrn.com/abstract=3830666 or http://dx.doi.org/10.2139/ssrn.3830666

João Marinotti (Contact Author)

Center for Law, Society and Culture, Indiana University Maurer School of Law ( email )

211 S. Indiana Avenue
Bloomington, IN 47405
United States

Information Society Project, Yale Law School ( email )

P.O. Box 208215
New Haven, CT 06520-8215
United States

HOME PAGE: http://law.yale.edu/joao-marinotti

Center for Intellectual Property Research, Indiana University Maurer School of Law ( email )

211 South Indiana Avenue
Bloomington, IN 47405
United States

The City University of New York - The Graduate Center, City University of New York ( email )

365 Fifth Avenue
New York, NY 10016
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
2
Abstract Views
194
PlumX Metrics