Towards the Automated Evaluation of Crowd Work: Machine-Learning Based Classification of Complex Texts Simplified by Laymen

46th Hawaii International Conference on System Sciences, January 2013

11 Pages Posted: 1 Aug 2014

See all articles by Holger Hoffmann

Holger Hoffmann

University of Kassel

Angelika Cosima Bullinger

University of Erlangen-Nuremberg; University of Pennsylvania - School of Arts & Sciences

Christiane Fellbaum

Princeton University

Date Written: 2013

Abstract

The work paradigm of crowdsourcing holds huge potential for organizations by providing access to a large workforce. However, an increase of crowd work entails increasing effort to evaluate the quality of the submissions. As evaluations by experts are inefficient, time-consuming, expensive, and are not guaranteed to be effective, our paper presents a concept for an automated classification process for crowd work. Using the example of crowd generated patent transcripts we build on interdisciplinary research to present an approach to classifying them along two dimensions – correctness and readability. To achieve this, we identify and select text attributes from different disciplines as input for machine- learning classification algorithms and evaluate the suitability of three well regarded algorithms, Neural Networks, Support Vector Machines and k-Nearest Neighbor algorithms. Key findings are that the proposed classification approach is feasible and the SVM classifier performs best in our experiment.

Suggested Citation

Hoffmann, Holger and Bullinger, Angelika Cosima and Fellbaum, Christiane, Towards the Automated Evaluation of Crowd Work: Machine-Learning Based Classification of Complex Texts Simplified by Laymen (2013). 46th Hawaii International Conference on System Sciences, January 2013, Available at SSRN: https://ssrn.com/abstract=2474180

Holger Hoffmann (Contact Author)

University of Kassel ( email )

Fachbereich 05
Nora-Platiel-Straße 1
34109 Kassel, Hessen 34127
Germany

Angelika Cosima Bullinger

University of Erlangen-Nuremberg ( email )

Schloßplatz 4
Nuremberg, DE Bavaria 91054
Germany

University of Pennsylvania - School of Arts & Sciences ( email )

Philadelphia, PA 19104
United States

Christiane Fellbaum

Princeton University ( email )

22 Chambers Street
Princeton, NJ 08544-0708
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
18
Abstract Views
461
PlumX Metrics