Experts, Coders, and Crowds: An Analysis of Substitutability

122 Pages Posted: 3 Oct 2017  

Kyle L. Marquardt

V-Dem Institute, University of Gothenburg

Daniel Pemstein

North Dakota State University; Göteborg University - V-Dem Institute

Constanza Sanhueza Petrarca

University of Gothenburg

Brigitte Seim

University of North Carolina (UNC) at Chapel Hill

Steven Lloyd Wilson

Göteborg University - V-Dem Institute

Michael Bernhard

University of Florida

Michael Coppedge

University of Notre Dame - Kellogg Institute; University of Notre Dame, Department of Political Science

Staffan I. I. Lindberg

Göteborg University - Varieties of Democracy Institute; Göteborg University - Department of Political Science

Date Written: October 2017

Abstract

Recent work suggests that crowd workers can replace experts and trained coders in common coding tasks. However, while many political science applications require coders to both and relevant information and provide judgment, current studies focus on a limited domain in which experts provide text for crowd workers to code. To address potential over-generalization, we introduce a typology of data producing actors - experts, coders, and crowds - and hypothesize factors which affect crowd-expert substitutability. We use this typology to guide a comparison of data from crowdsourced and expert surveys. Our results provide sharp scope conditions for the substitutability of crowd workers: when coding tasks require contextual and conceptual knowledge, crowds produce substantively dierent data from coders and experts. We also find that crowd workers can cost more than experts in the context of cross-national panels, and that one purported advantage of crowdsourcing - replicability - is undercut by an insucient number of crowd workers.

Suggested Citation

Marquardt, Kyle L. and Pemstein, Daniel and Sanhueza Petrarca, Constanza and Seim, Brigitte and Wilson, Steven Lloyd and Bernhard, Michael and Coppedge, Michael and Lindberg, Staffan I. I., Experts, Coders, and Crowds: An Analysis of Substitutability (October 2017). V-Dem Working Paper 2017:53. Available at SSRN: https://ssrn.com/abstract=3046462 or http://dx.doi.org/10.2139/ssrn.3046462

Kyle L. Marquardt (Contact Author)

V-Dem Institute, University of Gothenburg ( email )

Department of Political Science
Sprängkullsgatan 19, PO 711
Gothenburg, SE 40530
Sweden

HOME PAGE: http://v-dem.net

Daniel Pemstein

North Dakota State University ( email )

Fargo, ND 58105
United States

Göteborg University - V-Dem Institute ( email )

United States

Constanza Sanhueza Petrarca

University of Gothenburg ( email )

Viktoriagatan 30
Goeteborg, 405 30
Sweden

Brigitte Seim

University of North Carolina (UNC) at Chapel Hill ( email )

102 Ridge Road
Chapel Hill, NC 27514
United States

Steven Lloyd Wilson

Göteborg University - V-Dem Institute ( email )

United States

Michael Bernhard

University of Florida ( email )

Michael Coppedge

University of Notre Dame - Kellogg Institute ( email )

Hesburgh Center
Notre Dame, IN 46556
United States

University of Notre Dame, Department of Political Science

216 Hesburgh Center
Notre Dame, IN New South Wales 46556-5646
United States

HOME PAGE: http://www.nd.edu/~mcoppedg/crd

Staffan I. I. Lindberg

Göteborg University - Varieties of Democracy Institute ( email )

Sprängkullsgatan 19
Gothenburg, Gothenburg 405 30
Sweden

HOME PAGE: http://www.pol.gu.se/varianter-pa-demokrati--v-dem-/

Göteborg University - Department of Political Science ( email )

Box 711
Gothenburg, S-405 30
Sweden

HOME PAGE: http://www.pol.gu.se

Register to save articles to
your library

Register

Paper statistics

Downloads
17
Abstract Views
276
PlumX