Idea Evaluation Mechanisms for Collective Intelligence in Open Innovation Communities: Do Traders Outperform Raters?

Proceedings of 32nd International Conference on Information Systems, 2011

24 Pages Posted: 2 Oct 2011  

Ivo Blohm

University of St. Gallen

Christoph Riedl

Northeastern University - D’Amore-McKim School of Business; Northeastern University - College of Computer and Information Science; Harvard University - Institute for Quantitative Social Science

J. M. Leimeister

University of St. Gallen; University of Kassel - Information Systems

Helmut Krcmar

Technische Universität München (TUM)

Date Written: October 1, 2011

Abstract

The increasing popularity of open innovation approaches has led to the rise of various open innovation communities on the Internet which might contain several thousand user-generated ideas. However, a company’s absorptive capacity is limited regarding such an amount of ideas so that there is a strong need for mechanisms supporting the evaluation of these ideas. In this paper, we focus on the evaluation of such mechanisms for collective idea evaluation. Applying a multi-method approach, we compare six different configurations of a prediction market with a multi-criteria rating scale that performed best in previous research. We combine a web-based experiment with 448 participants, data from a participant survey, and an independent expert jury. Based on cognitive load theory, we explain why a multi-criteria rating scale outperforms prediction markets in terms of evaluation accuracy and evaluation satisfaction. This study contributes to theory building in the emerging field of collective intelligence.

Keywords: open innovation, collective intelligence, rating scales, prediction markets, idea evaluation, communities, crowdsourcing, cognitive load theory

JEL Classification: D80

Suggested Citation

Blohm, Ivo and Riedl, Christoph and Leimeister, J. M. and Krcmar, Helmut, Idea Evaluation Mechanisms for Collective Intelligence in Open Innovation Communities: Do Traders Outperform Raters? (October 1, 2011). Proceedings of 32nd International Conference on Information Systems, 2011. Available at SSRN: https://ssrn.com/abstract=1936590

Ivo Blohm

University of St. Gallen ( email )

Müller-Friedberg-Strass 8
St. Gallen, 9000
Switzerland

Christoph Riedl (Contact Author)

Northeastern University - D’Amore-McKim School of Business ( email )

360 Huntington Ave.
Boston, MA 02115
United States

HOME PAGE: http://www.christophriedl.net

Northeastern University - College of Computer and Information Science ( email )

360 Huntington Avenue
Boston, MA 02115
United States

Harvard University - Institute for Quantitative Social Science ( email )

1737 Cambridge St.
Cambridge, MA 02138
United States

HOME PAGE: http://christophriedl.net

Jan Marco Leimeister

University of St. Gallen ( email )

Varnbuelstr. 14
Saint Gallen, St. Gallen CH-9000
Switzerland

University of Kassel - Information Systems ( email )

Nora-Platiel 4
Kassel, 34127
Germany

Helmut Krcmar

Technische Universität München (TUM) ( email )

Arcisstrasse 21
Munich, 80333
Germany

Register to save articles to
your library

Register

Paper statistics

Downloads
402
rank
64,649
Abstract Views
1,549
PlumX