Rating Scales for Collective Intelligence in Innovation Communities: Why Quick and Easy Decision Making Does Not Get it Right

Proceedings of Thirty First International Conference on Information Systems, 2010

21 Pages Posted: 25 Nov 2010

See all articles by Christoph Riedl

Christoph Riedl

Northeastern University - D’Amore-McKim School of Business; Northeastern University - College of Computer and Information Science; Harvard University - Institute for Quantitative Social Science

Ivo Blohm

University of St. Gallen

J. M. Leimeister

University of St. Gallen; University of Kassel - Information Systems

Helmut Krcmar

Technische Universität München (TUM)

Date Written: November 24, 2010

Abstract

The increasing popularity of open innovation approaches has lead to the rise of various innovation platforms on the Internet which might contain 10.000s user-generated ideas. However, a company’s absorptive capacity is limited regarding such an amount of ideas so that there is a strong need for mechanism to identify the best ideas. Extending previous decision management research we focus on analyzing effective idea rating and selection mechanisms in online innovation communities and underlying explanations. Using a multi-method approach our research comprises a web-based rating experiment with 313 participants evaluating 24 ideas from a real-world innovation community, data from a survey measuring rating satisfaction of participants, and idea ratings from an independent expert jury. Our findings show that, despite its popular use in online innovation communities, simple rating mechanisms such as thumbs up/down rating or 5-star rating do not produce valid idea rankings and are significantly outperformed by the multi-attribute scale.

Keywords: Open, Innovation, Absorptive Capacity, Rating, Decision Making, Idea Evaluation, Collecting Intelligence

JEL Classification: D80

Suggested Citation

Riedl, Christoph and Blohm, Ivo and Leimeister, Jan Marco and Krcmar, Helmut, Rating Scales for Collective Intelligence in Innovation Communities: Why Quick and Easy Decision Making Does Not Get it Right (November 24, 2010). Proceedings of Thirty First International Conference on Information Systems, 2010, Available at SSRN: https://ssrn.com/abstract=1714524

Christoph Riedl (Contact Author)

Northeastern University - D’Amore-McKim School of Business ( email )

360 Huntington Ave.
Boston, MA 02115
United States

HOME PAGE: http://www.christophriedl.net

Northeastern University - College of Computer and Information Science ( email )

360 Huntington Avenue
Boston, MA 02115
United States

Harvard University - Institute for Quantitative Social Science ( email )

1737 Cambridge St.
Cambridge, MA 02138
United States

HOME PAGE: http://christophriedl.net

Ivo Blohm

University of St. Gallen ( email )

Müller-Friedberg-Strass 8
St. Gallen, 9000
Switzerland

Jan Marco Leimeister

University of St. Gallen ( email )

Varnbuelstr. 14
Saint Gallen, St. Gallen CH-9000
Switzerland

University of Kassel - Information Systems ( email )

Pfannkuchstraße 1
Kassel, 34121
Germany

Helmut Krcmar

Technische Universität München (TUM) ( email )

Arcisstrasse 21
Munich, DE 80333
Germany

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
732
Abstract Views
3,739
rank
45,379
PlumX Metrics