Abstract

http://ssrn.com/abstract=2201651
 
 

References (34)



 
 

Citations (2)



 


 



Evaluating Design Solutions Using Crowds


Jin Bao


Stevens Institute of Technology - Wesley J. Howe School of Technology Management

Yasuaki Sakamoto


Stevens Institute of Technology - Wesley J. Howe School of Technology Management

Jeffrey V. Nickerson


Stevens Institute of Technology - Wesley J. Howe School of Technology Management

2011

Seventeenth Americas Conference on Information Systems, August 4th-7th 2011, Detroit, Michigan
Howe School Research Paper No. 2013-5

Abstract:     
Crowds can be used to generate and evaluate design solutions. To increase a crowdsourcing system’s effectiveness, we propose and compare two evaluation methods, one using five-point Likert scale rating and the other prediction voting. Our results indicate that although the two evaluation methods correlate, they have different goals: whereas prediction voting focuses evaluators on identifying the very best solutions, the rating focuses evaluators on the entire range of solutions. Thus, prediction voting is appropriate when there are many poor quality solutions that need to be filtered out, and rating is suited when all ideas are reasonable and distinctions need to be made across all solutions. The crowd prefers participating in prediction voting. The results have pragmatic implications, suggesting that evaluation methods should be assigned in relation to the distribution of quality present at each stage of crowdsourcing.

Number of Pages in PDF File: 10

Keywords: crowdsourcing, creativity, human-based genetic algorithms, evaluation, Mechanical Turk

working papers series


Download This Paper

Date posted: January 16, 2013 ; Last revised: March 19, 2013

Suggested Citation

Bao, Jin and Sakamoto, Yasuaki and Nickerson, Jeffrey V., Evaluating Design Solutions Using Crowds (2011). Seventeenth Americas Conference on Information Systems, August 4th-7th 2011, Detroit, Michigan; Howe School Research Paper No. 2013-5. Available at SSRN: http://ssrn.com/abstract=2201651 or http://dx.doi.org/10.2139/ssrn.2201651

Contact Information

Jin Bao (Contact Author)
Stevens Institute of Technology - Wesley J. Howe School of Technology Management ( email )
Hoboken, NJ 07030
United States

Yasuaki Sakamoto
Stevens Institute of Technology - Wesley J. Howe School of Technology Management ( email )
Hoboken, NJ 07030
United States

Jeffrey V. Nickerson
Stevens Institute of Technology - Wesley J. Howe School of Technology Management ( email )
Hoboken, NJ 07030
United States

Feedback to SSRN


Paper statistics
Abstract Views: 329
Downloads: 54
Download Rank: 223,826
References:  34
Citations:  2

© 2014 Social Science Electronic Publishing, Inc. All Rights Reserved.  FAQ   Terms of Use   Privacy Policy   Copyright   Contact Us
This page was processed by apollo6 in 0.297 seconds