Scaling Human Effort in Idea Screening and Content Evaluation

61 Pages Posted: 18 Sep 2020 Last revised: 22 Jan 2021

Date Written: September 3, 2020

Abstract

Brands and advertisers often tap into the crowd to generate ideas for new products and ad creatives by hosting ideation contests. Content evaluators then winnow thousands of submitted ideas before a separate stakeholder, such as a manager or client, decides on a small subset to pursue. We demonstrate the information value of data generated by content evaluators in past contests and propose a proof-of-concept machine learning approach to efficiently surface the best submissions in new contests with less human effort. The approach combines ratings by different evaluators based on their correlation with the past stakeholder choices, controlling for submission characteristics and textual content features. Using field data from a crowdsourcing platform, we demonstrate that the approach improves performance by identifying nonlinear transformations and efficiently reweighting evaluator ratings. Implementing the proposed approach can affect the optimal assignment of internal experts to ideation contests. Two evaluators whose votes were a priori equally correlated with sponsor choices may provide substantially different incremental information to improve the model-based idea ranking. We provide additional support for our findings using simulations based on a product design survey.

Keywords: Crowdsourcing, Innovation, Content Evaluation, Wisdom of Crowds, Machine Learning

Suggested Citation

Kireyev, Pavel and Timoshenko, Artem and Yang, Cathy L., Scaling Human Effort in Idea Screening and Content Evaluation (September 3, 2020). INSEAD Working Paper No. 2020/42/MKT, HEC Paris Research Paper No. MKG-2020-1384, Available at SSRN: https://ssrn.com/abstract=3685882 or http://dx.doi.org/10.2139/ssrn.3685882

Pavel Kireyev (Contact Author)

INSEAD ( email )

Boulevard de Constance
77305 Fontainebleau Cedex
France

HOME PAGE: http://www.pavelkireyev.com

Artem Timoshenko

Kellogg School of Management, Northwestern University ( email )

2001 Sheridan Road
Evanston, IL 60208
United States

Cathy L. Yang

HEC Paris - Marketing ( email )

Paris
France

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
75
Abstract Views
514
rank
371,541
PlumX Metrics