Real-Effort Incentives in Online Labor Markets: Punishments and Rewards for Individuals and Groups

Abstract

Online labor markets and the humans that power them serve a critical role in the advancement of artificial intelligence and supervised machine learning via the creation of high-quality training datasets. The use of human effort in online labor markets is not enough, however, as a key factor is understanding the possible interventions that market operators can leverage to incentivize high-quality human effort among their labor force. We propose that platforms could implement mechanisms such as rewards or punishments at individual or group levels to incentivize real-effort and high-quality output. We apply our interventions using a collaborative image tagging experiment and the results provide interesting insights and non-obvious consequences. On average, interventions applied at the group level outperformed interventions applied at the individual level. Punishing the group provides the most controversial incentive strategy and provides a non-obvious significant improvement in effort. Rewarding or sanctioning an individual had similar effects on average, with both treatments leading to significant increases in effort post-intervention. In contrast to predictions, sanctioning appears to significantly motivate those that were punished. Overall, interventions applied in our real-effort collaborative image tagging experiment had a significant impact on behavior and provides guidance for online labor market operators.

Keywords: online labor market, free-riding, incentive mechanism, economic experiment, economics of IS