Toward Human-Centered AI Management: Methodological Challenges and Future Directions

Abstract

As algorithms powered by Artificial Intelligence (AI) are increasingly involved in the management of organizations, it becomes imperative to understand people’s feelings and behaviors when machines gain power over humans. There are two mainstream methods for doing so, vignette studies and case studies. Both can reveal important insights into human-centered AI management, but they also yield inconsistent findings, for example on the attitude people have toward AI management. We discuss how the respective limitations of the two methods may be the drivers of these inconsistent findings, and emphasize the advantages of a third method for mitigating these limitations: field experiments on crowdsourced marketplaces. Such field experiments go beyond using crowdsourced marketplaces as human research subject pools, and use them instead as models of workplaces where workers can experience actual AI management under different configurations. Through a proof-of-concept study on Amazon Mechanical Turk (Mturk; as a world-leading crowdsourcing platform), we showed unique human reactions to AI management, which were not perfectly aligned with those in vignettes or case studies. We suggest that field experiments on crowdsourced marketplaces can provide participants with the actual experience of AI management, facilitating robust predictions and allowing for timely behavioral research on AI-powered workflows and organizations.

Keywords: Artificial Intelligence, algorithmic management, algorithm aversion, algorithm appreciation, future of work, work design, crowdsourcing