When Advanced AI Isn't Enough: Human Factors as Drivers of Success in Generative AI-Human Collaborations

Abstract

In this comprehensive study, we explore the dynamics of human-AI collaboration through two randomized controlled experiments, focusing on the role of generative AI and its interaction with humans. Our investigation demonstrates that access to generative AI significantly enhances performance outcomes, highlighting its importance as a performance determinant. However, our findings challenge the notion of AI as a great equalizer; while AI usage leads to improved performance, it does not necessarily compress variance among individuals, indicating the emergence of new skill disparities in the AI era. We found that working with advanced AI models, such as GPT-4.0, only slightly improves performance compared to using a less advanced model, suggesting that technological advancement is not the sole determining factor in collaboration outcomes. This underscores the importance of AI literacy as a unique and essential ability in the era of AI. Furthermore, our results reveal that AI collaboration training significantly improves performance by changing human-AI interaction patterns, as evidenced by the analysis of human-AI conversation logs. Our study provides valuable insights for organizations and policymakers, emphasizing the need to invest in human capital and AI literacy to harness the full potential of generative AI collaborations. As AI technologies continue to evolve, understanding and nurturing the human-AI partnership will be crucial for achieving optimal performance in the workplace.

Keywords: Generative AI, Human-AI Collaboration, AI Training, Performance Outcomes, Human Factors

JEL Classification: C91 - Laboratory, Individual Behavior