Generative AI on the Loose: Impact of Improved AI and Expert Oversight on Knowledge Sharing

Abstract

Generative Artificial Intelligence (GenAI) is transforming knowledge sharing in organizations by generating content, solving complex problems, and providing instant, dialogue-based interactions. However, limitations such as hallucinations and a lack of contextual understanding pose risks, particularly for novices who struggle to assess AI-generated content. This study examines whether advancements in GenAI reduce these concerns and alleviate the need for expert oversight. Using the release of GPT-4 and Stack Overflow's moderation policy changes as a natural experiment, we find that combining GenAI improvements with strict expert oversight reduces the volume of knowledge-sharing activities while significantly enhancing content quality. In contrast, reduced moderation leads to a proliferation of lower-quality content, underscoring the ongoing need for expert-driven oversight. Our findings also reveal heterogeneous effects across user skill levels: while advanced GenAI tools improve the quality of contributions from middle- and low-skilled users, expert oversight remains crucial in maintaining quality, particularly for novice contributors. These insights highlight the evolving role of GenAI in knowledge sharing and the continued importance of expert intervention in ensuring information reliability.

Keywords: generative AI, content moderation, knowledge sharing, online platforms