Unraveling the Impact: An Empirical Investigation of ChatGPT's Exclusion from Stack Overflow
26 Pages Posted: 18 Jun 2023 Last revised: 4 Oct 2023
Date Written: June 04, 2024
Abstract
This study explores the implications of a policy change announced by Stack Overflow, a popular question-and-answer platform, to prohibit the use of ChatGPT for generating questions or answers. To analyze data from Stack Overflow and Reddit, we employ the difference-indifferences (DID) empirical strategy to examine the impact of Stack Overflow's policy. Our analysis focuses on quality measures using natural language processing (NLP) techniques and voting features on both platforms. The results indicate that, after the restriction, users on Stack Overflow exhibited a shift in their answer generation, producing answers with a more positive tone and increased number of words and characters compared to users on the AskProgramming subreddit. However, there was no discernible impact on the quality of questions asked on Stack Overflow. Subjectivity in the text of questions and answers did not exhibit significant changes either. Notably, the answers on Stack Overflow received a higher number of upvotes after the restriction, suggesting that the community perceived them to be of higher quality compared to answers on the subreddit, even though the complexity of the answers went up. The frequency analysis reveals a noticeable decrease in the number of questions and answers posted by users on Stack Overflow following the restriction. Additionally, our supplementary analysis suggests a decrease in first time contributors on Stack Overflow in the aftermath of the restriction. This study contributes to the existing literature by providing empirical evidence and insights into the decision to prohibit ChatGPT on a question-and-answer platform and provides practical implications to the platform managers, informing them about the impact of restriction on the change in content quality.
Keywords: generative AI, discussion platform, stackoverflow, ChatGPT, Reddit
Suggested Citation: Suggested Citation