Unraveling the Impact: An Empirical Investigation of ChatGPT's Exclusion from Stack Overflow

26 Pages Posted: 18 Jun 2023 Last revised: 4 Oct 2023

See all articles by Sameer Borwankar

Sameer Borwankar

McGill University - Desautels Faculty of Management; Purdue University - Krannert School of Management

Warut Khern-am-nuai

McGill University - Desautels Faculty of Management

Karthik Natarajan Kannan

Purdue University

Date Written: June 04, 2024

Abstract

This study explores the implications of a policy change announced by Stack Overflow, a popular question-and-answer platform, to prohibit the use of ChatGPT for generating questions or answers. To analyze data from Stack Overflow and Reddit, we employ the difference-indifferences (DID) empirical strategy to examine the impact of Stack Overflow's policy. Our analysis focuses on quality measures using natural language processing (NLP) techniques and voting features on both platforms. The results indicate that, after the restriction, users on Stack Overflow exhibited a shift in their answer generation, producing answers with a more positive tone and increased number of words and characters compared to users on the AskProgramming subreddit. However, there was no discernible impact on the quality of questions asked on Stack Overflow. Subjectivity in the text of questions and answers did not exhibit significant changes either. Notably, the answers on Stack Overflow received a higher number of upvotes after the restriction, suggesting that the community perceived them to be of higher quality compared to answers on the subreddit, even though the complexity of the answers went up. The frequency analysis reveals a noticeable decrease in the number of questions and answers posted by users on Stack Overflow following the restriction. Additionally, our supplementary analysis suggests a decrease in first time contributors on Stack Overflow in the aftermath of the restriction. This study contributes to the existing literature by providing empirical evidence and insights into the decision to prohibit ChatGPT on a question-and-answer platform and provides practical implications to the platform managers, informing them about the impact of restriction on the change in content quality.

Keywords: generative AI, discussion platform, stackoverflow, ChatGPT, Reddit

Suggested Citation

Borwankar, Sameer and Khern-am-nuai, Warut and Kannan, Karthik Natarajan, Unraveling the Impact: An Empirical Investigation of ChatGPT's Exclusion from Stack Overflow (June 04, 2024). Available at SSRN: https://ssrn.com/abstract=4481959

Sameer Borwankar (Contact Author)

McGill University - Desautels Faculty of Management ( email )

1001 Sherbrooke St W
Montreal, Quebec h3A 1G5

Purdue University - Krannert School of Management ( email )

1310 Krannert Building
West Lafayette, IN 47907-1310
United States

Warut Khern-am-nuai

McGill University - Desautels Faculty of Management ( email )

1001 Sherbrooke St. West
Montreal, Quebec H3A1G5 H3A 1G5
Canada

Karthik Natarajan Kannan

Purdue University ( email )

Krannert School of Management
West Lafayette, IN 47907
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
362
Abstract Views
1,102
Rank
170,939
PlumX Metrics