The Language of (Non)replicable Social Science

Columbia Business School Research Paper No. 4798327

Forthcoming at Psychological Science

66 Pages Posted: 19 Apr 2024

See all articles by Michal Herzenstein

Michal Herzenstein

University of Delaware

Sanjana Rosario

Columbia University - Columbia Business School, Marketing

Shin Oblander

Columbia University - Columbia Business School, Marketing

Oded Netzer

Columbia University - Columbia Business School, Marketing

Date Written: April 11, 2024

Abstract

Using publicly available data from 299 pre-registered replications from the social sciences, we find that the language used to describe a study can predict its replicability above and beyond a large set of controls related to the paper characteristics, study design and results, author information, and replication effort. To understand why, we analyze the textual differences between replicable and nonreplicable studies. Our findings suggest that the language in replicable studies is transparent and confident, written in a detailed and complex manner, and generally exhibits markers of truthful communication, possibly demonstrating the researchers’ confidence in the study. Nonreplicable studies, however, are vaguely written and have markers of persuasion techniques such as the use of positivity and clout. Thus, our findings allude to the possibility that authors of nonreplicable studies are more likely to make an effort, through their writing, to persuade readers of their (possibly weaker) results.

Keywords: Open Science, Replication Prediction, Text Analysis, Psychometric Properties of Language, Machine Learning Models, Computational Social Sciences

Suggested Citation

Herzenstein, Michal and Rosario, Sanjana and Oblander, Shin and Netzer, Oded, The Language of (Non)replicable Social Science (April 11, 2024). Columbia Business School Research Paper No. 4798327, Forthcoming at Psychological Science, Available at SSRN: https://ssrn.com/abstract=4798327

Michal Herzenstein (Contact Author)

University of Delaware ( email )

Newark, DE 19711
United States

Sanjana Rosario

Columbia University - Columbia Business School, Marketing ( email )

New York, NY 10027
United States

Shin Oblander

Columbia University - Columbia Business School, Marketing ( email )

New York, NY 10027
United States

HOME PAGE: http://www.shin.marketing

Oded Netzer

Columbia University - Columbia Business School, Marketing ( email )

New York, NY 10027
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
84
Abstract Views
324
Rank
551,099
PlumX Metrics