Crowd-sourcing Hypothesis Tests: Making Transparent How Design Choices Shape Research Results

In Press at Psychological Bulletin, 2020

83 Pages Posted: 9 Sep 2020

See all articles by Justin Landy

Justin Landy

University of Chicago - Booth School of Business

Miaolei Jia

affiliation not provided to SSRN

Isabel Ding

affiliation not provided to SSRN

Domenico Viganola

affiliation not provided to SSRN

Warren Tierney

affiliation not provided to SSRN

Anna Dreber

Stockholm School of Economics - Department of Economics

Magnus Johannesson

Stockholm School of Economics - Department of Economics

Thomas Pfeiffer

affiliation not provided to SSRN

Charles Ebersole

affiliation not provided to SSRN

Quentin Gronau

affiliation not provided to SSRN

Alexander Ly

affiliation not provided to SSRN

Don van den Bergh

affiliation not provided to SSRN

Maarten Marsman

affiliation not provided to SSRN

Koen Derks

affiliation not provided to SSRN

Eric-Jan Wagenmakers

University of Amsterdam

Andrew Proctor

affiliation not provided to SSRN

Daniel M. Bartels

University of Chicago - Booth School of Business

Christopher W. Bauman

University of California, Irvine

William J. Brady

Yale University

Felix Cheung

affiliation not provided to SSRN

Andrei Cimpian

affiliation not provided to SSRN

Simone Dohle

ETH Zürich

M. Brent Donnellan

affiliation not provided to SSRN

Adam Hahn

affiliation not provided to SSRN

Michael P. Hall

affiliation not provided to SSRN

William Jiménez-Leal

Universidad de los Andes

David J. Johnson

affiliation not provided to SSRN

Richard E. Lucas

Michigan State University

Benoit Monin

Stanford Graduate School of Business

Andres Montealegre

affiliation not provided to SSRN

Elizabeth Mullen

San Jose State University - Organization and Management

Jun Pang

Renmin University of China - School of Business

Jennifer Ray

affiliation not provided to SSRN

Diego A. Reinero

affiliation not provided to SSRN

Jesse Reynolds

affiliation not provided to SSRN

Walter Sowden

affiliation not provided to SSRN

Daniel Storage

affiliation not provided to SSRN

Runkun Su

affiliation not provided to SSRN

Christina M. Tworek

affiliation not provided to SSRN

Jay Van Bavel

New York University (NYU) - Department of Psychology; NHH Norwegian School of Economics

Daniel Walco

University of Chicago - Booth School of Business

Julian Wills

affiliation not provided to SSRN

Xiaobing XU

affiliation not provided to SSRN

Kai Chi Yam

National University of Singapore (NUS) - Department of Management and Organization

Xiaoyu Yang

affiliation not provided to SSRN

William A. Cunningham

affiliation not provided to SSRN

Martin Schweinsberg

ESMT European School of Management and Technology

Molly Urwitz

affiliation not provided to SSRN

Eric Luis Uhlmann

HEC Paris - Management & Human Resources

Date Written: July 17, 2020

Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from two separate large samples (total N > 15,000) were then randomly assigned to complete one version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: materials from different teams rendered statistically significant effects in opposite directions for four out of five hypotheses, with the narrowest range in estimates being d = -0.37 to +0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for two hypotheses, and a lack of support for three hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, while considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowd-sourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.

Keywords: Crowd-sourcing, Scientific Transparency, Stimulus Sampling, Forecasting, Conceptual Replications, Research Robustness

Suggested Citation

Landy, Justin and Jia, Miaolei and Ding, Isabel and Viganola, Domenico and Tierney, Warren and Dreber, Anna and Johanneson, Magnus and Pfeiffer, Thomas and Ebersole, Charles and Gronau, Quentin and Ly, Alexander and van den Bergh, Don and Marsman, Maarten and Derks, Koen and Wagenmakers, Eric-Jan and Proctor, Andrew and Bartels, Daniel M. and Bauman, Christopher W. and Brady, William J. and Cheung, Felix and Cimpian, Andrei and Dohle, Simone and Donnellan, M. Brent and Hahn, Adam and Hall, Michael P. and Jiménez-Leal, William and Johnson, David J. and Lucas, Richard E. and Monin, Benoit and Montealegre, Andres and Mullen, Elizabeth and Pang, Jun and Ray, Jennifer and Reinero, Diego A. and Reynolds, Jesse and Sowden, Walter and Storage, Daniel and Su, Runkun and Tworek, Christina M. and Van Bavel, Jay and Walco, Daniel and Wills, Julian and XU, Xiaobing and Yam, Kai Chi and Yang, Xiaoyu and Cunningham, William A. and Schweinsberg, Martin and Urwitz, Molly and Uhlmann, Eric Luis, Crowd-sourcing Hypothesis Tests: Making Transparent How Design Choices Shape Research Results (July 17, 2020). In Press at Psychological Bulletin, 2020, Available at SSRN: https://ssrn.com/abstract=3654406 or http://dx.doi.org/10.2139/ssrn.3654406

Justin Landy

University of Chicago - Booth School of Business ( email )

5807 S. Woodlawn Avenue
Chicago, IL 60637
United States

Miaolei Jia

affiliation not provided to SSRN

Isabel Ding

affiliation not provided to SSRN

Domenico Viganola

affiliation not provided to SSRN

Warren Tierney

affiliation not provided to SSRN

Anna Dreber

Stockholm School of Economics - Department of Economics ( email )

P.O. Box 6501
Sveavagen 65
S-113 83 Stockholm
Sweden

Magnus Johanneson

Stockholm School of Economics - Department of Economics ( email )

P.O. Box 6501
Sveavagen 65
S-113 83 Stockholm
Sweden
+46 8 736 9443 (Phone)

HOME PAGE: http://www.hhs.se/Faculty/showperson.htm?personid=198

Thomas Pfeiffer

affiliation not provided to SSRN

Charles Ebersole

affiliation not provided to SSRN

Quentin Gronau

affiliation not provided to SSRN

Alexander Ly

affiliation not provided to SSRN

Don Van den Bergh

affiliation not provided to SSRN

Maarten Marsman

affiliation not provided to SSRN

Koen Derks

affiliation not provided to SSRN

Eric-Jan Wagenmakers

University of Amsterdam ( email )

Spui 21
Amsterdam, 1018 WB
Netherlands

Andrew Proctor

affiliation not provided to SSRN

Daniel M. Bartels (Contact Author)

University of Chicago - Booth School of Business ( email )

5807 S. Woodlawn Avenue
Chicago, IL 60637
United States

Christopher W. Bauman

University of California, Irvine ( email )

Paul Merage School of Business
University of California, Irvine
Irvine, CA 92697
United States

HOME PAGE: http://https://merage.uci.edu/research-faculty/faculty-directory/Christopher-Bauman.html

William J. Brady

Yale University ( email )

493 College St
New Haven, CT CT 06520
United States

Felix Cheung

affiliation not provided to SSRN

Andrei Cimpian

affiliation not provided to SSRN

Simone Dohle

ETH Zürich ( email )

Universitatsstrasse 22
CHN H75.3
Zürich, 8092
Switzerland

M. Brent Donnellan

affiliation not provided to SSRN

Adam Hahn

affiliation not provided to SSRN

Michael P. Hall

affiliation not provided to SSRN

William Jiménez-Leal

Universidad de los Andes ( email )

Colombia

David J. Johnson

affiliation not provided to SSRN

Richard E. Lucas

Michigan State University ( email )

Agriculture Hall
East Lansing, MI 48824-1122
United States

HOME PAGE: http://www.msu.edu/user/lucasri/

Benoit Monin

Stanford Graduate School of Business ( email )

655 Knight Way
Stanford, CA 94305-5015
United States

Andres Montealegre

affiliation not provided to SSRN

Elizabeth Mullen

San Jose State University - Organization and Management ( email )

United States

Jun Pang

Renmin University of China - School of Business ( email )

Beijing, 100872
China

Jennifer Ray

affiliation not provided to SSRN

Diego A. Reinero

affiliation not provided to SSRN

Jesse Reynolds

affiliation not provided to SSRN

Walter Sowden

affiliation not provided to SSRN

Daniel Storage

affiliation not provided to SSRN

Runkun Su

affiliation not provided to SSRN

Christina M. Tworek

affiliation not provided to SSRN

Jay Van Bavel

New York University (NYU) - Department of Psychology ( email )

New York, NY 10003
United States

NHH Norwegian School of Economics

Helleveien 30
Bergen, NO-5045
Norway

Daniel Walco

University of Chicago - Booth School of Business ( email )

5807 S. Woodlawn Avenue
Chicago, IL 60637
United States

Julian Wills

affiliation not provided to SSRN

Xiaobing XU

affiliation not provided to SSRN

Kai Chi Yam

National University of Singapore (NUS) - Department of Management and Organization ( email )

Singapore
Singapore

Xiaoyu Yang

affiliation not provided to SSRN

William A. Cunningham

affiliation not provided to SSRN

Martin Schweinsberg

ESMT European School of Management and Technology ( email )

Schlossplatz 1
10117 Berlin
Germany

Molly Urwitz

affiliation not provided to SSRN

Eric Luis Uhlmann

HEC Paris - Management & Human Resources ( email )

1 rue de la Liberation
Jouy-en-Josas Cedex, 78351
France

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
167
Abstract Views
953
Rank
380,763
PlumX Metrics