References (42)


Citations (2)



Small Telescopes: Detectability and the Evaluation of Replication Results

Uri Simonsohn

University of Pennsylvania - The Wharton School

December 10, 2013

How should we evaluate replication results? By far the most common approach is to consider them successful if p<.05, failures otherwise. By ignoring effect size this approach is fundamentally flawed, easily leading to inferences opposite of what the evidence merits. These problems are demonstrated revisiting replications of three famous findings: the embodiment of morality, the endowment effect, and weather & happiness. A new approach is proposed. It combines effect-size estimation with null hypothesis testing into a single test. It evaluates if replications rule-out an effect big enough to be detectable with the original study. Benefits include: (i) distinguishing p>.05 replications that are simply too-noisy from those that indicate the effect is undetectably different from zero, (ii) “protecting” true findings from underpowered replications, (iii) incorporating effect size estimation within a hypothesis-testing framework, and vice versa, and (iv) arriving at intuitively compelling inferences in general and for the revisited replications in particular.

Number of Pages in PDF File: 31

Keywords: Replications, power, p-values, effect size

working papers series

Download This Paper

Date posted: May 3, 2013 ; Last revised: June 11, 2014

Suggested Citation

Simonsohn, Uri, Small Telescopes: Detectability and the Evaluation of Replication Results (December 10, 2013). Available at SSRN: http://ssrn.com/abstract=2259879 or http://dx.doi.org/10.2139/ssrn.2259879

Contact Information

Uri Simonsohn (Contact Author)
University of Pennsylvania - The Wharton School ( email )
3730 Walnut Street
JMHH 500
Philadelphia, PA 19104-6365
United States
Feedback to SSRN

Paper statistics
Abstract Views: 7,194
Downloads: 1,139
Download Rank: 9,167
References:  42
Citations:  2
People who downloaded this paper also downloaded:
1. P-Curve: A Key to the File Drawer
By Uri Simonsohn, Leif Nelson, ...

© 2014 Social Science Electronic Publishing, Inc. All Rights Reserved.  FAQ   Terms of Use   Privacy Policy   Copyright   Contact Us
This page was processed by apollo3 in 0.360 seconds