Correlations in Social Neuroscience Aren't Voodoo: A Reply to Vul Et Al.

Posted: 28 Oct 2016

See all articles by Matthew D. Lieberman

Matthew D. Lieberman

University of California, Los Angeles (UCLA) - Department of Psychology

Elliot Berkman

University of Oregon - Department of Psychology

Tor Wager

University of Colorado at Boulder

Date Written: October 24, 2016

Abstract

Vul et al. claim that brain-personality correlations in many social neuroscience studies are “implausibly high,” “likely…spurious,” and “should not be believed.” Several of their conclusions are incorrect due to flawed reasoning, statistical errors, and sampling anomalies. First, the conceptual issues discussed by Vul et al. have little to do with social neuroscience per se and are equally relevant for nearly all fMRI analyses that report measures of effect size from searches over multiple voxels or regions (r, t, or Z statistics). Second, Vul et al. incorrectly claim that whole-brain regression analyses use an invalid and “non-independent” two-step inferential procedure. We explain how whole-brain regressions are a valid single-step method of identifying brain regions that have reliable correlations with individual difference measures. Third, Vul et al. claim that large correlations obtained using whole-brain regression analyses may be the result of noise alone. We provide a simulation to demonstrate that typical fMRI sample sizes (N = 15-20) will only rarely produce large correlations in the absence of any true effect. Fourth, Vul et al. claim that the reported correlations are inflated to the point of being “implausibly high”. Though biased post hoc correlation estimates are a well-known consequence of conducting multiple tests, Vul et al. make inaccurate assumptions when estimating the theoretical ceiling of such correlations. Moreover, Vul et al.’s own meta-analysis suggests that the magnitude of the bias is an increase of approximately .12, a rather modest estimate to inspire the label ‘voodoo’. In addition, after correcting for likely restricted range in several of the “independent” correlations that Vul et al. treat as the gold standard, the means of the “non-independent” and “independent” correlations are nearly identical. Finally, it is troubling that almost 25% of the “non-independent” correlations in the papers reviewed by Vul et al. were omitted from their own meta-analysis without explanation.

Suggested Citation

Lieberman, Matthew D. and Berkman, Elliot and Wager, Tor, Correlations in Social Neuroscience Aren't Voodoo: A Reply to Vul Et Al. (October 24, 2016). Available at SSRN: https://ssrn.com/abstract=2858535

Matthew D. Lieberman

University of California, Los Angeles (UCLA) - Department of Psychology ( email )

Los Angeles, CA 90095-1563
United States

Elliot Berkman

University of Oregon - Department of Psychology ( email )

Eugene, OR 97403
United States
541-346-4909 (Phone)

HOME PAGE: http://sanlab.uoregon.edu

Tor Wager (Contact Author)

University of Colorado at Boulder ( email )

Boulder, CO

Register to save articles to
your library

Register

Paper statistics

Abstract Views
72
PlumX Metrics