Bias-Correction Techniques Alone Cannot Determine Whether Ego Depletion is Different from Zero: Commentary on Carter, Kofler, Forster, & McCullough, 2015

Abstract

Carter, Kofler, Forster, & McCullough (2015) conducted a bias-corrected meta-analysis of the so-called ego depletion effect to determine its real size and robustness. Their efforts have raised awareness of how badly meta-analyses can mislead when the articles that go into them are products of publication bias. Despite our genuine enthusiasm for their work, we worry that in their zeal to correct the record of publication bias, they have drawn too heavily on largely untested statistical techniques that can be insensitive and sometimes misleading. We tested a set of bias-correction techniques, including those favored by Carter and colleagues, by simulating 40,000 meta-analyses in a range of situations that approximate what is found in the ego depletion literature, most notably the presence of heterogeneous effects filtered by publication bias. Our simulations revealed that not one of the bias-correction techniques revealed itself superior in all conditions, with corrections performing adequately in some situations but inadequately in others. Such a result implies that meta-analysts ought to present a range of possible effect sizes and to consider them all as being possible. The problem with the ego depletion literature is that the bias-corrected estimates for the overall effect do not converge, with estimates ranging from g=0 to g=0.24 to g=0.26. Despite our admiration for this program of meta-research, we suggest that bias-corrected meta-analyses cannot yet resolve whether the overall ego depletion is different from zero or not.

Keywords: ego depletion; self-control; meta-analysis; publication bias