Bias Neglect: A Blind Spot in the Evaluation of Scientific Results

Abstract

Experimenter bias occurs when scientists’ hypotheses influence their results, even if involuntarily. Meta-analyses (e.g. Rosenthal & Rubin, 1978) have suggested that in some domains, such as psychology, up to 1/3 of the studies could be unreliable due to such biases. A series of experiments demonstrates that while people are aware of the possibility that scientists can be more biased when the conclusions of their experiments fit their initial hypotheses, they robustly fail to appreciate that they should also be more skeptical of such results. This is true even when participants read descriptions of studies that have been shown to be biased. Moreover, participants take other sources of bias — such as financial incentives — into account, showing that this bias neglect may be specific to theory driven hypothesis testing. In combination with a common style of scientific reporting, bias neglect could lead the public to accept premature conclusions.

Keywords: Experimenter bias, scientific reasoning, decision making, epistemic vigilance