Do We Really 'Know' What We Think We Know? A Case Study of Seminal Research and its Subsequent Overgeneralization

Abstract

This study provides evidence that accounting researchers have not fully appreciated the sensitivity of empirical results to (necessarily) subjective research design choices, and that this failure has led to the subsequent overgeneralization of early evidence. Specifically, we trace the evolution of thought regarding the information content of accounting earnings announcements, highlighting the role of research design choices. A citation analysis reveals that the likely effects of research design choices made by early influential studies have been largely overlooked by subsequent research, as has "anomalous" evidence that was inconsistent with the early studies' inferences.

Our study provides empirical evidence on the effects of two research design choices: (1) market reaction measures, and (2) sample selection criteria. We replicate the strong information content results reported in early research by following the choices employed in that research. However, we show how other research design choices yield very different inferences. In particular, investigation of the proportion of individual earnings announcements that generate significant reactions reveals no evidence of a significant price reaction for the majority of earnings announcements. This evidence suggests that the focus on cross-sectional means has obscured the fact that significant mean reactions have been driven by a small proportion of announcements. We also find that sample selection criteria significantly affect the magnitude and pervasiveness of market reactions to earnings announcements.

While accounting scholars "think we understand" the impact of research design choices, the evidence presented here suggests that we have collectively failed to apply this understanding in practice. Such failure delays the acquisition of knowledge. Furthermore, we conjecture that earlier awareness of evidence such as that presented here may have changed the evolutionary path of subsequent price-earnings research.

JEL Classification: M41, G14, C10